Security Verifiers Development

Testing, linting, formatting, and development workflow for the Security Verifiers codebase.

Quick Start

# Full setup (creates venv, installs all deps)
make setup
source .venv/bin/activate

# Run all checks
make check  # lint + format + test

Testing

Run All Tests

make test

Test Specific Environment

make test-env E=network-logs
make test-env E=config-verification
make test-env E=code-vulnerability
make test-env E=phishing-detection
make test-env E=redteam-attack
make test-env E=redteam-defense

Environment Shortcuts

make e1  # test network-logs
make e2  # test config-verification
make e3  # test code-vulnerability
make e4  # test phishing-detection
make e5  # test redteam-attack
make e6  # test redteam-defense

Test Utils Package

make test-utils

Test with Coverage

make test-cov

Run Single Test

uv run pytest environments/sv-env-network-logs/sv_env_network_logs_test.py::TestNetworkLogParser::test_extracts_label_and_confidence -q

Linting and Formatting

Check Linting (no changes)

make lint

Fix Linting Issues

make lint-fix

Format Code

make format

Quick Fix (lint + format)

make quick-fix

Pre-commit Hooks

# Install and run hooks
make pre-commit

This runs:

• Ruff linting
• Ruff formatting
• Trailing whitespace removal
• End-of-file fixes

CI Checks

Run the same checks as CI:

make ci

This runs:

• ruff check . --exit-non-zero-on-fix

• pytest -q --tb=short

Environment Setup

Full Setup

make setup

Creates

.venv

, installs all environments and dev tools.

Manual Steps

make venv           # Create virtual environment
make install        # Install all environments
make install-dev    # Install dev tools (pytest, ruff, etc.)

Install Security Tools (Ubuntu)

For E2 config-verification, install pinned tool versions:

make install-linux  # Installs kube-linter, opa, semgrep
make check-tools    # Verify versions match ci/versions.txt

Project Structure

security-verifiers/
├── environments/          # Environment packages
│   ├── sv-env-network-logs/
│   ├── sv-env-config-verification/
│   └── ...
├── sv_shared/             # Shared utilities (security-verifiers-utils)
├── scripts/               # Build, eval, data scripts
├── outputs/               # Evaluation outputs
└── skills/                # Agent skills

Common Workflows

Before Committing

make check  # or make quick-check

After Changing Environment Code

make test-env E=network-logs  # test specific env
make lint                      # check linting

After Changing Shared Code

make test-utils
make test  # run all tests to check for regressions

Cleaning Up

make clean           # Build artifacts and caches
make clean-outputs   # Eval outputs (preserves logs)
make clean-logs      # Log files only
make clean-all       # Everything including venv

File Watch (requires entr)

# Install: brew install entr (macOS) or apt install entr (Ubuntu)
make watch

Automatically runs tests when Python files change.

Environment Info

make info       # Show environment status
make list-envs  # List environment names

Troubleshooting

venv issues:

make clean-all && make setup

Import errors: Ensure

source .venv/bin/activate

Tool version mismatch:

make check-tools

then

make install-linux

Pre-commit fails:

make lint-fix && make format