Test Orchestrator Skill V3 - Pragmatic Testing

Purpose

Coordinates adaptive testing strategy that scales with project maturity, focusing on high-value tests that match implementation reality.

Core Principle

"Test what exists, not what we imagine"

Adaptive Testing Phases

Phase Detection

def detect_project_phase():
    """Auto-detect based on signals"""
    if not os.path.exists('manage.py'):
        return 'prototype'
    elif User.objects.count() < 100:
        return 'mvp'
    elif User.objects.count() < 10000:
        return 'growth'
    else:
        return 'scale'

Phase Requirements

prototype:
  required_tests: [smoke, happy_path]
  optional_tests: []
  coverage_target: 40%
  quality_gate: 6.0
  time_budget: "20% of dev time"

mvp:
  required_tests: [smoke, happy_path, critical_errors, auth]
  optional_tests: [edge_cases]
  coverage_target: 60%
  quality_gate: 7.0
  time_budget: "30% of dev time"

growth:
  required_tests: [happy_path, errors, edge_cases, performance, security]
  optional_tests: [visual, invalid_input]
  coverage_target: 70%
  quality_gate: 7.5
  time_budget: "35% of dev time"

scale:
  required_tests: [all_8_types]
  optional_tests: []
  coverage_target: 80%
  quality_gate: 8.0
  time_budget: "40% of dev time"

Implementation-First Testing

The New Workflow

def generate_tests_from_implementation():
    """
    1. Analyze actual code
    2. Extract real signatures
    3. Generate matching tests
    """
    # Read the actual implementation
    implementation = read_source_file('payment.py')

    # Extract actual method signatures
    actual_fields = extract_model_fields(implementation)
    actual_methods = extract_method_signatures(implementation)

    # Generate tests that match reality
    tests = []
    for method in actual_methods:
        test = generate_test_for_method(method)
        tests.append(test)

    return tests

Example: Reality-Based Test Generation

# Step 1: Read actual model
class Payment(models.Model):
    user = models.ForeignKey(User)  # NOT 'uploaded_by'
    amount = models.DecimalField()   # NOT 'total'

# Step 2: Generate test that matches
def test_payment_creation():
    payment = Payment.objects.create(
        user=test_user,  # Matches actual field name
        amount=100.00     # Matches actual field name
    )

Critical Path Testing

High-Value Test Identification

CRITICAL_PATHS = {
    'authentication': {
        'priority': 'CRITICAL',
        'reason': 'No auth = no users',
        'tests': ['login', 'logout', 'token_valid']
    },
    'payment': {
        'priority': 'CRITICAL',
        'reason': 'No payment = no revenue',
        'tests': ['process_payment', 'handle_failure']
    },
    'data_persistence': {
        'priority': 'CRITICAL',
        'reason': 'Data loss = catastrophic',
        'tests': ['save_data', 'no_deletion']
    }
}

Test Priority Algorithm

def prioritize_tests(feature):
    if feature.affects_revenue:
        return 'CRITICAL'
    elif feature.affects_security:
        return 'HIGH'
    elif feature.affects_core_flow:
        return 'MEDIUM'
    else:
        return 'LOW'

Escape Hatches

Quick Commands

# Skip all tests temporarily
/test-mode skip --reason "hotfix"

# Reduce test requirements
/test-mode simple --only "smoke,happy"

# Override coverage
/test-coverage 40

# Defer testing
/test-defer --until "next-sprint"

Conditional Test Execution

def should_run_tests(context):
    if context.is_hotfix:
        return False
    if context.is_prototype:
        return 'minimal'
    if context.deadline_pressure:
        return 'critical-only'
    return True

ROI-Based Test Selection

ROI Calculator

def calculate_test_roi(test_type, feature):
    """
    ROI = Value / Cost
    """
    bug_prevention_value = estimate_bug_cost(test_type, feature)
    test_creation_cost = estimate_time(test_type)
    test_maintenance_cost = estimate_maintenance(test_type)

    roi = bug_prevention_value / (test_creation_cost + test_maintenance_cost)

    return {
        'roi': roi,
        'worth_writing': roi > 1.5
    }

Test Value Matrix

TEST_VALUES = {
    'auth_test': {'bug_cost': 10000, 'write_time': 30},
    'ui_animation_test': {'bug_cost': 10, 'write_time': 60},
    'payment_test': {'bug_cost': 5000, 'write_time': 45},
    'tooltip_test': {'bug_cost': 5, 'write_time': 30}
}

Test Debt Tracking

Acceptable Debt Levels

def get_acceptable_debt(phase):
    DEBT_LEVELS = {
        'prototype': 0.8,  # 80% debt acceptable
        'mvp': 0.5,        # 50% debt acceptable
        'growth': 0.3,     # 30% debt acceptable
        'scale': 0.1       # 10% debt acceptable
    }
    return DEBT_LEVELS[phase]

Debt Monitoring

def monitor_test_debt():
    metrics = {
        'untested_critical_paths': count_untested_critical(),
        'failing_tests': count_failing(),
        'missing_test_types': identify_gaps(),
        'debt_ratio': calculate_debt_ratio()
    }

    if metrics['debt_ratio'] > get_acceptable_debt(current_phase):
        alert('Test debt exceeding acceptable levels')
        suggest_high_roi_tests()

Progressive Test Addition

Week-by-Week Guide

TESTING_PROGRESSION = {
    'week_1': {
        'focus': 'Get it running',
        'tests': ['smoke_test', 'basic_happy_path'],
        'count': 2
    },
    'week_2': {
        'focus': 'Core features work',
        'tests': ['main_user_flows'],
        'count': 5
    },
    'week_4': {
        'focus': 'Handle failures',
        'tests': ['error_handling', 'auth_tests'],
        'count': 10
    },
    'week_8': {
        'focus': 'Edge cases (if bugs found)',
        'tests': ['reported_bugs_only'],
        'count': '15-20'
    }
}

Test Maintenance Strategy

Fix Priority

def prioritize_test_fixes(failing_tests):
    for test in failing_tests:
        if test.is_implementation_mismatch():
            # Fix test to match code (quick)
            update_test_to_match_implementation(test)
        elif test.reveals_actual_bug():
            # Fix code (important)
            create_bug_ticket(test)
        elif test.is_flaky():
            # Delete or simplify (not worth it)
            mark_for_deletion(test)

Metrics That Matter

Track These

MEANINGFUL_METRICS = {
    'test_maintenance_hours': track_time_spent_fixing_tests(),
    'bugs_caught_by_tests': count_prevented_issues(),
    'false_failure_rate': count_test_vs_implementation_mismatches(),
    'time_to_first_commit': measure_feature_velocity(),
    'test_roi': calculate_value_over_cost()
}

Ignore These (Early Stages)

VANITY_METRICS = {
    'coverage_percentage': "Meaningless if tests are wrong",
    'total_test_count': "Quality > Quantity",
    'test_types_per_feature': "Forced diversity",
    'test_line_count': "Longer != Better"
}

Quick Decision Framework

def decide_testing_strategy(feature):
    # Prototype? Minimal tests
    if project_phase == 'prototype':
        return ['smoke_test', 'one_happy_path']

    # Critical feature? Full tests
    if feature.loses_money_or_data():
        return ['comprehensive_testing']

    # Has users? Test main flows
    if active_users > 0:
        return ['happy_paths', 'basic_errors']

    # Third bug here? Add edge cases
    if bug_count_in_area >= 3:
        return existing_tests + ['edge_cases']

    # Otherwise, minimal
    return ['basic_tests_only']

Migration from V2 (8-Test System)

Simplification Steps

# Step 1: Audit existing tests
/test-audit --show-roi

# Step 2: Delete low-value tests
/test-cleanup --delete-low-roi

# Step 3: Fix mismatches
/test-align --match-implementation

# Step 4: Reduce to critical
/test-reduce --keep-critical-only

# Step 5: Set phase
/test-phase prototype|mvp|growth|scale

Summary

This pragmatic approach:

1. Starts simple - 2 tests minimum
2. Scales with project - More tests as you grow
3. Tests reality - Implementation-first
4. Values ROI - High-value tests only
5. Provides escapes - Skip when needed
6. Reduces overhead - Target 20-35% time on tests, not 60%

The goal: Ship features with confidence, not ceremony.