Extract and Summarize Test Failures

Parse test logs to extract failure information and create summary.

When to Use

• CI test run failed, need to understand failures
• Large test suite with many failures, need summary
• Categorizing failures by type (assertion, runtime, timeout)
• Creating PR review comments about failures
• Tracking which tests are failing over time

Quick Reference

# Extract failed test names
grep "FAILED" test_output.log

# Get failure details with context
grep -A 10 "FAILED\|Error\|AssertionError" test_output.log

# Count failures by type
grep "FAILED\|AssertionError\|ValueError\|TypeError" test_output.log | sort | uniq -c

# Extract test summary line
tail -20 test_output.log | grep -E "passed|failed|error"

# Get specific failure info
grep -B 5 "AssertionError" test_output.log | head -50

Failure Extraction Workflow

1. Collect log: Get test output from CI or local run
2. Find failures: Extract FAILED markers and error types
3. Group by type: Categorize assertion vs runtime vs timeout
4. Extract details: Get error messages and stack traces
5. Count totals: Summary of how many failures
6. Identify patterns: Common failure causes
7. Summarize: Create concise failure report

Failure Categories

Assertion Failures:

• AssertionError

  - Expected value check failed

• assert_equal()

  - Values don't match

• assert_true()

  - Condition not true
• Fix: Check test logic and expected values

Type/Attribute Errors:

• AttributeError

  - Missing method/attribute

• TypeError

  - Wrong argument type

• ValueError

  - Invalid value for operation
• Fix: Check code syntax and types

Runtime Errors:

• IndexError

  - Out of bounds access

• KeyError

  - Dictionary key not found

• ZeroDivisionError

  - Division by zero
• Fix: Check boundary conditions

Timeout/Hanging:

• Test takes too long
• Infinite loop or deadlock
• Resource exhaustion
• Fix: Optimize or add timeout

Output Format

Report failures with:

1. Total Failures - Count and percentage of total tests
2. Failure Summary - By type (assertion, runtime, timeout, other)
3. Failed Tests - List of test names that failed
4. Top Issues - Most common failure patterns
5. Error Messages - Representative error snippets
6. Recommendations - Which tests to focus on first

Error Handling

Problem	Solution
No FAILED markers	Check log format, may use different pattern
Truncated output	Get full log from artifacts instead of view
Mixed output types	Filter by log level or timestamp
Encoding issues	Convert to UTF-8 first
Very large logs	Split and process in chunks

References

• See test-diff-analyzer for identifying flaky tests
• See analyze-ci-failure-logs for CI-specific failures
• See generate-fix-suggestions for automated fixes