AgentV Prompt Optimizer

Input Variables

• eval-path

  : Path or glob pattern to the AgentV evaluation file(s) to optimize against

• optimization-log-path

  (optional): Path where optimization progress should be logged

Workflow

1. Initialize

   • Verify

     <eval-path>

     (file or glob) targets the correct system.
   • Identify Prompt Files:
     • Infer prompt files from the eval file content (look for

       file:

       references in

       input

       that match these patterns).
     • Recursively check referenced prompt files for other prompt references (dependencies).
     • If multiple prompts are found, consider ALL of them as candidates for optimization.
   • Identify Optimization Log:
     • If

       <optimization-log-path>

       is provided, use it.
     • If not, create a new one in the parent directory of the eval files:

       optimization-[timestamp].md

       .
   • Read content of the identified prompt file.

2. Optimization Loop (Max 10 iterations)

   • Execute (The Generator): Run

     agentv eval <eval-path>

     .
     • Targeted Run: If iterating on specific stubborn failures, use

       --test-id <test_id>

       to run only the relevant tests.
   • Analyze (The Reflector):
     • Locate the results file path from the console output (e.g.,

       .agentv/results/eval_...jsonl

       ).
     • Orchestrate Subagent: Use

       runSubagent

       to analyze the results.
       • Task: Read the results file, calculate pass rate, and perform root cause analysis.
       • Output: Return a structured analysis including:
         • Score: Current pass rate.
         • Root Cause: Why failures occurred (e.g., "Ambiguous definition", "Hallucination").
         • Insight: Key learning or pattern identified from the failures.
         • Strategy: High-level plan to fix the prompt (e.g., "Clarify section X", "Add negative constraint").
   • Decide:
     • If 100% pass: STOP and report success.
     • If Score decreased: Revert last change, try different approach.
     • If No improvement (2x): STOP and report stagnation.
   • Refine (The Curator):
     • Orchestrate Subagent: Use

       runSubagent

       to apply the fix.
       • Task: Read the relevant prompt file(s), apply the Strategy from the Reflector, and generate the log entry.
       • Output: The Log Entry describing the specific operation performed.

         markdown     ### Iteration [N]     - **Operation**: [ADD / UPDATE / DELETE]     - **Target**: [Section Name]     - **Change**: [Specific text added/modified]     - **Trigger**: [Specific failing test case or error pattern]     - **Rationale**: [From Reflector: Root Cause]     - **Score**: [From Reflector: Current Pass Rate]     - **Insight**: [From Reflector: Key Learning]     

     • Strategy: Treat the prompt as a structured set of rules. Execute atomic operations:
       • ADD: Insert a new rule if a constraint was missed.
       • UPDATE: Refine an existing rule to be clearer or more general.
         • Clarify: Make ambiguous instructions specific.
         • Generalize: Refactor specific fixes into high-level principles (First Principles).
       • DELETE: Remove obsolete, redundant, or harmful rules.
         • Prune: If a general rule covers specific cases, delete the specific ones.
       • Negative Constraint: If hallucinating, explicitly state what NOT to do. Prefer generalized prohibitions over specific forbidden tokens where possible.
       • Safety Check: Ensure new rules don't contradict existing ones (unless intended).
     • Constraint: Avoid rewriting large sections. Make surgical, additive changes to preserve existing behavior.
   • Log Result:
     • Append the Log Entry returned by the Curator to the optimization log file.

3. Completion

   • Report final score.
   • Summarize key changes made to the prompt.
   • Finalize Optimization Log: Add a summary header to the optimization log file indicating the session completion and final score.

Guidelines

• Generalization First: Prefer broad, principle-based guidelines over specific examples or "hotfixes". Only use specific rules if generalized instructions fail to achieve the desired score.
• Simplicity ("Less is More"): Avoid overfitting to the test set. If a specific rule doesn't significantly improve the score compared to a general one, choose the general one.
• Structure: Maintain existing Markdown headers/sections.
• Progressive Disclosure: If the prompt grows too large (>200 lines), consider moving specialized logic into a separate file or skill.
• Quality Criteria: Ensure the prompt defines a clear persona, specific task, and measurable success criteria.