Claude-skill-registry-data mlflow-evaluation
MLflow 3 GenAI evaluation for agent development. Use when (1) writing mlflow.genai.evaluate() code, (2) creating @scorer functions, (3) building evaluation datasets from traces, (4) using built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), (5) analyzing traces for latency/errors/architecture, (6) optimizing agent context/prompts/token usage, (7) debugging evaluation failures. Covers the full eval workflow: trace analysis -> dataset building -> scorer creation -> evaluation execution.
git clone https://github.com/majiayu000/claude-skill-registry-data
T=$(mktemp -d) && git clone --depth=1 https://github.com/majiayu000/claude-skill-registry-data "$T" && mkdir -p ~/.claude/skills && cp -r "$T/data/mlflow-evaluation" ~/.claude/skills/majiayu000-claude-skill-registry-data-mlflow-evaluation && rm -rf "$T"
data/mlflow-evaluation/SKILL.mdMLflow 3 GenAI Evaluation
Before Writing Any Code
- Read GOTCHAS.md - 15+ common mistakes that cause failures
- Read CRITICAL-interfaces.md - Exact API signatures and data schemas
End-to-End Workflows
Follow these workflows based on your goal. Each step indicates which reference files to read.
Workflow 1: First-Time Evaluation Setup
For users new to MLflow GenAI evaluation or setting up evaluation for a new agent.
| Step | Action | Reference Files |
|---|---|---|
| 1 | Understand what to evaluate | |
| 2 | Learn API patterns | |
| 3 | Build initial dataset | |
| 4 | Choose/create scorers | |
| 5 | Run evaluation | |
Workflow 2: Production Trace -> Evaluation Dataset
For building evaluation datasets from production traces.
| Step | Action | Reference Files |
|---|---|---|
| 1 | Search and filter traces | |
| 2 | Analyze trace quality | |
| 3 | Tag traces for inclusion | |
| 4 | Build dataset from traces | |
| 5 | Add expectations/ground truth | |
Workflow 3: Performance Optimization
For debugging slow or expensive agent execution.
| Step | Action | Reference Files |
|---|---|---|
| 1 | Profile latency by span | |
| 2 | Analyze token usage | |
| 3 | Detect context issues | |
| 4 | Apply optimizations | |
| 5 | Re-evaluate to measure impact | |
Workflow 4: Regression Detection
For comparing agent versions and finding regressions.
| Step | Action | Reference Files |
|---|---|---|
| 1 | Establish baseline | |
| 2 | Run current version | |
| 3 | Compare metrics | |
| 4 | Analyze failing traces | |
| 5 | Debug specific failures | |
Workflow 5: Custom Scorer Development
For creating project-specific evaluation metrics.
| Step | Action | Reference Files |
|---|---|---|
| 1 | Understand scorer interface | |
| 2 | Choose scorer pattern | |
| 3 | For multi-agent scorers | |
| 4 | Test with evaluation | |
Reference Files Quick Lookup
| Reference | Purpose | When to Read |
|---|---|---|
| Common mistakes | Always read first before writing code |
| API signatures, schemas | When writing any evaluation code |
| Running evals, comparing | When executing evaluations |
| Custom scorer creation | When built-in scorers aren't enough |
| Dataset building | When preparing evaluation data |
| Trace debugging | When analyzing agent behavior |
| Token/latency fixes | When agent is slow or expensive |
| High-level workflows | When starting a new evaluation project |
Critical API Facts
- Use:
(NOTmlflow.genai.evaluate()
)mlflow.evaluate() - Data format:
(nested structure required){"inputs": {"query": "..."}} - predict_fn: Receives
(not a dict)**unpacked kwargs
See
GOTCHAS.md