Experiment Design Expert

Overview

Design fast, low-cost experiments to validate product hypotheses before committing to full development. This skill applies Alberto Savoia's pretotyping philosophy ("Make sure you are building The Right It before you build It right") alongside lean experimentation methods for both new and existing products.

When to Use

• You have a product idea or feature hypothesis and need to validate it cheaply.
• You want to test willingness to pay or genuine user interest, not just stated preference.
• You need to choose the right experiment method for your context (new vs. existing product).

Core Principles

1. XYZ Hypothesis Format

Every experiment starts with a falsifiable hypothesis:

"At least X% of Y will do Z."

A good XYZ hypothesis is specific, measurable, and has a clear pass/fail threshold set before the experiment runs.

2. Skin-in-the-Game (SITG)

Stated interest is unreliable. Valid experiments measure actions that require commitment:

• Money -- Pre-orders, deposits, paid waitlists.
• Time -- Signing up, completing a multi-step flow, scheduling a demo.
• Reputation -- Sharing with colleagues, posting publicly.

Always prefer SITG signals over surveys, likes, or verbal feedback.

3. Your Own Data (YODA)

Do not rely on market reports, competitor benchmarks, or industry averages. Run your own experiment with your own audience to get Your Own Data. Others' data reflects their context, not yours.

Experiment Types

For New Products

For Existing Products

Methodology

Step 1: Write the XYZ Hypothesis

Start with the assumption you need to test. Convert it into XYZ format.

Weak: "Users will like the new dashboard." Strong: "At least 30% of active users who see the new dashboard will set it as their default view within 5 days."

Step 2: Select the Experiment Method

Choose based on:

• Product type (new vs. existing)
• What you are testing (demand, usability, willingness to pay, engagement)
• Available effort (team capacity and timeline)
• Required confidence (directional signal vs. statistically significant result)

Step 3: Define the Metric and Threshold

Step 4: Run the Experiment

• Set a timebox. Every experiment has a fixed end date.
• Do not peek. Avoid checking results daily and making early calls.
• Document everything. Record setup, audience, duration, and any anomalies.

Step 5: Evaluate Results

Python Tool: experiment_designer.py

Design experiments from hypotheses using the CLI tool:

# Run with demo data
python3 scripts/experiment_designer.py --demo

# Run with custom input
python3 scripts/experiment_designer.py input.json

# Output as JSON
python3 scripts/experiment_designer.py input.json --format json

Input Format

{
  "hypotheses": [
    {
      "hypothesis_text": "At least 20% of trial users will click Upgrade within 7 days",
      "target_segment": "trial users on free plan",
      "product_type": "existing"
    }
  ]
}

Output

For each hypothesis, the tool suggests 2-3 experiment designs with method, metric, success threshold, effort level, and duration estimate.

See

scripts/experiment_designer.py

Output Template

Use

assets/experiment_plan_template.md

• Experiment card with hypothesis, method, metric, threshold, owner, timeline
• Experiment tracker for managing multiple concurrent experiments
• Results documentation for recording outcomes and decisions

Integration with Other Discovery Skills

• Use

  brainstorm-ideas/

  to generate ideas that become hypotheses.
• Use

  identify-assumptions/

  to find the riskiest assumptions to test.
• After experiments, use

  pre-mortem/

  before committing to full build.

Troubleshooting

product_type

hypotheses

hypotheses

hypothesis_text

target_segment

product_type

identify-assumptions/

execution/outcome-roadmap/

Success Criteria

• Every product hypothesis has a falsifiable XYZ statement before experiment design begins
• Experiments measure Skin-in-the-Game (SITG) signals, not stated preferences
• Pass/fail thresholds are defined before the experiment runs, not after
• Experiment duration does not exceed 4 weeks for any single hypothesis
• At least 70% of experiments produce a clear pass or fail verdict (not inconclusive)
• Results directly feed the build/pivot/abandon decision within 1 week of experiment completion
• Your Own Data (YODA) principle is followed -- no reliance on industry benchmarks for go/no-go decisions

Scope & Limitations

In Scope:

• XYZ hypothesis formulation and validation for product ideas
• Experiment method selection for both new products (landing page, pre-order, concierge, explainer video) and existing products (fake door, feature stub, A/B test, Wizard of Oz, in-app survey)
• Automated experiment design suggestions based on hypothesis keyword analysis
• Metric selection, success threshold definition, and effort/duration estimation

Out of Scope:

• Statistical power analysis or sample size calculation (use dedicated A/B test platforms)
• Experiment infrastructure setup (feature flags, analytics instrumentation)
• Running the actual experiment (this skill designs experiments, not executes them)
• Long-term product strategy or roadmap decisions (see

  execution/outcome-roadmap/

  )

Important Caveats:

• Pretotyping is for validating demand and value, not for measuring usability or performance.
• In-app surveys are the weakest SITG signal. Use them only when behavioral experiments are impractical.
• The tool's keyword-to-signal matching is heuristic-based. Review suggested experiments and override when domain knowledge dictates a better method.

Integration Points

brainstorm-ideas/

identify-assumptions/

pre-mortem/

execution/create-prd/

execution/brainstorm-okrs/

execution/outcome-roadmap/

Tool Reference

experiment_designer.py

Suggests 2-3 experiment designs for each product hypothesis based on keyword signal analysis.

input_file

--demo

--format

text

text

json

References

• Alberto Savoia, The Right It (2019)
• Eric Ries, The Lean Startup (2011)
• Jeff Gothelf & Josh Seiden, Lean UX (2013)
• Teresa Torres, Continuous Discovery Habits (2021)