Skill: Question Framing

Purpose

Structure analytical questions using the Question Ladder framework so every analysis starts with a clear decision context, measurable success criteria, and testable hypotheses.

When to Use

Apply this skill when starting any new analysis, when a user asks a vague question ("How are we doing?"), or when an analysis request lacks decision context. Always frame before analyzing.

Instructions

Pre-flight: Load Learnings

Before executing, check

.knowledge/learnings/index.md

for relevant entries:

• Read the file. If it doesn't exist or is empty, skip silently.
• Scan for entries under "Question Framing" and "General" headings (or related categories like "Business Context", "Methodology Notes").
• If entries exist, incorporate them as constraints or context for this execution.
• Never block execution if learnings are unavailable.

The Question Ladder

Every analytical question climbs four rungs:

GOAL        → What business outcome are we trying to achieve?
DECISION    → What specific decision will this analysis inform?
METRIC      → What will we measure to inform that decision?
HYPOTHESIS  → What do we expect to find, and why?

The rule: Never start analyzing data until you can state all four rungs. If the requester only gives you a goal ("improve retention"), your first job is to climb the ladder before touching data.

Framing Process

Step 1: Extract the decision Ask: "What will you DO differently based on the answer?"

• If the answer is "nothing" or "I'm just curious" → this is reporting, not analysis. Redirect to a dashboard or quick stat.
• If the answer is a specific action → you have a decision. Proceed.

Step 2: Define success criteria Ask: "How will you know the analysis answered your question?"

• The answer should be specific: "If conversion rate dropped >10% in segment X, we'll prioritize a fix"
• Not vague: "We'll understand our users better"

Step 3: Form testable hypotheses Ask: "What do you think is happening, and why?"

• Good: "I think mobile conversion dropped because the checkout redesign broke on small screens"
• Bad: "I think things are bad"

Step 4: Identify data requirements Ask: "What data do we need, and do we have it?"

• Map each hypothesis to specific metrics, segments, and time ranges
• Flag gaps early (see Tracking Gap Identification skill)

Good vs. Bad Questions

Bad Question	Problem	Good Question
"How are our users doing?"	No decision context, unmeasurable	"Did the onboarding redesign improve Day-7 retention for new users?"
"Analyze our funnel"	No hypothesis, no scope	"Where in the signup-to-purchase funnel are we losing the most users, and does it differ by acquisition channel?"
"What's our conversion rate?"	Reporting, not analysis	"Why did conversion rate drop 15% in March, and is it affecting all segments equally?"
"Tell me about churn"	Too broad, no decision	"Which user segments have the highest 90-day churn rate, and what behaviors predict churn in the first 30 days?"
"Is our product doing well?"	Unmeasurable, no comparison	"How does our monthly active user growth compare to Q3, and which features are driving engagement?"

Impact × Feasibility Prioritization

When multiple questions emerge, prioritize:

                    HIGH IMPACT
                        │
          ┌─────────────┼─────────────┐
          │   DO FIRST   │   PLAN FOR  │
          │  (Quick win)  │  (Strategic) │
HIGH      │               │              │
FEASIBILITY ──────────────┼──────────────── LOW
          │               │              │ FEASIBILITY
          │   DO IF TIME  │    SKIP     │
          │  (Nice to have)│  (Not worth) │
          └─────────────┼─────────────┘
                        │
                    LOW IMPACT

Impact criteria:

• Revenue/cost implication >$100K → High
• Affects >10% of users → High
• Informs a decision being made this quarter → High
• Curiosity-driven, no pending decision → Low

Feasibility criteria:

• Data exists and is clean → High
• Can be answered in <4 hours → High
• Requires new instrumentation → Low
• Requires data from another team → Low

Output Format: Question Brief

# Question Brief: [Title]
## Date: [YYYY-MM-DD]

### Business Context
[2-3 sentences: what's happening, why this matters now]

### The Question Ladder
| Rung | Statement |
|------|-----------|
| **Goal** | [Business outcome] |
| **Decision** | [Specific action this informs] |
| **Metric** | [What we'll measure] |
| **Hypothesis** | [What we expect to find and why] |

### Success Criteria
[How we'll know the analysis answered the question — specific thresholds or conditions]

### Data Requirements
| Data Needed | Source | Available? | Notes |
|-------------|--------|-----------|-------|
| [metric/field] | [table/system] | Yes/No/Partial | [gaps, caveats] |

### Scope
- **Time range:** [specific dates]
- **Segments:** [which user groups, geographies, platforms]
- **Exclusions:** [what we're intentionally leaving out and why]

### Priority
- **Impact:** [High/Medium/Low — with justification]
- **Feasibility:** [High/Medium/Low — with justification]
- **Recommendation:** [Do First / Plan For / Do If Time / Skip]

Examples

Example 1: Vague → Well-framed

Incoming request: "Can you look at our signup numbers?"

Reframed:

Rung	Statement
Goal	Increase new user signups by 20% in Q1
Decision	Should we invest in fixing the mobile signup flow or increasing top-of-funnel traffic?
Metric	Signup completion rate by device type + traffic source conversion rate
Hypothesis	Mobile signup completion rate is <50% of desktop because the form doesn't render properly on small screens. Fixing mobile is higher ROI than more traffic.

Example 2: Curiosity → Decision-driven

Incoming request: "I'm curious about our power users"

Reframed:

Rung	Statement
Goal	Increase the percentage of users who become power users (>10 sessions/month)
Decision	Which onboarding interventions should we prioritize to convert casual → power users?
Metric	Behaviors in first 7 days that predict power user status at Day 30
Hypothesis	Users who complete the tutorial AND create a project in their first session are 3x more likely to become power users. The tutorial completion rate is only 23%.

Example 3: Broad → Scoped

Incoming request: "Analyze our churn"

Reframed:

Rung	Statement
Goal	Reduce 90-day churn from 35% to 25%
Decision	Which segment's churn should we tackle first — low-engagement users or users who hit a specific friction point?
Metric	90-day churn rate by: (a) engagement tier in first 30 days, (b) last feature used before churning
Hypothesis	Users who never use Feature X churn at 2x the rate of users who do. Feature X has a discoverability problem, not a value problem.

Anti-Patterns

1. Never start analyzing before framing — "just pulling some numbers" without a question leads to interesting-but-useless findings
2. Never accept "just curious" as the decision — push for "what would you do differently?" If the answer is truly nothing, redirect to a dashboard
3. Never frame questions with implied answers — "Can you prove that Feature X works?" is not a question, it's confirmation bias. Reframe as "What is the impact of Feature X on [metric]?"
4. Never frame questions too broadly — "How are we doing?" needs scoping. What metric? What time range? Compared to what?
5. Never skip the hypothesis — hypotheses prevent fishing expeditions and give you something specific to test