Questionnaire Design Guide

Design valid and reliable survey instruments with proper question types, Likert scale construction, response coding, and data preparation for analysis.

Survey Design Principles

Question Types

Type	Example	Best For	Analysis
Likert scale	"Rate your agreement: 1-5"	Attitudes, perceptions	Ordinal/interval statistics
Multiple choice	"Select your field"	Demographics, categories	Frequencies, chi-square
Ranking	"Rank these 5 options"	Preferences, priorities	Rank correlations
Open-ended	"Describe your experience"	Exploratory, rich data	Qualitative coding
Matrix/grid	Multiple items, same scale	Efficient battery of items	Factor analysis, reliability
Slider/VAS	0-100 visual analog scale	Continuous measures	Parametric statistics
Semantic differential	"Easy __ __ __ __ __ Difficult"	Bipolar attitudes	Factor analysis

The Four C's of Good Questions

1. Clear: Avoid jargon, double-barreled questions, and ambiguity
2. Concise: Keep questions short (ideally under 20 words)
3. Complete: Include all relevant response options
4. Consistent: Use the same scale direction and format throughout

Likert Scale Design

Scale Points

Points	Scale Example	Recommended Use
4-point	Strongly Disagree to Strongly Agree	Forces choice (no neutral), less discriminating
5-point	SD, D, Neutral, A, SA	Most common, good balance of simplicity and discrimination
7-point	SD, D, Somewhat D, Neutral, Somewhat A, A, SA	More discriminating, better for experienced respondents
11-point (0-10)	Not at all to Completely	NPS, continuous-like measures

Anchoring Labels

5-Point Agreement Scale:
1 = Strongly Disagree
2 = Disagree
3 = Neither Agree nor Disagree
4 = Agree
5 = Strongly Agree

5-Point Frequency Scale:
1 = Never
2 = Rarely
3 = Sometimes
4 = Often
5 = Always

5-Point Satisfaction Scale:
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neutral
4 = Satisfied
5 = Very Satisfied

Reverse-Coded Items

Include 2-3 reverse-coded items per construct to detect acquiescence bias:

Regular:  "I find research methods interesting."        (1-5: SD to SA)
Reversed: "I find research methods tedious and dull."   (1-5: SD to SA)

# Recode reversed items before analysis:
# reversed_score = (max_scale + 1) - raw_score
# For a 5-point scale: reversed_score = 6 - raw_score

Constructing a Multi-Item Scale

Step-by-Step Process

1. Define the construct: Write a clear conceptual definition
2. Generate items: Write 1.5-2x the number of items you plan to keep (e.g., write 15 items for an 8-item scale)
3. Expert review: Have 3-5 experts rate each item for relevance (Content Validity Index)
4. Pilot test: Administer to 30-50 respondents
5. Item analysis: Calculate item-total correlations, check reliability
6. Exploratory Factor Analysis (EFA): Confirm dimensionality
7. Finalize scale: Remove weak items, re-test reliability

Example: Research Self-Efficacy Scale

Construct: Belief in one's ability to conduct academic research

Items (5-point Likert, Strongly Disagree to Strongly Agree):
RSE1: I can formulate clear research questions.
RSE2: I can design an appropriate research methodology.
RSE3: I can analyze data using statistical software.
RSE4: I can write a publishable research paper.
RSE5: I can critically evaluate published research.
RSE6: I can present research findings at a conference.
RSE7R: I struggle to interpret statistical results. [REVERSED]
RSE8R: I find it difficult to synthesize literature. [REVERSED]

Data Coding and Preparation

Coding Scheme

import pandas as pd
import numpy as np

# Define coding scheme
likert_coding = {
    "Strongly Disagree": 1,
    "Disagree": 2,
    "Neither Agree nor Disagree": 3,
    "Agree": 4,
    "Strongly Agree": 5
}

# Apply coding
df["Q1_coded"] = df["Q1_raw"].map(likert_coding)

# Reverse code specific items
reverse_items = ["RSE7R", "RSE8R"]
max_scale = 5
for item in reverse_items:
    df[f"{item}_recoded"] = (max_scale + 1) - df[item]

# Calculate composite score (mean of items)
scale_items = ["RSE1", "RSE2", "RSE3", "RSE4", "RSE5", "RSE6",
               "RSE7R_recoded", "RSE8R_recoded"]
df["RSE_mean"] = df[scale_items].mean(axis=1)

Missing Data Handling

# Check missing data patterns
print(df[scale_items].isnull().sum())
print(f"Complete cases: {df[scale_items].dropna().shape[0]} / {df.shape[0]}")

# Common strategies:
# 1. Listwise deletion (if < 5% missing)
df_complete = df.dropna(subset=scale_items)

# 2. Mean imputation per item (simple but biased)
df[scale_items] = df[scale_items].fillna(df[scale_items].mean())

# 3. Person-mean imputation (if < 20% of items missing per person)
def person_mean_impute(row, items, max_missing=2):
    if row[items].isnull().sum() <= max_missing:
        return row[items].fillna(row[items].mean())
    return row[items]  # leave as NaN if too many missing

df[scale_items] = df.apply(lambda r: person_mean_impute(r, scale_items), axis=1)

Reliability Analysis

Cronbach's Alpha

import pingouin as pg

# Calculate Cronbach's alpha
alpha = pg.cronbach_alpha(df[scale_items])
print(f"Cronbach's alpha: {alpha[0]:.3f}")
# Interpretation: >= 0.70 acceptable, >= 0.80 good, >= 0.90 excellent

library(psych)

# Cronbach's alpha with item-level diagnostics
alpha_result <- alpha(data[, scale_items])
print(alpha_result)
# Check "raw_alpha if item dropped" to identify weak items

Item-Total Correlations

# Corrected item-total correlations (should be > 0.30)
item_stats <- alpha_result$item.stats
print(item_stats[, c("r.drop", "raw.alpha")])
# r.drop < 0.30: consider removing the item
# raw.alpha increases if dropped: item is weakening the scale

Validity Assessment

Validity Type	Method	Criterion
Content validity	Expert panel rating (CVI)	I-CVI >= 0.78, S-CVI/Ave >= 0.90
Construct validity	Exploratory Factor Analysis (EFA)	Eigenvalue > 1, loadings > 0.40
Convergent validity	Correlation with related construct	r > 0.30
Discriminant validity	Correlation with unrelated construct	r < 0.30
Criterion validity	Correlation with external criterion	Significant correlation
Test-retest reliability	ICC or Pearson r over 2-4 weeks	ICC > 0.70

Common Design Mistakes

Mistake	Example	Fix
Double-barreled question	"This course is interesting and useful"	Split into two separate items
Leading question	"Don't you agree that X is important?"	"How important is X to you?"
Absolute terms	"Do you always check citations?"	"How often do you check citations?"
Missing option	No "Not Applicable" when needed	Add N/A option or filter logic
Inconsistent scale direction	Some items 1=good, others 1=bad	Standardize direction; clearly mark reversed items
Too many items	100-item survey	Aim for 5-8 items per construct, 15-30 min total
No pilot test	Skip straight to full deployment	Always pilot with 30-50 respondents

Survey Platform Comparison

Platform	Cost	Features	Best For
Qualtrics	Institutional	Advanced logic, panels, API	Large academic studies
SurveyMonkey	Freemium	Easy to use, basic analysis	Quick surveys
Google Forms	Free	Simple, integrates with Sheets	Classroom, pilot testing
LimeSurvey	Free/self-hosted	Open source, full control	Privacy-sensitive research
REDCap	Free (academic)	Clinical data, HIPAA compliant	Medical/clinical research
Prolific	Per-response	Participant recruitment	Online experiments