Awesome-Agent-Skills-for-Empirical-Research election-data-source-countypres
git clone https://github.com/brycewang-stanford/Awesome-Agent-Skills-for-Empirical-Research
T=$(mktemp -d) && git clone --depth=1 https://github.com/brycewang-stanford/Awesome-Agent-Skills-for-Empirical-Research "$T" && mkdir -p ~/.claude/skills && cp -r "$T/skills/17-DAAF-Contribution-Community-daaf/dot-claude/skills/election-data-source-countypres" ~/.claude/skills/brycewang-stanford-awesome-agent-skills-for-empirical-research-election-data-sou && rm -rf "$T"
skills/17-DAAF-Contribution-Community-daaf/dot-claude/skills/election-data-source-countypres/SKILL.mdCounty Presidential Data Source Reference
County Presidential Election Returns 2000-2024 from MIT Election Data and Science Lab (MEDSL). Use when analyzing county-level presidential vote shares, party trends, turnout, or geographic voting patterns. Key join column county_fips enables linking to census, education (CCD/SAIPE), and demographic datasets. Requires Harvard Dataverse API key (HARVARD_DATAVERSE_API_KEY env var). Categorical variables use uppercase strings, not Portal integer codes. Critical caveat: naive mode='TOTAL' filtering silently drops ~1,000 counties in 2020+ data — use 3-pattern reconstruction.
The authoritative source for county-level U.S. presidential election returns spanning 2000-2024. Provides candidate-level vote counts across all 50 states and DC, enabling vote share analysis, partisan trend mapping, and cross-domain geographic research via FIPS code joins.
CRITICAL: Value Encoding
This dataset uses uppercase string codes for categorical variables (party, mode, candidate, state) rather than integer codes. Empty strings (
) appear as undocumented values in""(501 rows, 2024) andparty(2,795 rows, 2024).mode
Context party mode candidate Standard values ,DEMOCRATREPUBLICANTOTALBARACK OBAMAAggregate/meta values ,OTHER"" ,""ELECTION DAY ,OTHERUNDERVOTESSee
for complete encoding tables../references/variable-definitions.md
Prerequisites
API Key Required: This data source requires a Harvard Dataverse API key to fetch data. Unlike education data sources (which use the Urban Institute's free, unauthenticated API), election data is hosted on Harvard Dataverse and requires authentication.
Setup instructions:
- Create a free Harvard Dataverse account at https://dataverse.harvard.edu/
- Log in, navigate to your account name (top-right) → API Token
- Click "Create Token" and copy it
- Set the environment variable before launching Claude Code:
For Docker users: run this inside the container afterexport HARVARD_DATAVERSE_API_KEY="your_token_here"but beforedocker compose exec daaf-docker bash. To make it persistent across sessions, add it toclaude.~/.bashrcIf the key is missing, any fetch script will fail with a
. The orchestrator should check for this variable's existence before dispatching Stage 5 fetch tasks that use this data source.KeyError: 'HARVARD_DATAVERSE_API_KEY'
What is the MEDSL County Presidential Returns Dataset?
- Producer: MIT Election Data and Science Lab (MEDSL)
- Coverage: County-level presidential election returns, 50 states + DC
- Frequency: Every 4 years (presidential election cycle)
- Available years: 2000, 2004, 2008, 2012, 2016, 2020, 2024
- Primary identifier:
(5-digit FIPS code, stored as integer)county_fips - Record unit: One row per county-year-candidate-party-mode combination
- Total records: 94,151 rows x 12 columns (~8.4 MB). Note: rows per year vary dramatically — 2020/2024 have ~2x rows due to mode breakdowns (~22K vs ~9.5K for earlier years)
- Source: Harvard Dataverse (DOI: 10.7910/DVN/VOQCHQ)
Reference File Structure
| File | Purpose | When to Read |
|---|---|---|
| Complete column specs, party/mode/candidate value tables | Interpreting specific columns or coded values |
| All categorical value mappings with frequencies | Filtering or recoding party, mode, candidate |
| Detailed per-column profiling (types, nulls, ranges) | Understanding column characteristics |
| Known issues, anomalies, duplicates, null patterns | Assessing data reliability |
| 3-pattern TOTAL mode reconstruction for 2020+ data | Cleaning any 2020+ analysis (CRITICAL) |
| Preliminary semantic interpretations (flagged for review) | Understanding column meanings |
Decision Trees
What analysis do I need?
Analyzing presidential election data? ├─ County-level vote shares → Use 3-pattern mode reconstruction (./references/mode-reconstruction.md) │ └─ Longitudinal (cross-year) → MUST reconstruct TOTAL for 2020+ (naive filter drops ~1,000 counties) │ └─ Single year (pre-2020) → Safe to filter mode='TOTAL' │ └─ Single year (2020/2024) → Reconstruct unless analyzing a known TOTAL-only state ├─ Party trends → Group by year + party, use party column (not candidate name) │ └─ Third parties → See ./references/coded-values.md (GREEN/LIBERTARIAN vary by year) ├─ Turnout analysis → Use totalvotes column (dedup per county-year before summing!) ├─ Joining with other data → Use county_fips as join key (zero-pad to 5 chars first!) │ └─ Census/ACS data → Join on county_fips (standard 5-digit string) │ └─ Education data (CCD/SAIPE) → Join on county_fips │ └─ Null FIPS? → See ./references/quality-notes.md (CT, ME, RI) └─ Voting method analysis → 2020 and 2024 only, see mode breakdown
Is this a data quality issue?
Unexpected values? ├─ county_fips is null → CT/ME/RI in pre-2020 years (52 rows) ├─ county_fips > 72999 → Kansas City MO (FIPS 2938000, non-standard) ├─ county_fips join failures → Zero-pad to 5 chars! (AR codes = 4 digits as int) ├─ AR FIPS 5135 has two counties → Source data error: St. Francis under Sharp County │ └─ See ./references/quality-notes.md #arkansas-fips-contamination (BLOCKER) ├─ CT counties missing from shapefile → 2022+ TIGER uses planning regions, not counties │ └─ See ./references/quality-notes.md #connecticut-fips-geography-mismatch ├─ ~1,000 counties missing after mode filter → Use 3-pattern reconstruction, not naive filter │ └─ See ./references/mode-reconstruction.md ├─ candidate is not a person → UNDERVOTES/OVERVOTES/SPOILED/TOTAL VOTES CAST │ └─ Filter these OUT for candidate-level analysis ├─ party is empty string → 501 rows in 2024, undocumented ├─ mode is empty string → 2,795 rows in 2024 (may be totals OR breakdowns per state) ├─ candidatevotes is null → 37 rows (NM 2024, mode breakdown) ├─ sum(candidatevotes) > totalvotes → 49 county-years (minor rounding) ├─ Duplicate rows → 83 exact duplicates exist └─ Alaska 2004 → District-level data, not county; FIPS = 2001-2099 └─ See ./references/quality-notes.md #alaska-2004
Quick Reference: Election Variables
Party Values by Year
| Year | DEMOCRAT | REPUBLICAN | LIBERTARIAN | GREEN | OTHER | |
|---|---|---|---|---|---|---|
| 2000 | Y | Y | - | Y | Y | - |
| 2004-2016 | Y | Y | - | - | Y | - |
| 2020 | Y | Y | Y | Y | Y | - |
| 2024 | Y | Y | Y | - | Y | Y |
Mode Values by Year
| Year | Modes Available |
|---|---|
| 2000-2016 | only |
| 2020 | + 15 breakdown modes (11 states) |
| 2024 | + 9 breakdown modes + (varies by state) |
For complete mode values see
./references/coded-values.md.
Key Identifiers
| ID | Format | Level | Example | Notes |
|---|---|---|---|---|
| Int64 (5-digit) | County | (LA County, CA) | 52 nulls (CT/ME/RI); join key for census/education data |
| String (2-char) | State | | USPS abbreviation; 1:1 with |
| String | State | | Full uppercase name |
WARNING: FIPS Zero-Padding Required for Joins
is stored as Int64. When converting to string for joins with Census, SAIPE, CCD, or other datasets, zero-pad to 5 characters or Arkansas and other small-FIPS states will produce 4-digit codes that fail to match.county_fipsdf = df.with_columns(pl.col("county_fips").cast(pl.Utf8).str.zfill(5).alias("county_fips_str"))
Missing Data Codes
| Code | Column(s) | Meaning | Frequency |
|---|---|---|---|
| | FIPS not assigned | 52 rows (CT, ME, RI pre-2020) |
| | Vote count unavailable | 37 rows (NM 2024 mode breakdowns) |
| | No votes recorded | 50 rows |
| | Zero votes for candidate | 3,908 rows (4.15%) |
| | Party not specified | 501 rows (2024 only) |
| | Mode not specified | 2,795 rows (2024 only) |
Non-Candidate Entries in candidate Column
| Value | Rows | Meaning |
|---|---|---|
| 27,548 | Aggregate of minor candidates |
| 427 | County total (redundant with totalvotes) |
| 402 | Ballots with no presidential selection |
| 380 | Ballots with multiple presidential selections |
| 14 | Invalidated ballots |
Filter these out for candidate-level vote share analysis.
Best Practice: Party-Based Identification
Always use
column for identifying party affiliation, neverpartyname. Candidate names are inconsistent across years (e.g., "DONALD TRUMP" in 2016 vs "DONALD J TRUMP" in 2020/2024). Thecandidatecolumn (party,DEMOCRAT) is stable across all years.REPUBLICAN# CORRECT — stable across years dem = df.filter(pl.col("party") == "DEMOCRAT") # WRONG — misses 2016 or 2020/2024 depending on which name you use trump = df.filter(pl.col("candidate") == "DONALD TRUMP")
Data Access
Dataset Paths
| Topic | Type | Path |
|---|---|---|
| County presidential returns | Single file | Harvard Dataverse DOI: |
| Codebook | Single file | Bundled: |
| Sources per state | Single file | Bundled: |
Codebooks
| Dataset | Codebook Path |
|---|---|
| County presidential returns 2000-2024 | (bundled in Dataverse) |
Codebook is a Markdown file bundled in the Harvard Dataverse deposit. For human reference. The QA methodology paper is at: https://www.nature.com/articles/s41597-022-01745-0
Truth Hierarchy: When interpreting variable values, apply this priority:
- Actual data file (what you observe in the TSV) -- this IS the truth
- Live codebook (Markdown file in Dataverse) -- authoritative documentation, may lag
- This skill documentation -- convenient summary, may drift from codebook
If this documentation contradicts the codebook, trust the codebook. If the codebook contradicts observed data, trust the data and investigate.
Example Fetch
# Fetch from Harvard Dataverse API import os, requests, polars as pl, io api_key = os.environ["HARVARD_DATAVERSE_API_KEY"] # Get file ID from dataset metadata first, then download # File: countypres_2000-2024.tab (TSV format) file_url = "https://dataverse.harvard.edu/api/access/datafile/{file_id}" r = requests.get(file_url, params={"key": api_key, "format": "original"}) df = pl.read_csv(io.BytesIO(r.content), separator='\t') # Filter to California, 2020, TOTAL mode only ca_2020 = df.filter( (pl.col("state_po") == "CA") & (pl.col("year") == 2020) & (pl.col("mode") == "TOTAL") )
Filtering
# Common filter patterns for county presidential data # 1. Cross-year analysis: WARNING — naive filter drops ~1,000 counties in 2020+! # Use 3-pattern mode reconstruction instead. See ./references/mode-reconstruction.md # The single-line filter below is ONLY safe for single-year analysis on a state # known to have TOTAL rows (e.g., 2000-2016 data, or a confirmed TOTAL-only state). longitudinal = df.filter(pl.col("mode") == "TOTAL") # UNSAFE for 2020+ multi-state! # 2. Remove non-candidate rows (UNDERVOTES, OVERVOTES, etc.) candidates_only = df.filter( ~pl.col("candidate").is_in(["TOTAL VOTES CAST", "UNDERVOTES", "OVERVOTES", "SPOILED"]) ) # 3. Major party analysis — RECOMMENDED: use party column, not candidate name # Candidate names change across years (e.g., "DONALD TRUMP" vs "DONALD J TRUMP") two_party = df.filter(pl.col("party").is_in(["DEMOCRAT", "REPUBLICAN"])) # 4. Exclude Alaska 2004 anomaly clean = df.filter(~((pl.col("state_po") == "AK") & (pl.col("year") == 2004))) # 5. Exclude rows with null county_fips (for join operations) joinable = df.filter(pl.col("county_fips").is_not_null())
Common Pitfalls
| Pitfall | Issue | Solution |
|---|---|---|
| Cross-year mode mismatch | 2000-2016 has only ; 2020/2024 have breakdowns. Mixing modes inflates counts | Always filter for longitudinal analysis |
| Non-candidate rows | , , , appear as "candidates" | Filter out before computing candidate vote shares |
| Alaska 2004 | District-level data with non-standard FIPS codes (2001-2099); vote counts overstated | Exclude AK 2004 or handle separately; do not join on county_fips |
| Null FIPS for joins | CT, ME, RI have null in some years; breaks joins | Use + as fallback join key |
| Kansas City MO FIPS | Uses non-standard FIPS 2938000 (not a real county FIPS) | Handle as special case in joins; Kansas City is an independent city |
| Empty string party/mode | 2024 has in party (501 rows) and mode (2,795 rows) | Treat as missing/undocumented; filter or investigate by state |
| Vote share > 100% | 49 county-years where sum(candidatevotes) > totalvotes | Minor rounding; use caution with strict validation |
| Duplicate rows | 83 exact duplicate rows exist in dataset | Deduplicate before analysis |
| FIPS zero-padding | Int64 → string without padding gives 4-digit codes for AR and other small-FIPS states | before joins |
| Name-based party ID | Candidate names change across years ("DONALD TRUMP" vs "DONALD J TRUMP") | Always use column, never name |
| CT FIPS mismatch | 2022+ Census TIGER uses CT planning regions, not legacy counties | Use 2020-vintage TIGER shapefiles for CT county joins |
| totalvotes duplication | is repeated per candidate row; summing without dedup inflates by ~3-4x | Deduplicate to one row per (county_fips, year) before aggregating |
Critical: Mode Column Behavior Change
WARNING: Naive
filtering drops ~1,000 counties in 2020+ data. Multiple states report ONLY mode breakdowns (no TOTAL rows). A simple filter silently removes all their counties. Use 3-pattern mode reconstruction instead. Seemode == "TOTAL"for the full code pattern and validation../references/mode-reconstruction.md
The
mode column behavior changed significantly starting in 2020:
- 2000-2016: All rows have
(aggregate county totals only)mode = 'TOTAL' - 2020: 11 states report by voting method alongside TOTAL; 10 states have ONLY breakdowns (AR, AZ, GA, IA, KY, MD, NC, OK, SC, VA)
- 2024: Additional breakdown states; some have empty string
(which may represent totals OR breakdowns depending on the state)mode
For any cross-year or multi-county analysis, use 3-pattern mode reconstruction:
- Pattern 1: TOTAL present → keep TOTAL, drop breakdowns
- Pattern 2: Only breakdowns, no TOTAL → sum
across modescandidatevotes - Pattern 3: Empty-string mode = totals → reclassify after per-state verification
Empty-string detection must be per-state — NC 2024 empty-string rows are breakdowns (multiple per county-candidate), while other states' empty-string rows are totals (one per county-candidate). See
./references/mode-reconstruction.md for detection logic and code.
Related Data Sources
| Source | Relationship | When to Use |
|---|---|---|
| Census/ACS | Join via | County demographics, population, income |
SAIPE () | Join via | County poverty estimates (cross-domain) |
CCD () | Join via | School district data (cross-domain) |
| MEDSL Precinct Returns | Finer geographic resolution | When county-level is insufficient |
| MEDSL Senate/House Returns | Same producer, different office | When analyzing down-ballot races |
Note: This is the first election domain dataset in DAAF. Cross-domain joins with education data are possible via
county_fips. No election-specific explorer or query skills exist yet.
Topic Index
| Topic | Reference File |
|---|---|
| Column specifications | |
| Column types and ranges | |
| Party values and year coverage | |
| Mode values and year behavior | |
| Candidate name mapping | |
| Non-candidate entries | |
| Complete encoding tables | |
| Null county_fips patterns | |
| Alaska 2004 anomaly | |
| Kansas City MO FIPS | |
| Duplicate rows | |
| Null candidatevotes | |
| Per-state data sources | |
| Missing votes flags | |
| Mode reconstruction (3-pattern) | |
| States without TOTAL rows | |
| Empty-string mode detection | |
| Row count estimation by year | |
| FIPS zero-padding for joins | |
| AR FIPS contamination (Sharp/St. Francis) | |
| CT geography mismatch (2022+) | |
| Party-based identification (best practice) | |
| totalvotes deduplication | |
| Preliminary interpretations | |
| Data profiling scripts | |