Download All Transcripts

Why? Manually downloading transcripts folder-by-folder is tedious and error-prone. This skill automates overnight batch processing across all channels and collections with built-in rate limiting and resumability.

Quick Start

# Run from repository root - handles everything automatically
./scripts/download_all_transcripts.sh

That's it. The script finds all folders with

videos.csv

Workflow

1. Verify Prerequisites

Before running, ensure:

• You're in the repository root directory
• The

  data/

  folder contains at least one subfolder with a

  videos.csv

  file
• The

  transcript-download

  CLI is installed (comes with the project's Python package)

# Check for valid data folders
ls data/*/videos.csv

[!TIP] If no

videos.csv

files exist, first run

extract-videos

or

sync-all-channels

to populate them.

2. Execute Batch Download

./scripts/download_all_transcripts.sh

The script will:

1. Find all folders in

   data/

   containing

   videos.csv

2. Process each folder sequentially
3. Download transcripts to

   <folder>/transcripts/

4. Wait 60 seconds between videos to avoid YouTube rate limiting
5. Update CSV with download status

[!CAUTION] This is a long-running operation. For a channel with 500 videos, expect 8+ hours. Run overnight or in a

tmux

/

screen

session.

3. Monitor Progress

The script outputs real-time progress:

📝 YTScribe - Download All Transcripts
=======================================
Started at: Thu Dec 26 09:00:00 PST 2024
Delay between videos: 60s

Found 12 folders with videos.csv

────────────────────────────────────────
[1/12] Processing: lex-fridman
  CSV: /path/to/data/lex-fridman/videos.csv
  Output: /path/to/data/lex-fridman/transcripts

4. Handle Completion or Interruption

On successful completion:

✅ All transcripts downloaded!
Finished at: Thu Dec 26 17:30:00 PST 2024

Summary of folders processed:
  - lex-fridman: 342 transcripts
  - huberman-lab: 156 transcripts
  ...

On interruption or IP block: Simply run the script again. It automatically skips videos where

transcript_downloaded=True

Output Structure

Transcripts are saved as markdown with YAML frontmatter:

data/huberman-lab/
├── videos.csv
└── transcripts/
    ├── 2024-01-15-abc123.md
    ├── 2024-01-20-def456.md
    └── ...

Each transcript file contains:

---
video_id: abc123
title: "Sleep Optimization Toolkit"
channel: Huberman Lab
published_at: 2024-01-15
duration: PT2H15M30S
---

[Transcript content here...]

Troubleshooting

🛑 IP BLOCKED

No videos.csv files found

extract-videos

sync-all-channels

transcript-download: command not found

pip install -e .

Common Mistakes

1. Running without checking disk space - Transcripts are small (~50KB each), but 10,000 videos = ~500MB. Verify space before overnight runs.

2. Interrupting during a download - Safe to Ctrl+C between videos. If you interrupt mid-download, that video's transcript may be incomplete. The CSV won't mark it as downloaded, so it will retry.

3. Running multiple instances - Don't run the script twice simultaneously. The 60s delay assumes single-threaded operation to respect rate limits.

4. Expecting instant results - The 60s delay is intentional. Faster rates trigger IP blocks. Plan for overnight runs.

Quality Checklist

Before considering batch download complete:

• All folders show transcript counts in summary output
• No

  🛑 IP BLOCKED

  errors (or resolved by VPN switch)
• Spot-check 2-3 random

  .md

  files have valid content
• CSV

  transcript_downloaded

  column reflects actual downloads

When to Use This vs. download-transcripts

download-all-transcripts

download-transcripts --folder <name>

download-transcripts

Technical Details

• Rate limiting: 60 second delay between videos (configurable in script's

  DELAY

  variable)
• Exit codes: 0 = success, 1 = general error, 2 = IP blocked (special handling)
• Resumability: Based on

  transcript_downloaded

  column in each CSV
• Dependencies: Requires

  transcript-download

  CLI from project's Python package