Claude-skill-registry context-scrapers

Web scraping spiders (Scrapy), parsers, and source management.

install

source · Clone the upstream repo

git clone https://github.com/majiayu000/claude-skill-registry

Claude Code · Install into ~/.claude/skills/

T=$(mktemp -d) && git clone --depth=1 https://github.com/majiayu000/claude-skill-registry "$T" && mkdir -p ~/.claude/skills && cp -r "$T/skills/data/context-scrapers" ~/.claude/skills/majiayu000-claude-skill-registry-context-scrapers && rm -rf "$T"

manifest: skills/data/context-scrapers/SKILL.md

Scrapers Context

Overview

Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).

Active Files

Spiders

backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py

- Legistar meeting scraper

backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py

- Municode scraper

Configuration

backend/affordabot_scraper/affordabot_scraper/pipelines.py

- Item pipelines (DB storage)

backend/affordabot_scraper/affordabot_scraper/settings.py

- Scrapy settings

Verification Scripts

```
scripts/verify_raw_scrapes.py
```
- Check raw_scrapes table
```
scripts/verify_municode_discovery.py
```
- Verify discovery logic

Usage

Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.