Marketplace docs-sources

Knowledge of documentation platforms and fetching strategies. Use when adding new documentation sources, determining fetch strategy for a docs site, detecting doc frameworks, or configuring the docs registry.

install

source · Clone the upstream repo

git clone https://github.com/aiskillstore/marketplace

Claude Code · Install into ~/.claude/skills/

T=$(mktemp -d) && git clone --depth=1 https://github.com/aiskillstore/marketplace "$T" && mkdir -p ~/.claude/skills && cp -r "$T/skills/consiliency/docs-sources" ~/.claude/skills/aiskillstore-marketplace-docs-sources && rm -rf "$T"

manifest: skills/consiliency/docs-sources/SKILL.md

source content

Documentation Sources Skill

Understand documentation platforms and how to efficiently fetch content from each. Supports 20+ platforms and frameworks.

Variables

Variable	Default	Description
PREFER_LLMSTXT	true	Check for llms.txt before other strategies
PREFER_GITHUB	true	Prefer raw GitHub over web crawling
BROWSER_FALLBACK	true	Use browser automation when curl fails
DEFAULT_STRATEGY	web_crawl	Fallback when detection fails

Instructions

MANDATORY - Follow the Workflow steps below when adding or analyzing documentation sources.

Always check for AI-native signals first (llms.txt)
Prefer raw content over rendered HTML
Use browser automation only when necessary

Red Flags - STOP and Reconsider

If you're about to:

Web crawl a site without checking for llms.txt first
Use browser automation without trying curl first
Add a source without detecting the documentation framework
Skip GitHub raw access when repo is available

STOP -> Read the appropriate cookbook file -> Follow detection order -> Then proceed

Quick Decision Tree

What documentation are you adding?
│
├─ Has /llms.txt? ──────────────────────► llmstxt-strategy.md
│
├─ GitHub repo available?
│   └─ Has docs.yml/mint.json/mkdocs.yml? ► github-strategy.md
│
├─ Has /openapi.json or /asyncapi.yaml? ─► openapi-strategy.md
│
├─ Has /sitemap.xml? ───────────────────► sitemap-strategy.md
│
├─ Curl returns <1KB? (JS-rendered) ────► browser-strategy.md
│
└─ Nothing else works? ─────────────────► sitemap-strategy.md (crawl mode)

Workflow

Identify the documentation URL
CHECKPOINT: Check for AI-native signals (llms.txt, ai.txt)
Check if GitHub repo is available
Detect documentation framework from config files
Select appropriate strategy from cookbook
CHECKPOINT: Test fetch with curl before using browser
Configure registry entry with correct strategy
Validate fetch works correctly

Strategy Priority Order


cookbook/llmstxt-strategy.md
cookbook/github-strategy.md
cookbook/openapi-strategy.md
cookbook/sitemap-strategy.md
cookbook/browser-strategy.md
cookbook/sitemap-strategy.md

Cookbook

llms.txt (AI-Native)

IF: Site has
```
/llms.txt
```
or
```
/llms-full.txt
```
THEN: Read
```
cookbook/llmstxt-strategy.md
```
CHECK:
```
curl -sI {url}/llms.txt | head -1
```

GitHub Raw

IF: GitHub repo is linked from docs site
THEN: Read
```
cookbook/github-strategy.md
```
SUPPORTS: Fern, Docusaurus, MkDocs, Mintlify, Sphinx, Nextra, Starlight

OpenAPI / AsyncAPI / GraphQL

IF: Site has API spec (
```
/openapi.json
```
,
```
/asyncapi.yaml
```
,
```
/graphql
```
)
THEN: Read
```
cookbook/openapi-strategy.md
```

Sitemap / Web Crawl

IF: Site has
```
/sitemap.xml
```
or need to crawl
THEN: Read
```
cookbook/sitemap-strategy.md
```

Browser Automation

IF: Curl fails (JS-rendered, blocked, <1KB response)
THEN: Read
```
cookbook/browser-strategy.md
```
ALSO: See
```
browser-discovery
```
skill for IDE browser tools

Quick Detection Commands

# Check for llms.txt
curl -sI "https://docs.viperjuice.dev/llms.txt" | head -1

# Check for sitemap
curl -sI "https://docs.viperjuice.dev/sitemap.xml" | head -1

# Check for OpenAPI
curl -sI "https://api.viperjuice.dev/openapi.json" | head -1

# Test curl response size (detect JS-rendered)
curl -s "https://docs.viperjuice.dev" | wc -c
# If < 1000 bytes, likely JS-rendered -> use browser

Framework Detection

Framework	Config File	Typical Location
Fern	`docs.yml`	`fern/docs.yml`
Docusaurus	`docusaurus.config.js`	repo root
MkDocs	`mkdocs.yml`	repo root
Mintlify	`mint.json`	repo root
Sphinx	`conf.py`	`docs/conf.py`
Nextra	`_meta.json`	`pages/_meta.json`
Starlight	`astro.config.mjs`	repo root
Antora	`antora.yml`	repo root
GitBook	`SUMMARY.md`	repo root

See

reference/framework-detection.md

for detailed patterns.

Registry Configuration

See

reference/registry-examples.md

for configuration templates.

Basic structure:

{
  "source-id": {
    "name": "Display Name",
    "strategy": "strategy_name",
    "paths": {
      "homepage": "https://..."
    }
  }
}

Auto-Discovery from Manifests

When used with the

library-detection

skill, automatically suggest documentation sources based on detected dependencies.

Workflow

Receive stack info from
```
library-detection
```
skill
Map each framework/library to known documentation sources
Check if source already exists in
```
ai-docs/libraries/_registry.json
```
Suggest
```
/ai-dev-kit:docs-add
```
commands for missing sources

Library-to-Docs Mapping

Library	Documentation URL	Strategy
react	https://react.dev	llmstxt
next	https://nextjs.org/docs	github_raw
vue	https://vuejs.org/guide	github_raw
nuxt	https://nuxt.com/docs	github_raw
svelte	https://svelte.dev/docs	github_raw
fastapi	https://fastapi.tiangolo.com	github_raw
django	https://docs.djangoproject.com	web_sitemap
flask	https://flask.palletsprojects.com	web_sitemap
express	https://expressjs.com	web_sitemap
prisma	https://www.prisma.io/docs	llmstxt
drizzle	https://orm.drizzle.team/docs	github_raw
vitest	https://vitest.dev	github_raw
playwright	https://playwright.dev/docs	github_raw
tailwindcss	https://tailwindcss.com/docs	llmstxt
trpc	https://trpc.io/docs	github_raw
zod	https://zod.dev	llmstxt

Auto-Discovery Command

When

/ai-dev-kit:quickstart-codebase

or manual request:

# Get detected frameworks
FRAMEWORKS=$(library-detection --output json | jq -r '.frameworks[].name')

# For each framework, check if docs exist
for fw in $FRAMEWORKS; do
  if ! grep -q "\"$fw\"" ai-docs/libraries/_registry.json; then
    echo "Missing docs for: $fw"
    echo "Suggest: /ai-dev-kit:docs-add <url> $fw"
  fi
done

Output

Returns list of:

{
  "already_tracked": ["react", "typescript"],
  "missing": [
    {"library": "fastapi", "url": "https://fastapi.tiangolo.com", "command": "/ai-dev-kit:docs-add https://fastapi.tiangolo.com fastapi"}
  ],
  "unknown": ["custom-internal-lib"]
}

Output

After adding a source, verify:

Pages are discovered correctly
Content is fetched without errors
Registry entry is valid JSON

Priority	Signal	Strategy	Cookbook
1	`/llms.txt` exists	`llmstxt`	`cookbook/llmstxt-strategy.md`
2	GitHub repo available	`github_raw`	`cookbook/github-strategy.md`
3	OpenAPI/AsyncAPI spec	`openapi`	`cookbook/openapi-strategy.md`
4	`/sitemap.xml` exists	`web_sitemap`	`cookbook/sitemap-strategy.md`
5	JS-rendered / curl fails	`browser_crawl`	`cookbook/browser-strategy.md`
6	Nothing else works	`web_crawl`	`cookbook/sitemap-strategy.md`