Internal Link Audit

Crawls a domain to build an internal link graph. Identifies orphan pages, underlinked pages, and broken internal links. Suggests anchor text improvements.

Inputs

• domain

  : Target domain URL (e.g.,

  https://example.com

  ). Include protocol.

• max_pages

  (optional): Max pages to crawl. Default: 100. Maximum: 200.

Execution

Step 1: Crawl Site

Use

scripts/fetch_page.py

and

scripts/parse_html.py

to crawl the site:

# Fetch homepage
python3 scripts/fetch_page.py <domain>
# Parse HTML to extract links
python3 scripts/parse_html.py <html_content>

Follow internal links only (same domain hostname). Normalize URLs:

• Remove trailing slashes (treat

  /about

  and

  /about/

  as same)
• Remove URL fragments (

  #section

  )
• Preserve query strings only if they appear to be content (not tracking: strip

  utm_*

  ,

  ref=

  ,

  source=

  )

Respect robots.txt: fetch

<domain>/robots.txt

first, skip disallowed paths. Cap crawl at

max_pages

. Track crawl queue (BFS order from homepage).

Step 2: Build Link Graph

For each crawled page, record:

• Source URL
• Target URL (all internal links found)
• Anchor text for each link

Build adjacency structure:

• outbound[url]

  = list of (target, anchor_text)

• inbound[url]

  = list of (source, anchor_text)

Step 3: Calculate Per-Page Metrics

For each crawled page:

• Inbound internal links count (links from other crawled pages pointing here)
• Outbound internal links count (links from this page to other pages)
• Link depth from homepage (BFS level at which this page was first discovered)

Step 4: Identify Issues

• Orphan pages: Pages discovered via sitemap (

  <domain>/sitemap.xml

  ) or linked from other pages but with 0 inbound internal links. Fetch sitemap first to identify all known URLs.
• Underlinked pages: Pages with < 3 inbound internal links (high priority for internal linking).
• Excessive outbound: Pages with > 100 outbound internal links (PageRank dilution).
• Broken internal links: For top 20 most-linked pages, verify HTTP status. Flag 4xx/5xx.

Step 5: Anchor Text Suggestions for Top 5 Underlinked Pages

For each of the 5 most underlinked pages (fewest inbound links, excluding orphans):

1. Fetch the page and extract H1 and

   <title>

   tag
2. Identify top 3 relevant anchor text options based on: H1 noun phrases, title keywords, page URL slug
3. Find pages in crawl that mention related terms and could link to this page
4. Output: target URL, suggested anchors (ranked), recommended source pages

Step 6: Optional Ahrefs Enrichment

If Ahrefs available (ToolSearch '+ahrefs'):

• Fetch

  site-explorer-pages-by-internal-links

  for the domain
• Cross-reference crawl findings with Ahrefs data
• Add

  ### Ahrefs Internal Link Data

  section showing discrepancies

Output Format

## Internal Link Audit: [domain]

### Summary Stats

| Metric | Value |
|--------|-------|
| Pages crawled | N / max_pages |
| Total internal links | N |
| Avg inbound links per page | N.N |
| Orphan pages found | N |
| Underlinked pages (<3 inbound) | N |
| Pages with broken internal links | N |

### Orphan Pages

Pages with 0 inbound internal links:

| URL | Depth | Found via |
|-----|-------|-----------|
| /page | — | sitemap |

### Top 5 Underlinked Pages — Suggested Anchors

**1. [URL]** — N inbound links
- H1: "[heading]"
- Suggested anchors: "[anchor 1]", "[anchor 2]", "[anchor 3]"
- Recommended source pages: [list pages that mention related terms]

### Broken Internal Links

| Source Page | Broken Link | Status Code |
|---|---|---|

### Link Depth Distribution

| Depth | Pages | % of Total |
|-------|-------|------------|
| 1 (homepage) | 1 | X% |
| 2 | N | X% |
| 3+ | N | X% |

### Pages with Excessive Outbound Links (>100)

| Page | Outbound Links |
|------|----------------|

## Data Sources

- Internal crawl via fetch_page.py + parse_html.py ([N] pages, [date])
- [Ahrefs MCP — if used]