Server Log Analysis

Analyzes local server log files for crawl budget breakdown. No MCP or external calls required.

Inputs

• file

  : Absolute path to server log file (Apache Combined, Apache Common, or Nginx access log). If user provides relative path, resolve with

  Bash: realpath <path>

  .

Execution

Step 1: Format Detection

Read the first 10 lines of the log file to detect format:

• Apache Combined:

  %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"

  — 9+ fields, has referer and UA in quotes
• Apache Common:

  %h %l %u %t "%r" %>s %b

  — 7 fields, no referer/UA
• Nginx: similar to Apache Combined with slight field order differences
• Check for compressed files (.gz) — if detected, inform user to decompress first

Step 2: Parse Log Lines

Use Bash awk to extract fields. For Apache Combined/Nginx format (9 fields):

awk '{
  ip=$1; method_url=$7; status=$9; ua=$0
  match($0, /"([^"]+)"$/, arr)  # Extract UA from last quoted field
  print ip, $7, $9, arr[1]
}' logfile

For Apache Common (7 fields): ip=$1, request=$7, status=$9, ua="unknown"

Step 3: Classify User-Agents

Group each request into categories:

• Googlebot:

  Googlebot

  ,

  Googlebot-Image

  ,

  Googlebot-News

  ,

  AdsBot-Google

• Bingbot:

  bingbot

  ,

  BingPreview

  ,

  MicrosoftPreview

• Other search bots:

  Slurp

  (Yahoo),

  DuckDuckBot

  ,

  Baiduspider

  ,

  YandexBot

  ,

  Sogou

• AI crawlers:

  GPTBot

  ,

  ClaudeBot

  ,

  PerplexityBot

  ,

  Bytespider

  ,

  CCBot

  ,

  anthropic-ai

• Monitoring tools:

  Pingdom

  ,

  UptimeRobot

  ,

  StatusCake

  ,

  NewRelic

  ,

  Datadog

• Real users: everything else (browsers:

  Mozilla

  ,

  Chrome

  ,

  Safari

  ,

  Firefox

  ,

  Edge

  )
• Unknown: no UA or unrecognized

Step 4: Calculate Metrics

Using awk/grep on the log file:

1. Total request count
2. Requests by bot category (count per category, % of total)
3. Requests by HTTP status code (200, 301, 302, 404, 500, etc.)
4. Top 20 crawled URLs by frequency — sort by count descending
5. Top 10 crawled path prefixes (first 2 URL segments, e.g.,

   /blog/

   ,

   /products/

   ) — aggregate by prefix
6. Requests by hour-of-day (extract hour from timestamp field

   [DD/Mon/YYYY:HH:MM:SS]

   )

Step 5: Identify Crawl Budget Concerns

Flag these patterns:

• 4xx error rate >5%: crawlers wasting budget on broken URLs
• 5xx error rate >1%: server errors burning crawl budget
• Duplicate crawl patterns: same URL crawled >10x without apparent content change
• Low-value paths: bots crawling

  /wp-admin

  ,

  /search?

  ,

  ?sort=

  ,

  ?page=

  , session URLs
• 302 redirect overuse: temporary redirects don't pass full crawl equity
• Non-canonical crawls:

  ?utm_

  or tracking parameters in crawled URLs

Output Format

## Server Log Analysis: [filename]

**File:** [path] | **Format:** [Apache Combined/Common/Nginx] | **Total Requests:** [N]

### Crawl Budget Summary

| Metric | Value |
|--------|-------|
| Total requests | N |
| Bot traffic | N (X%) |
| Human traffic | N (X%) |
| Crawl error rate | X% (4xx+5xx) |
| Date range | [first log entry] to [last log entry] |

### Bot Traffic Breakdown

| Bot Category | Requests | % of Total | Top URL |
|---|---|---|---|
| Googlebot | N | X% | /path |
| Bingbot | N | X% | /path |
| AI Crawlers | N | X% | /path |
| Monitoring | N | X% | /path |
| Real Users | N | X% | — |
| Other/Unknown | N | X% | — |

### Top 20 Crawled URLs

| Rank | URL | Requests | Status Codes |
|------|-----|----------|--------------|
| 1 | /path | N | 200: N, 404: N |

### Crawl Frequency by Path

| Path Prefix | Requests | % of Bot Traffic |
|---|---|---|
| /blog/ | N | X% |

### Status Code Distribution

| Status | Count | % | Interpretation |
|--------|-------|---|----------------|
| 200 | N | X% | OK |
| 301 | N | X% | Permanent redirect |
| 404 | N | X% | Not found (crawl waste) |

### Crawl Budget Recommendations

[Prioritized list of issues found — Critical/High/Medium/Low]

## Data Sources

- Source: Local server log file (no external calls)