OpenSkillIndex← back to search
claude-codeopenclawautomation

Skills mrscraper

Run AI-powered, unblockable web scraping, data extraction with natural language via the MrScraper API

install
source · Clone the upstream repo
git clone https://github.com/openclaw/skills
Claude Code · Install into ~/.claude/skills/
T=$(mktemp -d) && git clone --depth=1 https://github.com/openclaw/skills "$T" && mkdir -p ~/.claude/skills && cp -r "$T/skills/ai-mrscraper/mrscraper" ~/.claude/skills/openclaw-skills-mrscraper && rm -rf "$T"
OpenClaw · Install into ~/.openclaw/skills/
T=$(mktemp -d) && git clone --depth=1 https://github.com/openclaw/skills "$T" && mkdir -p ~/.openclaw/skills && cp -r "$T/skills/ai-mrscraper/mrscraper" ~/.openclaw/skills/openclaw-skills-mrscraper && rm -rf "$T"
manifest: skills/ai-mrscraper/mrscraper/SKILL.md
tags
#scraping#data-extraction#web-automation#api#stealth-browser#web-crawling
source content

MrScraper

Run AI-powered, unblockable web scraping, data extraction with natural language via the MrScraper API

Actions

This skill supports:

  • Opening blocked pages through unblocker (stealth browser + IP rotation)
  • Starting AI scraper runs from natural-language instructions
  • Rerunning existing scraper configurations on one or multiple URLs
  • Running manual workflow-based reruns
  • Fetching paginated results and detailed results by ID

This skill is API-only and does not depend on bundled local scripts.

Base URLs

  • Unblocker API: 
    https://api.mrscraper.com
  • Platform API: 
    https://api.app.mrscraper.com

Authentication

Unblocker API auth

Use query-param auth on unblocker endpoint:

  • token=<MRSCRAPER_API_TOKEN>

Platform API auth

Use header-based auth on platform endpoints:

x-api-token: <MRSCRAPER_API_TOKEN>
accept: application/json
content-type: application/json

How to get 
MRSCRAPER_API_TOKEN
?

An API token lets your applications securely interact with MrScraper APIs and rerun scrapers created in the dashboard.

Follow these steps in the dashboard:

  1. Click your User Profile at the top-right corner.
  2. Select API Tokens.
  3. Click New Token.
  4. Enter a name and set an expiration date.
  5. Click Create.
  6. Copy the new token and store it securely as 
    MRSCRAPER_API_TOKEN
    .
  7. Use it in requests through the 
    x-api-token
    header.

Security rule:

  • Never expose tokens in client-side code (browser/mobile app bundles).
  • Store tokens in environment variables or server-side secret managers.

Notes from the auth docs:

  • The API key works for all V3 Platform endpoints.
  • The same key can be used for endpoints on 
    sync.scraper.mrscraper.com
    .
  • For access to endpoints on other hosts, contact 
    support@mrscraper.com
    .

Install and Runtime

  • No local install step is required by this skill document.
  • No bundled 
    scripts/
    are required.
  • Calls are direct HTTPS requests to the two base URLs above.

Data and Scope

  • Data is sent only to 
    api.app.mrscraper.com
    and 
    api.mrscraper.com
    .
  • Responses may contain extracted page content and scrape metadata.
  • This skill does not define hidden persistence or background jobs.
  • Never expose tokens in logs, commits, or output.

Endpoints

1. Unblocker

  • Method: 
    GET
  • URL: 
    https://api.mrscraper.com
  • Auth: 
    token
    query parameter

Opens a target URL through stealth browsing and IP rotation, then returns HTML. Use this when direct access is blocked by captcha or anti-bot protections.

Query parameters:

FieldTypeRequiredDefaultDescription
token
string
Yes—Unblocker token (
MRSCRAPER_API_TOKEN
)
url
string
Yes—URL-encoded target URL
timeout
number
No60Max wait in seconds (example 
120
)
geoCode
string
NoNoneGeographic routing code (example 
SG
)
blockResources
boolean
NofalseBlock non-essential resources

Request example:

curl --location 'https://api.mrscraper.com?token=<MRSCRAPER_API_TOKEN>&timeout=120&geoCode=SG&url=https%3A%2F%2Fwww.lazada.sg%2Fproducts%2Fpdp-i111650098-s23209659764.html&blockResources=false'

Response example:

<!doctype html>
<html>
  <head>...</head>
  <body>...</body>
</html>

Notes:

  • Prefer explicit 
    geoCode
    and practical timeouts for repeatable behavior.
  • Only pass cookies when session-specific content is required.

2. Create AI Scraper

  • Method: 
    POST
  • Host: 
    https://api.app.mrscraper.com
  • Path: 
    /api/v1/scrapers-ai
  • Auth: 
    x-api-token

Create a new AI scraper run from natural-language instructions.

Payload parameters (for 
agent
: 
general
or 
agent
: 
listing
):

FieldTypeRequiredDefaultDescription
url
stringYes—Target URL
message
stringYes—Extraction instruction
agent
stringNogeneralThe AI agent type to use for scraping: 
general
, 
listing
, or 
map
proxyCountry
stringNoNoneISO country code for proxy-based scraping

Payload parameters (for 
agent
: 
map
):

FieldTypeRequiredDefaultDescription
url
string
Yes—Target URL
agent
string
NomapThe AI agent type to use for scraping (for this case it is 
map
)
maxDepth
number
No2Maximum depth level for crawling links from the starting URL.<br>0 = only the starting URL, 1 = +direct links
maxPages
number
No50Maximum number of pages to scrape during the crawling process.
limit
number
No1000Maximum number of data records to extract across all pages. Scraping stops when this limit is reached.
includePatterns
string
No""Regex patterns to include (separate multiple with 
||
)
excludePatterns
string
No""Regex patterns to exclude (separate multiple with 
||
)

Request example:

curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai" \
  -H "x-api-token: <MRSCRAPER_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "message": "Extract title, price, stocks, and rating",
    "agent": "general"
  }'

Response example:

{
  "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
  "createdAt": "2019-08-24T14:15:22Z",
  "createdById": "e13e432a-5323-4484-a91d-b5969bc564d9",
  "updatedAt": "2019-08-24T14:15:22Z",
  "updatedById": "d8bc6076-4141-4a88-80b9-0eb31643066f",
  "deletedAt": "2019-08-24T14:15:22Z",
  "deletedById": "8ef578ad-7f1e-4656-b48b-b1b4a9aaa1cb",
  "userId": "2c4a230c-5085-4924-a3e1-25fb4fc5965b",
  "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
  "type": "AI",
  "url": "http://example.com",
  "status": "Finished",
  "error": "string",
  "tokenUsage": 0,
  "runtime": 0,
  "data": {}, // MAIN SCRAPED DATA
  "htmlPath": "string",
  "recordingPath": "string",
  "screenshotPath": "string",
  "dataPath": "string"
}

Notes:

  • Choose agent type correctly as each agent is specialized for specified use cases. Use 
    general
    for most standard web scraping tasks. The go to agent if the user doesn't specify or the connected LLM is not confident about the type of page. But mostly used for scraping product page, but handles any type of page very well as well. Use 
    listing
    for scraping listing pages like product listings, job listings, etc. Choose this if the connected LLM can confidently identify whether the given URL is a listing page. Use 
    map
    for crawling and getting all subdomain or subpages of a website. Choose this if the user specifies that the given URL is a website and not a specific page. For 
    map
    agent type, there is a special args that can be used to configure the scraping process.
  • For the 
    map
    agent, you can use special arguments to control crawling:<br>
    maxDepth
    (lower values 1–2 for focused scraping, max 3 recommended),<br>
    maxPages
    (limits total pages regardless of depth),<br>
    limit
    (caps total records extracted),<br>and 
    includePatterns
    /
    excludePatterns
    (regex patterns separated by 
    ||
    to specify which URLs to crawl or skip, e.g., 
    */products/*||*/blog/*
    or 
    */cart/*||*.pdf
    ).<br>If 
    includePatterns
    is an empty string, all URLs are included. If 
    excludePatterns
    is an empty string, no URLs are excluded.

3. Rerun AI Scraper

  • Method: 
    POST
  • Host: 
    https://api.app.mrscraper.com
  • Path: 
    /api/v1/scrapers-ai-rerun
  • Auth: 
    x-api-token

Reruns an existing scraper configuration on a new URL.

Payload parameters:

FieldTypeRequiredDefaultDescription
scraperId
string
Yes—Scraper ID retrieved from created AI scraper
url
string
Yes—Target URL

Optional payload parameters for 
map
agent:

FieldTypeRequiredDefaultDescription
maxDepth
numberNo2Crawl depth
maxPages
numberNo50Maximum pages to crawl
limit
numberNo1000Result limit
includePatterns
stringNo""Regex patterns to include (separate multiple with 
||
)
excludePatterns
stringNo""Regex patterns to exclude (separate multiple with 
||
)

Request example:

curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun" \
  -H "accept: application/json" \
  -H "x-api-token: <MRSCRAPER_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "url": "https://shopee.sg/"
  }'

Response example:

{
  "message": "Successful operation!",
  "data": {
    "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
    "createdAt": "2019-08-24T14:15:22Z",
    "createdById": "e13e432a-5323-4484-a91d-b5969bc564d9",
    "updatedAt": "2019-08-24T14:15:22Z",
    "updatedById": "d8bc6076-4141-4a88-80b9-0eb31643066f",
    "deletedAt": "2019-08-24T14:15:22Z",
    "deletedById": "8ef578ad-7f1e-4656-b48b-b1b4a9aaa1cb",
    "userId": "2c4a230c-5085-4924-a3e1-25fb4fc5965b",
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "type": "Rerun-AI",
    "url": "http://example.com",
    "status": "Finished",
    "error": "string",
    "tokenUsage": 0,
    "runtime": 0,
    "data": {}, // MAIN SCRAPED DATA
    "htmlPath": "string",
    "recordingPath": "string",
    "screenshotPath": "string",
    "dataPath": "string",
    "htmlContent": "string"
  }
}

4. Bulk Rerun AI Scraper

  • Method: 
    POST
  • Host: 
    https://api.app.mrscraper.com
  • Path: 
    /api/v1/scrapers-ai-rerun/bulk
  • Auth: 
    x-api-token

Runs one scraper configuration over multiple URLs.

Payload parameters:

FieldTypeRequiredDefaultDescription
scraperId
string
Yes—Existing AI scraper configuration ID
urls
array[string]
Yes—Target URLs to run

Request example:

curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai-rerun/bulk" \
  -H "x-api-token: " \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "urls": [
      "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "https://books.toscrape.com/catalogue/soumission_998/index.html"
    ]
  }'

Response example:

{
  "message": "Bulk rerun started successfully",
  "data": {
    "bulkResultId": "f89f8f58-3c9a-42e5-a72e-59fa6c389f09",
    "status": "Running",
    "totalUrls": 3
  }
}

5. Rerun Manual Scraper

  • Method: 
    POST
  • Host: 
    https://api.app.mrscraper.com
  • Path: 
    /api/v1/scrapers-manual-rerun
  • Auth: 
    x-api-token

Executes a rerun using a manual browser workflow.

Creating a Manual Scraper

Before calling the manual rerun endpoint, you need to create and save a manual scraper from the dashboard. Follow these steps:

  1. Open the 
    MrScraper
    dashboard and go to 
    Scraper
    .
  2. Click 
    New Manual Scraper +
    .
  3. Enter your target URL.
  4. Add workflow steps that match your site's behavior (e.g., 
    Input
    , 
    Click
    , 
    Delay
    , 
    Extract
    , 
    Inject JavaScript
    ).
  5. Configure pagination if needed (using options like 
    Query Pagination
    , 
    Directory Pagination
    , or 
    Next Page Link
    ).
  6. Test and save the scraper, then copy its 
    scraperId
    to use in API reruns.

Payload parameters:

FieldTypeRequiredDefaultDescription
scraperId
string
Yes—ID of the manual scraper to rerun.
url
string
Yes—Target URL for the rerun.
workflow
array<object>
NoNoneAllows overriding the saved workflow steps. By default, uses the workflow saved during manual creation.

Request example:

curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-manual-rerun" \
  -H "accept: application/json" \
  -H "x-api-token: " \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "url": "https://books.toscrape.com/",
    "workflow": [
      {
        "type": "extract",
        "data": {
          "extraction_type": "text",
          "attribute": null,
          "name": "book",
          "selector": "h3 a"
        }
      }
    ],
    "record": false,
    "paginator": {
      "type": "query_pagination",
      "max_page": 1,
      "enabled": false
    }
  }'

Response example:

{
  "message": "Successful operation!",
  "data": {
    "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
    "createdAt": "2019-08-24T14:15:22Z",
    "createdById": "e13e432a-5323-4484-a91d-b5969bc564d9",
    "updatedAt": "2019-08-24T14:15:22Z",
    "updatedById": "d8bc6076-4141-4a88-80b9-0eb31643066f",
    "deletedAt": "2019-08-24T14:15:22Z",
    "deletedById": "8ef578ad-7f1e-4656-b48b-b1b4a9aaa1cb",
    "userId": "2c4a230c-5085-4924-a3e1-25fb4fc5965b",
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "type": "Rerun-AI",
    "url": "http://example.com",
    "status": "Finished",
    "error": "string",
    "tokenUsage": 0,
    "runtime": 0,
    "data": {}, // MAIN SCRAPED DATA
    "htmlPath": "string",
    "recordingPath": "string",
    "screenshotPath": "string",
    "dataPath": "string",
    "htmlContent": "string"
  }
}

6. Bulk Rerun Manual Scraper

  • Method: 
    POST
  • Host: 
    https://api.app.mrscraper.com
  • Path: 
    /api/v1/scrapers-manual-rerun/bulk
  • Auth: 
    x-api-token

Runs one scraper configuration over multiple URLs.

Payload parameters:

FieldTypeRequiredDefaultDescription
scraperId
string
Yes—Existing manual scraper configuration ID
urls
array[string]
Yes—Target URLs to run

Request example:

curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-manual-rerun/bulk" \
  -H "x-api-token: " \
  -H "Content-Type: application/json" \
  -d '{
    "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
    "urls": [
      "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "https://books.toscrape.com/catalogue/soumission_998/index.html"
    ]
  }'

Response example:

{
  "message": "Bulk rerun started successfully",
  "data": {
    "bulkResultId": "f89f8f58-3c9a-42e5-a72e-59fa6c389f09",
    "status": "Running",
    "totalUrls": 3
  }
}

7. Fetch Results

  • Method: 
    GET
  • Host: 
    https://api.app.mrscraper.com
  • Path: 
    /api/v1/results
  • Auth: 
    x-api-token

Returns paginated scrape results.

Query parameters:

FieldTypeRequiredDefaultDescription
sortField
stringYes
updatedAt
Sort column
sortOrder
stringYes
DESC
Sort direction
page
numberYes1Page number
pageSize
numberYes10Items per page
search
stringNoNoneSearch keyword
dateRangeColumn
stringNo
createdAt
Date field to filter
startAt
stringNoNoneDate range start (ISO)
endAt
stringNoNoneDate range end (ISO)

Notes:

  • sortField
    options: 
    createdAt
    , 
    updatedAt
    , 
    id
    , 
    type
    , 
    url
    , 
    status
    , 
    error
    , 
    tokenUsage
    , 
    runtime
  • sortOrder
    options: 
    ASC
    , 
    DESC
  • dateRangeColumn
    options: 
    createdAt
    , 
    updatedAt

Request example:

curl -X GET "https://api.app.mrscraper.com/api/v1/results?sortField=updatedAt&sortOrder=DESC&pageSize=10&page=1" \
  -H "accept: application/json" \
  -H "x-api-token: <MRSCRAPER_API_TOKEN>"

Response example:

{
  "message": "Successful fetch",
  "data": [
    {
      "createdAt": "2025-11-11T09:50:09.722Z",
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "userId": "2c4a230c-5085-4924-a3e1-25fb4fc5965b",
      "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
      "type": "AI",
      "url": "http://example.com",
      "status": "Finished",
      "error": "string",
      "tokenUsage": 5,
      "runtime": 0,
      "data": "{ \"title\": \"Product A\", \"price\": \"$10\" }",
      "htmlPath": "string",
      "recordingPath": "string",
      "screenshotPath": "string",
      "dataPath": "string"
    }
  ],
  "meta": {
    "page": 1,
    "pageSize": 10,
    "total": 1,
    "totalPage": 1
  }
}

8. Fetch Detailed Result by ID

  • Method: 
    GET
  • Host: 
    https://api.app.mrscraper.com
  • Path: 
    /api/v1/results/{id}
  • Auth: 
    x-api-token

Returns one detailed result object for a specific result ID.

Query parameters:

FieldTypeRequiredDefaultDescription
id
string
Yes—Result ID

Request example:

curl -X GET "https://api.app.mrscraper.com/api/v1/results/497f6eca-6276-4993-bfeb-53cbbbba6f08" \
  -H "accept: application/json" \
  -H "x-api-token: <MRSCRAPER_API_TOKEN>"

Response example:

{
  "message": "Successful fetch",
  "data": [
    {
      "createdAt": "2025-11-11T09:50:09.722Z",
      "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08",
      "userId": "2c4a230c-5085-4924-a3e1-25fb4fc5965b",
      "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b",
      "type": "AI",
      "url": "http://example.com",
      "status": "Finished",
      "error": "string",
      "tokenUsage": 5,
      "runtime": 0,
      "data": "string",
      "htmlPath": "string",
      "recordingPath": "string",
      "screenshotPath": "string",
      "dataPath": "string"
    }
  ]
}

Errors

Standard platform API errors:

StatusMeaning
400
Invalid request payload
401
Missing or invalid API token
404
Scraper or result not found
429
Rate limit exceeded
500
Internal scraper error

Error format:

{
  "message": "string",
  "error": "string",
  "statusCode": "number"
}

Operating Rules

  • Validate required fields before every call.
  • Use pagination for large result sets.
  • Retry on 
    429
    with exponential backoff.
  • Never expose credentials in outputs.