$ man scrape-website
/scrape-website(1)
PRICE / CALL
$0.04
USDC · base mainnet · scheme: exact
METHOD
POST
CLUSTER
webprobeCATEGORY
utilities
STATUS
● live
NAME
scrape-website — web scraper / html to text / website content extractor
SYNOPSIS
POST https://x402.org/v1/scrape-website
Content-Type: application/json
X-PAYMENT: <signed-transferWithAuthorization>
{ ... }↳ first call →
402 Payment Required. Sign USDCtransferWithAuthorization, retry with theX-PAYMENT header.DESCRIPTION
Web scraper / HTML to text / website content extractor. Returns title, OG/Twitter meta, body in text/HTML/markdown. Optional links. Cheerio-based — fast, no headless browser.
INPUT — request schema
| property | type | description | req? |
|---|---|---|---|
| url | string | Public http/https URL to scrape. | required |
| format | string | 'text' (default), 'html', or 'markdown'. enum: text · html · markdown | optional |
| include_links | boolean | Include array of unique links found on the page (max 500). Default false. | optional |
| user_agent | string | Override the User-Agent header. | optional |
OUTPUT — response shape
| field | type | description |
|---|---|---|
| url | string | Final URL fetched after following redirects from the requested page. |
| final_url | string | Final URL after redirects. |
| status_code | number | HTTP status code returned by the target server for the scrape request. |
| title | string | Page title pulled from the <title> tag or og:title fallback. |
| description | string | Meta description from the page's description, og:description, or twitter:description tag. |
| canonical | string | Canonical URL declared by the page's <link rel="canonical"> tag. |
| lang | string | Language code from the <html lang="..."> attribute (e.g. en, es). |
| h1 | string | Text content of the page's first <h1> heading. |
| og | object | All og:* meta tags. |
| object | All twitter:* meta tags. | |
| text | string | Body when format=text. |
| html | string | Body when format=html. |
| markdown | string | Body when format=markdown. |
| format | string | Body format returned: text, html, or markdown depending on the requested format param. |
| links | array | Array of {href, text} when include_links=true. |
| body_chars | number | Character count of the extracted body content in the chosen format. |
EXAMPLES — two ways to call
EXAMPLE 1 · curl
curl -X POST https://x402.org/v1/scrape-website \
-H 'Content-Type: application/json' \
-d '{ }'first response =
402 Payment Required with payment requirements; sign + retry with X-PAYMENT.EXAMPLE 2 · mcp
# install once claude mcp add x402 --command "npx x402-deployer-mcp" # then ask Claude Code: # "use the scrape-website tool to ..."
MCP server handles payment automatically — your coding agent just calls the tool by name.
METADATA
- tags
- scrapefetchhtmlextractmetadataurl
- methods
- POST
- cluster
- webprobe
- price
- $0.04 USDC per call
ADJACENT — other endpoints in webprobe
| endpoint | description | price |
|---|---|---|
| arxiv-summarize | arXiv paper summarizer / research-paper TLDR. | $0.04 |
| rss-from-anything | RSS feed generator / HTML to RSS converter. | $0.04 |
| scrape | Web scraper / HTML extractor. | $0.04 |
| screenshot | Website screenshot / URL to PNG/JPG. | $0.04 |
| webpage-diff | Webpage change detection / website monitor / content diff. | $0.04 |
| website-screenshot | URL to PNG / JPG / website screenshot tool. | $0.04 |
| arxiv-search | arXiv full-text search. | $0.03 |
| domain-suggest | Brandable domain suggester / startup-name brainstormer / product-name proposer / domain candidate generator. | $0.05 |
SEE ALSO