agentutility(7)$ man scrape-to-json
/scrape-to-json
PRICE / CALL
$0.02
USDC · base mainnet · scheme: exact
METHOD
POST
CLUSTER
webprobeCATEGORY
uncategorized
STATUS
● live
NAME
scrape-to-json — turn any webpage into structured json that matches your schema
SYNOPSIS
POST https://x402.agentutility.ai/scrape-to-json
Content-Type: application/json
X-PAYMENT: <signed-transferWithAuthorization>
{ ... }↳ first call →
402 Payment Required. Sign USDCtransferWithAuthorization, retry with theX-PAYMENT header.DESCRIPTION
Turn any webpage into structured JSON that matches your schema. Fetches the page server-side (cheerio, no headless browser), converts it to clean markdown, then extracts data conforming to a caller-supplied JSON schema via Morpheus mistral-31-24b. The web scraper to reach for when you need website-to-JSON, page-to-structured-data, or scrape-and-extract with schema enforcement. Pair with watch-page or research-brief for richer agent pipelines.
INPUT — request schema
| property | type | description | req? |
|---|---|---|---|
| url | string | Public URL to scrape (http or https). Follows redirects. | required |
| schema | object | JSON schema describing expected output. Must include a 'properties' key. | required |
| instructions | string | Optional extra extraction guidance. | optional |
| format | string | Pre-LLM body normalization. Default 'markdown'. enum: text · markdown | optional |
| user_agent | string | — | optional |
OUTPUT — response shape
| field | type | description |
|---|---|---|
| url | string | — |
| final_url | string | — |
| status_code | string | — |
| extracted | string | — |
| model | string | — |
EXAMPLES — two ways to call
EXAMPLE 1 · curl
curl -X POST https://x402.agentutility.ai/scrape-to-json \
-H 'Content-Type: application/json' \
-d '{ }'first response =
402 Payment Required with payment requirements; sign + retry with X-PAYMENT.EXAMPLE 2 · mcp
# Install the MCP package for this endpoint's cluster npx -y @agentutility/mcp-<cluster> # Required: EVM private key with USDC on Base export X402_PRIVATE_KEY=0x... # Then call the scrape-to-json tool from your MCP-aware agent.
MCP server handles payment automatically — your coding agent just calls the tool by name.
METADATA
- tags
- web-probescrapetojsonscrape-to-json
- methods
- POST
- cluster
- webprobe
- price
- $0.02 USDC per call
ADJACENT — other endpoints in webprobe
| endpoint | description | price |
|---|---|---|
| archive-snapshot | Finds the closest archived snapshot of a public URL, like a Wayback Machine API. | $0.02 |
| archive-snapshot-api | Finds archived snapshots of public web pages, returning the closest Wayback Machine captures with timestamps, status codes, and replay URLs. | $0.02 |
| archive-url | Looks up the archived copy of a public URL closest to a given timestamp and can fetch cleaned archived page text. | $0.02 |
| brand-domain-check | Checks whether a brand or company domain is established, expiring, registrar-locked, or newly registered, on the same RDAP backend as who… | $0.02 |
| brand-watch-domain-risk | Checks new or suspicious brand-adjacent domains for age, registrar, expiry, DNSSEC, and transfer/lock statuses. | $0.02 |
| creator-domain-check | Checks creator, newsletter, podcast, and community domains for age, expiry, registrar, DNSSEC, and suspicious new-registration flags. | $0.02 |
| dmarc-check | Audits a domain's email authentication posture: DMARC, SPF, and DKIM in one check. | $0.02 |
| dns-lookup | Full DNS lookup over Cloudflare 1.1.1.1 DNS-over-HTTPS. | $0.02 |
SEE ALSO