$ man sitemap-fetch
/sitemap-fetch(1)
PRICE / CALL
$0.005
USDC · base mainnet · scheme: exact
METHOD
POST
CLUSTER
webprobeCATEGORY
uncategorized
STATUS
● live
NAME
sitemap-fetch — sitemap fetcher / sitemap.xml parser / sitemap index resolver / seo sitemap reader / robots.txt sitemap discovery / website url inventory
SYNOPSIS
POST https://x402.org/v1/sitemap-fetch
Content-Type: application/json
X-PAYMENT: <signed-transferWithAuthorization>
{ ... }↳ first call →
402 Payment Required. Sign USDCtransferWithAuthorization, retry with theX-PAYMENT header.DESCRIPTION
Sitemap fetcher / sitemap.xml parser / sitemap index resolver / SEO sitemap reader / robots.txt sitemap discovery / website URL inventory. Accepts a site root (will discover sitemap via robots.txt or /sitemap.xml convention) OR a direct sitemap.xml URL. Recurses through sitemap-index nesting. Returns the URL list with lastmod / changefreq / priority and aggregate stats (count, oldest/newest lastmod). Useful for SEO audits, content-freshness monitoring, RAG ingestion seeding.
INPUT — request schema
| property | type | description | req? |
|---|---|---|---|
| url | string | Site root (e.g. https://example.com) or direct sitemap URL. | required |
| limit | number | Max URL rows. 1-5000. Default 1000. | optional |
| recurse | boolean | Recurse into sitemap-index children. Default true. | optional |
| user_agent | string | Optional User-Agent header. | optional |
OUTPUT — response shape
| field | type | description |
|---|---|---|
| input_url | string | Site root or sitemap URL the caller passed in, echoed back for reference. |
| sitemaps_fetched | string | List of sitemap URLs actually retrieved, including nested sitemap-index children that were resolved. |
| url_count | string | Total number of URL entries found across all fetched sitemaps after recursion. |
| urls | string | Array of URL entries with loc, lastmod, changefreq, and priority fields parsed from the sitemap XML. |
| lastmod_oldest | string | Oldest lastmod timestamp seen across all URL entries, useful for content-freshness checks. |
| lastmod_newest | string | Newest lastmod timestamp seen across all URL entries, indicating most recent site update. |
| truncated | string | Boolean flag set when the URL list hit an internal cap and not every entry was returned. |
| bytes_total | string | Total bytes downloaded across every sitemap and sitemap-index file fetched during the call. |
| errors | string | Array of per-sitemap fetch or parse errors encountered, empty when all sitemaps loaded cleanly. |
EXAMPLES — two ways to call
EXAMPLE 1 · curl
curl -X POST https://x402.org/v1/sitemap-fetch \
-H 'Content-Type: application/json' \
-d '{ }'first response =
402 Payment Required with payment requirements; sign + retry with X-PAYMENT.EXAMPLE 2 · mcp
# install once claude mcp add x402 --command "npx x402-deployer-mcp" # then ask Claude Code: # "use the sitemap-fetch tool to ..."
MCP server handles payment automatically — your coding agent just calls the tool by name.
METADATA
- tags
- web-probesitemapseocrawlerrobots-txturl-inventorysitemap-xml
- methods
- POST
- cluster
- webprobe
- price
- $0.005 USDC per call
ADJACENT — other endpoints in webprobe
| endpoint | description | price |
|---|---|---|
| disposable-email-check | Disposable email detector / fake email filter / fraud signup defense. | $0.005 |
| domain-availability | Domain availability / RDAP lookup / WHOIS replacement / domain registration checker / expiry date lookup. | $0.005 |
| github-stars-history | GitHub stars history / star growth chart / repo popularity over time / trending tracker. | $0.005 |
| link-extract | Link extractor / extract links from URL / list all anchor links / page outlink crawler / scrape outbound links / get hrefs from page. | $0.005 |
| mcp-tools-list | MCP tools-list / Model Context Protocol discovery / list MCP server tools / tools/list JSON-RPC / agent capability discovery / MCP catalo… | $0.005 |
| wikipedia | Wikipedia API / encyclopedia lookup. | $0.005 |
| wikipedia-lookup | Wikipedia API / encyclopedia lookup / structured article fetch. | $0.005 |
| homoglyph-check | Homoglyph attack detector / Unicode lookalike scanner. | $0.003 |
SEE ALSO