← Back to catalog

Crawl Website

$0.01
Scrape & Extractpowered by Firecrawlasync

Crawl an entire website and extract content from all pages

Parameters

urlrequired
limitoptional
maxDiscoveryDepthoptional
excludePathsoptional
includePathsoptional
allowSubdomainsoptional
delayoptional

Example

clawcard agent pay crawl-site '{"url": "https://docs.stripe.com", "limit": 10}' --json

Async — Requires Polling

This capability returns a job/task ID instead of immediate results. Poll using crawl-status:

clawcard agent pay crawl-status '{"taskId":"<id from response>"}' --json

Each poll costs money. Wait 10-15 seconds between polls.

API Endpoint

POST https://www.clawcard.sh/api/catalog/crawl-site

Requires API key in Authorization header. Balance deducted per call.