Crawl a website#
POST /v1/crawl
Create an asynchronous job from a seed URL. The crawl stays on the seed hostname and its path or descendants, respects robots rules and uses a bounded page-attempt budget. A failed page attempt can use the attempt limit without consuming a page credit.
| Parameter | Type | Default | Description |
|---|---|---|---|
url | string | Required | Public seed URL. Use the site root to allow discovery across its paths. |
limit | integer | 10 | Maximum page attempts, from 1 to 100. |
maxDiscoveryDepth | integer | 3 | Link depth from the seed, from 0 to 5. |
includePaths | string[] | Omitted | Up to 10 path prefixes beginning with /; each at most 100 characters. |
excludePaths | string[] | Omitted | Up to 10 excluded path prefixes with the same bounds. |
scrapeOptions | object | Omitted | Supported scrape settings for each retrieved page. |
webhook | object | Omitted | Public callback URL and signing secret. See Webhooks. |
curl -X POST https://api.luni.gg/v1/crawl \
-H "Authorization: Bearer $LUNI_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: crawl-site" \
-d '{"url":"https://example.com","limit":10,"maxDiscoveryDepth":2}'{
"success": true,
"id": "<operation-id>",
"operationId": "<operation-id>",
"url": "https://api.luni.gg/v1/crawl/<operation-id>"
}curl "https://api.luni.gg/v1/crawl/$OPERATION_ID?skip=0&limit=25" \
-H "Authorization: Bearer $LUNI_API_KEY"
curl -X DELETE https://api.luni.gg/v1/crawl/$OPERATION_ID \
-H "Authorization: Bearer $LUNI_API_KEY"The poll route accepts skip ≥ 0 and limit from 1–100, returning a next URL when another result page exists. Its queued/running status is reported as scraping. total is the reserved page budget, not the number of discovered pages; completed counts currently retained results.
Cancellation preserves already completed results and charges, stops future work and releases unused reservations. Polling and cancellation do not add extraction credits. Subdomain expansion, external links, regular-expression path filters and robots bypass are outside the supported contract.