POST /v1/crawl

Website Crawling API

Read a website with a page budget. Create a job, collect saved page results and stop the work when you have enough.

curl -X POST https://api.luni.gg/v1/crawl \
  -H "Authorization: Bearer $LUNI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-crawl-0404309688f2" \
  -d '{
  "url": "https://example.com",
  "limit": 20,
  "maxDiscoveryDepth": 2,
  "scrapeOptions": {
    "formats": [
      "markdown"
    ]
  }
}'

Bound the work before it starts

A crawl accepts a seed URL, a page limit up to 100 and maxDiscoveryDepth from 0 to 5. It stays on the seed hostname and forward path, checks robots rules and follows your path-prefix filters. Start from the part of the site you actually need.

includePaths and excludePaths each accept up to ten path prefixes beginning with a slash. They are prefixes, not regular expressions. Combine a small page limit with a narrow path rather than relying only on depth to control the scope.

A job you can inspect and cancel

Crawl creation returns a job ID. Poll GET /v1/crawl/{id} for saved results and status. Use DELETE /v1/crawl/{id} to request cancellation. Cancellation preserves completed pages and releases the unused budget; it does not erase already completed work.

A job is asynchronous, so build your interface around state changes instead of waiting for an entire website in a single request. Webhooks are available when you prefer notification over repeated polling.

Credits follow successful pages

Luni reserves the maximum page cost before execution. Each successfully completed page settles one credit. Unused reservations are released when the job finishes or is cancelled. Polling and retrieving saved results do not add page charges.

Default mode uses public HTTP extraction. Interactive browser actions are outside the contract. Review a small sample before increasing the page budget, and design your application to handle partial results rather than assuming every discovered page will succeed.