Run a bounded crawl

Collect a website section without letting the job grow beyond its purpose. Set the page budget first, then watch the saved results.

Start building Read the docs

Set the page and path boundary

Choose a seed inside the part of the website you want to read. Set limit between 1 and 100 and maxDiscoveryDepth between 0 and 5. The crawler stays on the seed hostname and forward path and respects robots rules.

Use includePaths and excludePaths for path-prefix filtering. Each list allows up to ten entries beginning with a slash. These are plain prefixes, not regular expressions. A narrow seed and a small page budget are easier to inspect than a broad crawl followed by extensive filtering.

curl -X POST https://api.luni.gg/v1/crawl \
  -H "Authorization: Bearer $LUNI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-crawl-b9668b938505" \
  -d '{
  "url": "https://example.com/docs",
  "limit": 20,
  "maxDiscoveryDepth": 2,
  "excludePaths": [
    "/docs/archive"
  ],
  "scrapeOptions": {
    "formats": [
      "markdown"
    ]
  }
}'

Persist the job identity

Creation returns a job ID. Save it with the request body and idempotency key in your application. Poll GET /v1/crawl/{id} to observe the job and retrieve saved pages. The endpoint also supports skip and limit for paginated reads.

Do not interpret a temporary lack of completed pages as proof that the job failed. Read the job status and handle partial results. Your interface can show progress while successful pages are stored rather than waiting for the entire website to finish.

Cancel when you have enough

Send DELETE /v1/crawl/{id} to request cancellation. Completed pages remain available and the unused reserved budget is released. A cancellation is not a reversal of already completed page charges.

If a client loses its connection during creation, retry with the same idempotency key and unchanged body. Avoid starting a second job merely because your first HTTP response did not arrive.

Close the accounting loop

The maximum page budget is reserved before execution. Each successfully completed page settles one credit. Failed pages do not add successful-page charges; polling and reading the saved result do not create a new page charge.

Copy the results to your own storage if you need them beyond seven days. For larger ingestion workflows, use your own scheduler and queue around several bounded operations rather than assuming a single crawl is a permanent mirror of the site.