Web content ingestion

Bring public website content into your application with a workflow you can resume, inspect and budget.

curl -X POST https://api.luni.gg/v1/map \
  -H "Authorization: Bearer $LUNI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-map-1fe27a39e50b" \
  -d '{
  "url": "https://example.com",
  "limit": 100
}'

Define what belongs in the collection

Write down the hosts and paths you want to ingest before running discovery. Map the site to inspect candidate URLs, then select individual pages or create a bounded crawl. Exclude destinations that have no value for your application, such as checkout or account flows.

Keep source identity separate from the stored document ID. That lets your application update a document when the same source is retrieved again without accidentally creating a new record for every retry.

Make progress durable

Record each logical operation and its idempotency key in your own queue. Save successful documents to your storage before marking the ingestion step complete. A worker can then retry a failed step without making a completed Luni operation billable twice.

For crawls, poll the job or receive a webhook and retrieve saved page results. Results are retained for seven days; your application is responsible for longer-term storage. Expiry should trigger an explicit decision rather than a silent paid rerun.

Separate extraction from processing

Normalize, chunk, classify or embed the documents after retrieval, using your own processing stack. Keep an original copy with the source URL so a later transformation can be checked or rerun without losing provenance.

Scheduled crawls, destination storage, vector indexing and downstream transformations are not hosted ingestion services in this release. Luni supplies the bounded extraction step and its operation history; your application owns the rest of the pipeline.