Web data glossary
The terms behind a web-data workflow, explained in the context of the operations Luni actually provides.
Scrape, map and crawl
Scraping retrieves one public page and extracts a requested representation of its content. Mapping discovers candidate URLs on a website without fetching all of their document content. Crawling combines bounded discovery and retrieval as an asynchronous job.
A seed URL is the starting point for discovery. A path prefix selects a part of the URL path; it is not a regular expression. Crawl depth describes how far discovery proceeds from the seed, while the page limit is a separate maximum work budget.
Markdown, HTML and source metadata
Markdown is a text representation of document structure that is useful for reading and retrieval. Processed HTML retains markup after extraction. rawHtml is the fetched page source. Links are discovered destinations that can inform another stage of the workflow.
Source metadata connects returned content to the page it came from. Preserve that association when storing documents, creating chunks or presenting a citation. The presence of a source URL makes review possible but does not establish that the page is authoritative.
Job, operation and webhook
An operation records a unit of API work and its accounting context. A job is work that continues asynchronously, such as a crawl or company profile. Polling asks for its current saved status; it is not a new extraction request.
A webhook is an HTTP notification sent to a receiver you provide. Signed delivery lets your application verify the sender's shared secret. At-least-once delivery means the same event may arrive again, so receivers must make duplicate handling safe.
Idempotency, retry and refresh
An idempotency key names a logical operation. Repeating the same key and unchanged body can retrieve a completed result without charging again. Reusing it with a different body is a conflict.
A retry attempts the same logical work after an interruption. A refresh requests a genuinely new observation and therefore needs a new key. Expiry describes a saved result that is no longer retained; it should not be silently treated as a request to rerun paid work.
Reservation, settlement and release
A reservation holds the maximum credit budget before execution. Settlement turns completed work into consumed credits. Release returns the unused held budget when work completes or is cancelled.
Available credits, held credits and consumed credits describe different states. A public estimator predicts an upper budget using the known schedule; actual usage depends on completed queries, profiles and successful pages.
Retrieval, RAG and provenance
Retrieval finds documents relevant to a question. Retrieval-augmented generation, often called RAG, passes retrieved material to a language model as context. Luni can supply the source documents, while your application owns chunking, embeddings, storage and retrieval evaluation.
Provenance is the record of where a value or document came from. For company research, a harvested contact is a value observed on a public page. It is different from an independently verified contact or a guarantee of deliverability.