Web content for retrieval
Give your retrieval pipeline readable pages with traceable sources. Luni handles public-page extraction; your application controls chunking, embeddings and retrieval.
Choose a bounded corpus
Start with the documentation or help area that answers your users' questions. Map the website and inspect the candidate URLs, or crawl a specific seed path with a small page budget. Avoid mixing unrelated marketing, account and navigation pages into the knowledge base.
Keep the discovery list as an input to your ingestion job. Record why a source was included so you can remove or replace it later without rebuilding the entire collection blindly.
Keep source identity through the pipeline
Store the returned text with its source URL and your retrieval time. Split documents into chunks in your application, retaining a document-to-source relationship. Then create embeddings and write them to the vector or search store you already use.
Luni does not host an embedding model or vector database. Chunk sizes, overlap and indexing policy belong to your retrieval system. Test a small set of real questions before choosing those settings.
Refresh with evidence
Schedule refresh jobs in your own application. Compare the new document with your saved copy, replace stale chunks and remove sources that no longer belong in the corpus. Retain the source links used for an answer so readers can inspect the evidence.
A successful scrape is not proof that retrieval or generated answers are correct. Evaluate missing pages, content quality and citation accuracy independently. Public HTTP extraction may not capture content that only appears after browser interaction.