Website Mapping API

Discovery comes before extraction

A map returns discovered same-site URLs, not the full content of each page. That makes it useful when you need to inspect a site's structure, select a documentation area or remove irrelevant paths before paying to fetch pages.

Start with a small limit and review the list. A discovered URL is a candidate, not a guarantee that the destination is accessible, useful or current. Your application decides which pages to scrape next.

Control how URLs are found

Set limit from 1 to 1,000. The sitemap option accepts include, only or skip: include combines discovery sources, only uses sitemap discovery and skip leaves sitemaps out. A successful map costs one credit; fetching the discovered pages is separate work.

Mapping is bounded discovery rather than a complete search-engine index. Sites with unusual navigation, missing sitemaps or client-rendered links may expose fewer pages than a person sees in a browser.

• Keep URLs with the paths relevant to your application.
• Deduplicate before scheduling extraction.
• Save your selected URL list with the time it was discovered.

Turn a map into a useful workflow

For a knowledge base, choose documentation and help pages while excluding account and checkout paths. For company research, choose the homepage, about and contact pages. For a content inventory, keep the URL list as an audit input rather than claiming it is a complete site census.

Use the scrape endpoint for individual selected pages, or create a bounded crawl when you want Luni to discover and retrieve pages in one asynchronous job.