Plan website discovery
A useful website collection starts with a deliberate URL list. Discover candidates, inspect the paths and decide what your application actually needs.
Start from a specific purpose
Write down whether you want a help center, a documentation section, company information or a broad content inventory. Those goals produce different URL selections. A broad homepage seed is convenient, but it can also surface paths that add no value to your application.
Choose a public seed and a small initial discovery limit. Mapping finds URLs rather than fetching every page's content. It lets you inspect the candidate set before spending credits on downstream extraction.
Run a bounded map
The request below asks for up to 100 same-site URLs and includes sitemap discovery. The API accepts limits up to 1,000. sitemap can be include, only or skip, depending on whether you want combined discovery, sitemap-only results or page discovery without sitemaps.
A successful map costs one credit. It does not include the cost of scraping every discovered URL, and a larger limit does not guarantee that the site exposes that many useful pages.
Review the candidates
Keep paths that fit the purpose and remove account, checkout or unrelated destinations. Deduplicate the selected URLs and save the list with the discovery time. If a source URL changes, your application should be able to explain which document it replaced.
Discovery can miss links hidden behind JavaScript or unusual navigation. Review a representative sample against the website rather than describing the result as a complete index. A discovered link is a candidate; access and extraction may still fail.
Choose the next operation
Use separate scrape requests when you have a curated list and want precise control over each page. Use a crawl when the goal is bounded retrieval within a seed hostname and path. A crawl also provides job state, partial results and cancellation.
Before scaling, retrieve a small sample and verify the content format, source metadata and actual credit usage. Fix selection mistakes at the discovery stage instead of compensating for irrelevant documents later in the retrieval pipeline.