Source records#

Source records return one structured record from a supported public page, such as a video, job listing or app. Each source and resource pair has a fixed definition: the hosts and record path it accepts and the fields it returns. Browse every definition in the data API catalogue.

GET /v1/data

Returns the full catalogue as {"data": [definition, …]}. No API key is needed. The response carries an ETag; send it back in If-None-Match and an unchanged catalogue returns 304 Not Modified with no body.

curl https://api.luni.gg/v1/data

GET /v1/data/{source}/{resource}

Returns one definition under data. No API key is needed. Use it to check which URLs a resource accepts and which fields it returns before you spend a credit.

curl https://api.luni.gg/v1/data/youtube/videos

{
  "data": {
    "id": "youtube.videos",
    "source": "youtube",
    "resource": "videos",
    "title": "YouTube - Videos",
    "category": "Social",
    "status": "public_html_beta",
    "hosts": [
      "youtube.com",
      "www.youtube.com",
      "m.youtube.com"
    ],
    "pathPattern": "^/(?:watch|shorts/[A-Za-z0-9_-]{11})/?$",
    "exampleURL": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "fields": [
      {
        "name": "title",
        "type": "string",
        "required": true
      },
      {
        "name": "description",
        "type": "string",
        "required": true
      },
      {
        "name": "duration",
        "type": "string",
        "required": false
      },
      {
        "name": "upload_date",
        "type": "string",
        "required": false
      },
      {
        "name": "thumbnail_url",
        "type": "string",
        "required": false
      },
      {
        "name": "views",
        "type": "integer",
        "required": false
      }
    ],
    "required": [
      "title",
      "description"
    ],
    "endpoint": "POST /v1/data/youtube/videos"
  }
}

Key	Meaning
`id`	source.resource, for example youtube.videos.
`source, resource`	The path segments for the POST route.
`title, category`	Display name and catalogue group.
`status`	Availability of the resource.
`limitation`	Present only when requests to this source usually or sometimes fail, for example behind a bot check or a login. It says why, and failed requests repeat it in their error. Failed requests are not charged.
`hosts`	Exact hostnames a request URL may use.
`pathPattern`	Regular expression the URL path must fully match.
`exampleURL`	A URL the resource accepts.
`fields`	Every field the record can contain, with name, type and required.
`required`	Fields that must be present for the request to succeed.
`endpoint`	The POST route that retrieves a record.

POST /v1/data/{source}/{resource}

Retrieves one record. Authenticate with an API key that has the scrape scope, sent as Authorization: Bearer luni_… or X-API-Key. The request runs synchronously and returns within about 30 seconds.

Parameter	Type	Default	Description
`url`	string	Required	The record's public URL. The body must contain only this key.

The URL must use one of the definition's hosts and its path must match pathPattern. Query parameters are rejected except common tracking parameters such as utm_source, and the record ID for sources that keep it in the query:

Source	ID parameter
YouTube	`v`
Indeed	`jk`
Glassdoor	`jl`
Google Play	`id`
Meta Ad Library	`id`

curl -X POST https://api.luni.gg/v1/data/youtube/videos \
  -H "Authorization: Bearer $LUNI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: youtube-video" \
  -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'

{
  "success": true,
  "data": {
    "source": "youtube",
    "resource": "videos",
    "record": {
      "title": "<title>",
      "description": "<description>",
      "duration": "<duration>",
      "thumbnail_url": "<thumbnail-url>",
      "views": "<views>"
    },
    "missingFields": [
      "upload_date"
    ],
    "sourceURL": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "retrievedAt": "<retrieved-at>"
  },
  "operationId": "<operation-id>",
  "creditsUsed": 1
}

Values above are placeholders. record holds the fields that were found; any optional field that was not found is left out of record and listed in missingFields. If a required field is missing, the whole request fails with 424. There is no partial record.

Credits and retries

A successful record costs 1 credit. Failed and cancelled requests cost 0. The Idempotency-Key header is optional: a retry with the same key and body returns the original result without charging again, so network retries never cost twice. See Retries.

Errors

Errors use the standard error shape. When work was admitted, the response includes operationId.

HTTP	Code	Meaning
202	`OPERATION_PENDING`	You replayed an Idempotency-Key whose request is still running. Wait for Retry-After, then retry or inspect the operation.
400	`BAD_REQUEST`	The body is not exactly {"url": …}, or the URL does not match the source's host, record path or query rules.
401	`UNAUTHORIZED`	The API key is missing, expired or revoked.
402	`INSUFFICIENT_CREDITS`	The workspace does not have the one credit the request reserves.
403	`FORBIDDEN`	The key does not have the scrape scope.
404	`NOT_FOUND`	The source or resource is not in the catalogue.
409	`CONFLICT`	The Idempotency-Key was already used for this source and resource with a different URL.
409	`OPERATION_CANCELLED`	The operation was cancelled. Use a new Idempotency-Key to start again.
410	`RESULT_EXPIRED`	A replay found a completed operation whose result is no longer retained.
424	`SERVICE_UNAVAILABLE`	No complete record could be retrieved. The response includes operationId and costs 0 credits.
429	`RATE_LIMIT_EXCEEDED`	Too much work is running. Wait for Retry-After (60 seconds).

MCP

Agents connected over MCP use the same catalogue: luni_list_sources lists source and resource names, luni_source_schema returns one definition, and luni_get_source_record retrieves a record with the same URL rules and credit cost.