Automate requests

From playground to code

Every run in the playground shows a Use it in code panel with cURL, Python, JavaScript and SDK snippets for exactly the request you just made. Start there, then move to the scripts below.

export LIVE_WEB_DATA_API_KEY=1337_live_…   # never hard-code keys
python examples/python/batch_from_csv.py examples/urls.csv
node examples/node/batch_from_csv.mjs examples/urls.csv

Process a list of URLs

The simplest loop: one request per URL, one after the other.

import os, requests

API = os.environ.get("LIVE_WEB_DATA_API_BASE_URL", "https://livewebdata.com/api")
headers = {"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"}

for url in ["https://example.com/a", "https://example.com/b"]:
    r = requests.post(f"{API}/v1/contents", json={"urls": [url]}, headers=headers, timeout=60)
    status = r.json()["statuses"][0]
    print(url, status["status"], status["error"] and status["error"]["tag"])

const API = process.env.LIVE_WEB_DATA_API_BASE_URL ?? "https://livewebdata.com/api";
const headers = { Authorization: `Bearer ${process.env.LIVE_WEB_DATA_API_KEY}`, "Content-Type": "application/json" };

for (const url of ["https://example.com/a", "https://example.com/b"]) {
  const res = await fetch(`${API}/v1/contents`, { method: "POST", headers, body: JSON.stringify({ urls: [url] }) });
  const { statuses } = await res.json();
  console.log(url, statuses[0].status, statuses[0].error?.tag ?? "");
}

You can also send up to 10 URLs per request — see Several URLs.

Respect rate limits

• Keep a steady pace of your plan's URLs per second (from GET /v1/account → rateLimit.requestsPerSecond). Bursts up to rateLimit.burst URLs (the bucket size, also sent as X-RateLimit-Limit) are fine; the bucket refills at that rate.
• Watch X-RateLimit-Remaining on each response: it is how many URLs you can send right now.
• On 429, wait Retry-After seconds and retry the same request.

Retries

Retry 429, 5xx and per-URL CRAWL_LIVECRAWL_TIMEOUT up to 3 times with exponential backoff and jitter (e.g. 0.5s · 2^n + random(0–0.25s)). Do not retry:

Send your own Idempotency-Key with every POST /v1/contents (one per logical request, for example per URL of your list) and send the same key on every retry of that request. A connection cut after the API received the request (a network error, a timeout, a deploy) is then retried without a second charge: the repeat returns that fetch's result. A 202 (still fetching) is repeated the same way, after Retry-After. Use a new key for a new attempt at a URL that already has an answer, such as a per-URL CRAWL_LIVECRAWL_TIMEOUT: the same key returns the same answer. See Long requests and Idempotency-Key.

The SDKs implement this policy for you.

Check your balance first

Call GET /v1/account before a large job and compare credits.total with the number of URLs. Stop cleanly when a request returns 402.

curl https://livewebdata.com/api/v1/account -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"

Scheduling

Any scheduler works — the scripts are plain processes that exit when done.

0 6 * * * cd /opt/jobs && LIVE_WEB_DATA_API_KEY=… /usr/bin/python3 batch_from_csv.py urls.csv --out results.jsonl

name: Fetch pages
on:
  schedule: [{ cron: "0 6 * * *" }]
  workflow_dispatch:
jobs:
  fetch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node examples/node/batch_from_csv.mjs examples/urls.csv --out results.jsonl
        env:
          LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}

Other schedulers (Cloud Scheduler, Kubernetes CronJob, n8n, Airflow) run the same command.

Keep your key safe

• Store it in an environment variable (LIVE_WEB_DATA_API_KEY) or your platform's secret manager.
• Never put it in client-side code, mobile apps, browser extensions or repositories.
• Use one key per environment (e.g. production, staging, n8n) — you can label keys in the dashboard and see the label in the usage log.
• Rotate a key immediately if it leaks: create a new one, then revoke the old one.