# MCPScraper: guides and posts > Web scraping MCP server comparison on one published rubric: browser, crawl, self-hosted and content-API servers, with linked sources, check dates, no scores. # Best MCP server for scraping: the rubric, not a ranking Source: https://mcpscraper.net/guides/best-mcp-server-for-scraping > How we compare scraping MCP servers: a ten-row rubric, every table sourced and dated, and seven servers in category tables. No scores, no rankings. Most pages that answer "best MCP server for scraping" are rankings. This one is a rubric. A ranking tells you who won; a rubric tells you why a server fits one job and not another, and lets you check every claim yourself. The tables below cover seven servers in four categories. Each value comes from the project's own documentation and was checked on 2026-10-27. ## How we check We read each project's README, docs and recent release notes. A value goes into the table only if a primary source states it. Where the docs are silent, the cell says so instead of guessing. We do not run speed or accuracy benchmarks, so there are none here, and we do not copy numbers from other people's benchmarks into the tables. Every table has a check date. MCP servers change fast: Firecrawl's MCP server, for example, now lists a full profile of more than 25 tools, many more than its first releases. A row older than 90 days, or older than a release that touches it, gets re-checked. ## Rubric v1 explained The rubric has ten rows. They are the questions a team actually argues about before standardizing on a server. 1. **Transport and auth.** Local stdio, a remote Streamable HTTP URL, or both. Remote servers need a key, a bearer token or OAuth. This decides whether a server works in your client and whether it can run without anything installed. 2. **Tool surface.** A client that loads tools upfront describes every registered tool to the model in each session; Claude Code defers them by default through tool search. Fewer tools means less context spent before the first page is read. Some servers let you include or exclude tools in the URL. 3. **Output.** Markdown, text, HTML or JSON; main content only or the whole page; and a per-call cap on characters or tokens. 4. **Failure behavior.** Does the error tell the model what to do next? Does a failed call cost anything? 5. **Freshness.** Live fetch every time, a cache with a maximum age, or no control at all. 6. **Scope.** JavaScript rendering, interaction, crawling, search and structured extraction. More scope is not better if your job does not need it. 7. **Hosting and license.** Hosted only, self-hosted only, or both, and under which license. 8. **Pricing shape.** Credits per page, pay per event, browser minutes, or your own machines. See the [pricing models guide](/guides/scraping-mcp-pricing-models). 9. **Compliance posture.** An acceptable-use policy, a stated position on robots.txt, and what is logged. 10. **Maintenance.** The date of the last release and how active the repository is. ## Category tables (checked 2026-10-27) Every table has all ten rubric rows plus a source row. "Not documented" means the source linked in that column does not state the value; it is not a judgment about the project. ### Browser-control servers | Row | Playwright MCP | Scrapling MCP | | --- | --- | --- | | Transport and auth | stdio by default; HTTP with `--port` | stdio by default; Streamable HTTP with `--http` and an auth token | | Tool surface | About 25 core tools (`browser_navigate`, `browser_snapshot`, `browser_click` …); optional groups via `--caps` | 16 tools, from `make_request` to session and browser tools | | Output | Accessibility snapshots; screenshots as a separate tool | Markdown, HTML, text or an AI snapshot; main content only by default | | Failure behavior | Not documented | Not documented | | Freshness | Live page in a browser; no cache control documented | Live request; no cache control documented | | Scope | Interaction and JavaScript pages; no crawl tool | Requests, sessions and browser fetches; Markdown extraction | | Hosting and license | Your machine or Docker; Apache-2.0 | Your machine; BSD-3-Clause | | Pricing shape | Free to run; you pay compute | Free to run; you pay compute | | Compliance | No robots.txt stance documented | HTTP mode requires a token by default; no robots.txt stance documented | | Maintenance | Latest release 2026-09-28 | Latest release 2026-10-17 | | Source | [README](https://github.com/microsoft/playwright-mcp), [releases](https://github.com/microsoft/playwright-mcp/releases) | [MCP docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server/), [releases](https://github.com/D4Vinci/Scrapling/releases) | ### Crawl platforms | Row | Firecrawl MCP | Apify MCP | | --- | --- | --- | | Transport and auth | stdio, local Streamable HTTP, hosted URL; API key, OAuth endpoint for hosted use | Hosted Streamable HTTP at mcp.apify.com, or stdio via npx; OAuth or bearer token | | Tool surface | Full profile lists 25+ tools: scrape, map, crawl, search, interact, agent, monitors | Default set plus any Actor you add; the `tools` parameter narrows it | | Output | `formats` option, `onlyMainContent` | Whatever each Actor returns | | Failure behavior | Rate-limit errors surfaced to the client; automatic retries | Depends on the Actor | | Freshness | Not documented in the README | Not documented | | Scope | Scrape, map, crawl, search, browser interaction | Whatever the added Actors do | | Hosting and license | Hosted, or self-hosted via `FIRECRAWL_API_URL`; MCP server MIT, core AGPL-3.0 | Hosted or local stdio; some Actors only via the hosted server; MIT | | Pricing shape | Credits per request; error-status pages cost 1 credit | Per Actor: pay per event or pay per usage | | Compliance | No robots.txt stance in the README | Not documented in the README | | Maintenance | Latest MCP server release 2026-08-26 | Not documented | | Source | [README](https://github.com/firecrawl/firecrawl-mcp-server), [pricing](https://www.firecrawl.dev/pricing), [releases](https://github.com/firecrawl/firecrawl-mcp-server/releases) | [README](https://github.com/apify/apify-mcp-server), [pricing models](https://docs.apify.com/platform/actors/running/actors-in-store) | ### Self-hosted crawler | Row | Crawl4AI (Docker server) | | --- | --- | | Transport and auth | SSE at `/mcp/sse` and WebSocket at `/mcp/ws` | | Tool surface | 7 tools: `md`, `html`, `screenshot`, `pdf`, `execute_js`, `crawl`, `ask` | | Output | Markdown or HTML per page, plus screenshots and PDFs | | Failure behavior | Not documented on the self-hosting page | | Freshness | Not documented on the self-hosting page | | Scope | Crawling, JavaScript execution, screenshots | | Hosting and license | Your infrastructure; Apache-2.0, with an attribution clause in the README | | Pricing shape | Free to run; you pay compute | | Compliance | Not documented on the self-hosting page | | Maintenance | v0.9.5 released 2026-10-14 | | Source | [Self-hosting guide](https://docs.crawl4ai.com/core/self-hosting/), [releases](https://github.com/unclecode/crawl4ai/releases) | ### Single-page content APIs | Row | Jina MCP | Live Web Data API | | --- | --- | --- | | Transport and auth | Remote Streamable HTTP; bearer key optional for some tools | Remote Streamable HTTP, bearer key | | Tool surface | 12 tools; filter with `include_tools` / `exclude_tools` | 2 tools: `get_page_contents`, `get_account_status` | | Output | `read_url` returns Markdown; `max_tokens` caps it | Markdown, text or HTML; `max_characters` per call | | Failure behavior | Timeouts of 30 or 60 seconds; the README notes blocked pages can come back as ordinary content | One actionable error sentence; failed calls not charged | | Freshness | Not documented in the README | Always live; no cache | | Scope | Read, search (key needed), screenshots, reranking | One public page per call; the MCP tool has no JavaScript rendering, crawl, search or extraction | | Hosting and license | Hosted; repository Apache-2.0 | Hosted only | | Pricing shape | Billed in tokens; keyless use at lower rate limits | Credits per successful page | | Compliance | Not documented in the README | Acceptable-use policy; honoring robots.txt is not promised ([Terms](/terms), Section 8.6) | | Maintenance | Not documented | Not documented (hosted service, no public release log) | | Source | [README](https://github.com/jina-ai/MCP) | [MCP docs](/go/docs-mcp?from=guide:best-mcp-server-for-scraping) | ## What this means for your server choice The tables do not crown a winner, because the categories answer different jobs. If your agent clicks and types, the browser rows matter. If it maps a site, look at the crawl platforms, and if you are leaving one, the [Firecrawl MCP alternatives guide](/guides/firecrawl-mcp-alternatives) sorts the options by reason. If data must stay on your machines, read the [hosted vs self-hosted guide](/guides/self-hosted-vs-hosted-scraping-mcp). If it reads known public pages, the content APIs keep the context smallest; the [output and token cost guide](/guides/scraping-mcp-output-and-token-cost) shows why. For the compliance row, the [responsible scraping guide](/guides/responsible-scraping-with-mcp) turns it into a review checklist. The fastest way to a shortlist is the [pick-by-job guide](/guides/choose-a-scraping-mcp-server-by-job). ## Changelog - **Rubric v1, 2026-10-27.** First published version: ten rows, seven servers, four categories. - **2026-10-28.** Tool-surface row description: tool definitions reach the model in every session only in clients that load them upfront; Claude Code defers them by default through tool search. Table values unchanged. ## Where Live Web Data API sits in these tables Live Web Data API is checked on the same ten rows as the other servers, including the rows where it falls short: no JavaScript rendering in the MCP tool, no crawl and no search. If reading known public pages is your job, [read the MCP docs](/go/docs-mcp?from=guide:best-mcp-server-for-scraping). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. --- # Which scraping MCP server to use: pick by the job Source: https://mcpscraper.net/guides/choose-a-scraping-mcp-server-by-job > A decision guide from job to category to shortlist: when a browser MCP server fits, when a crawl platform fits, and when a small page-reading server is enough. "Which scraping MCP server should I use?" has no answer until you say what the agent is supposed to do. A browser server, a crawl platform, a self-hosted crawler and a single-page content API are four different tools. They overlap at the edges, but each one is the right fit for a different job and an expensive fit for the others. This guide starts from the job. ## Six common jobs 1. **A coding agent reads documentation.** Claude Code or Cursor needs one library page, a changelog or an API reference as text while it writes code. The URLs are known or easy to guess. 2. **An agent answers questions from a few public pages.** A support or research agent pulls three to ten known public pages into context and cites them. 3. **An agent steps through a flow.** It clicks, types and waits: testing your own web app, reproducing a bug, filling a form in a staging environment. 4. **A team maps a whole public docs site.** It needs every page under a path, not one page at a time. 5. **Data must stay on your own machines.** Policy says no third-party service sees the URLs or the content. 6. **Somebody wants pages from a specific named website at scale.** This one is out of scope here; check the site's terms and API first, and read the [responsible scraping guide](/guides/responsible-scraping-with-mcp). ## Decision table | Job | Category | Why | Shortlist from the rubric | | --- | --- | --- | --- | | Read docs in a coding agent | Single-page content API | Few tools, capped output, no local browser | Jina MCP, Live Web Data API | | Answer from a few known pages | Single-page content API | One URL per call, Markdown or text | Jina MCP, Live Web Data API | | Step through a flow | Browser control | Real clicks and typing | Playwright MCP, Scrapling MCP | | Map a public docs site | Crawl platform | Map and crawl tools, status checks | Firecrawl MCP, Apify MCP | | Keep everything local | Self-hosted crawler or browser | Runs on your machines | Crawl4AI, Playwright MCP, Scrapling MCP | The full rows for each server are in the [rubric guide](/guides/best-mcp-server-for-scraping). ## Choose it when, skip it when ### Single-page content API **Choose it when** the agent reads known public URLs and you care about context size. These servers return the main content of one page as Markdown or text, usually with a size cap. Jina's server lets you filter its 12 tools down in the URL; Live Web Data API's exposes two. The [output and token cost guide](/guides/scraping-mcp-output-and-token-cost) shows how much context each choice saves. **Skip it when** the page only shows content after JavaScript runs and your chosen API does not render it, or when the job needs clicks, crawling or search. Live Web Data API, for example, does not render JavaScript, crawl, search or extract structured fields. ### Browser-control server **Choose it when** the job is interaction. Playwright MCP drives a real browser through the page's accessibility tree, which is exactly right for form flows and UI testing. **Skip it when** the job is reading. About 25 tools, a browser process and a snapshot after each step are a heavy way to fetch a docs page. It is the mismatch to check for first. ### Crawl platform **Choose it when** you need many pages from one site: a map of URLs, a crawl with a status check, or batch scraping. Firecrawl and Apify both offer hosted MCP endpoints for this. Already on Firecrawl and unhappy with the fit? The [Firecrawl MCP alternatives guide](/guides/firecrawl-mcp-alternatives) sorts the options by your reason to switch. **Skip it when** you only ever read a handful of known URLs. In a client that loads tools upfront, a large tool surface then costs context in every session for features the agent never calls, unless you narrow it. ### Self-hosted crawler **Choose it when** URLs and content must not leave your infrastructure, or you want to change the code. Crawl4AI's Docker server exposes MCP over SSE and WebSocket; Scrapling and Playwright MCP run locally. **Skip it when** nobody on the team wants to own updates, browser versions, memory limits and the request-rate policy of the machine it runs on. ## Setting up the server you picked This site compares servers; it does not walk through client configuration. For step-by-step setup in Claude Code, Cursor and other clients, the [remote MCP server setup guide on fetchmcp.net](https://fetchmcp.net/guides/connect-a-remote-mcp-server) covers each one. Once a server is connected, run the test from the rubric: one known page and one known 404 on example.com, then compare the output size and the error text. ## If a content API fits Live Web Data API is one of the content APIs in the decision table. Its MCP server does not render JavaScript, crawl, search or extract structured fields. If reading known public pages is your job, [read the MCP docs](/go/docs-mcp?from=guide:choose-a-scraping-mcp-server-by-job). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. --- # Firecrawl MCP alternatives, sorted by why you are switching Source: https://mcpscraper.net/guides/firecrawl-mcp-alternatives > Firecrawl MCP alternatives for four reasons to switch: pricing shape, self-hosting, a smaller tool surface, or real browser interaction. Rubric rows, dated. Firecrawl's MCP server is one of the most capable scraping servers available: scrape, map, crawl, search, browser interaction, agents and monitors, behind one hosted URL or a local process. That breadth is also why people look for a Firecrawl MCP alternative. The right replacement depends entirely on why you are leaving, so this guide is organized by reason, not by brand. All rows were checked against each project's docs on 2026-10-27. ## Why people switch Four reasons cover nearly every switching question: 1. **The pricing shape does not match the workload.** Firecrawl bills in credits per request, with different costs per endpoint. Its pricing page also states that a page returning an error status such as 403 or 404 is still returned and costs one credit. 2. **You want to run it yourself.** Policy, data residency or cost pushes you to self-host. 3. **The tool surface is bigger than the job.** The README lists a full profile of more than 25 tools. If your agent only reads known pages, most of those tools go unused, and in a client that loads tools upfront their descriptions still take context. 4. **You need a real browser.** Firecrawl has interact tools, but some jobs are interaction from start to finish. ## Alternatives by reason ### Different pricing shape Apify's MCP server turns Actors into tools. Actors on Apify use their own pricing models, such as pay per event or pay per usage, set per Actor. That fits teams that want a specific ready-made Actor more than a general crawler. A single-page content API is the other option: it charges per page read and, in Live Web Data API's case, nothing for failed requests. Read the [pricing models guide](/guides/scraping-mcp-pricing-models) before deciding, because the cheapest shape depends on your failure rate and page count. ### Run it yourself The smallest change is to keep Firecrawl MCP and point it at your own Firecrawl with `FIRECRAWL_API_URL`. Note the license: the MCP server is MIT, Firecrawl's core is AGPL-3.0, and the cloud version has features the README does not list for self-hosting. The main open-source alternative is Crawl4AI (Apache-2.0, with an attribution clause), whose Docker server exposes MCP tools including `md`, `crawl` and `execute_js`. The [hosted vs self-hosted guide](/guides/self-hosted-vs-hosted-scraping-mcp) covers the operating work you take on. ### Fewer tools in context If your agent reads public pages whose URLs it already knows, a single-page content API keeps the context small. Jina's remote MCP server has 12 tools but lets you filter them with `include_tools` and `exclude_tools`, and caps `read_url` output with `max_tokens`. Live Web Data API exposes two tools and a `max_characters` setting. Neither is a crawler: you lose map, crawl and batch features. Live Web Data API also does not render JavaScript or search. ### Clicks and forms When the job is interaction, use a browser-control server. Playwright MCP (Apache-2.0) drives a browser through accessibility snapshots, and Scrapling's MCP server offers sessions, browser actions and Markdown extraction. Both run on your machine. ## Rubric rows side by side | Row | Firecrawl MCP | Crawl4AI | Apify MCP | Playwright MCP | Live Web Data API | | --- | --- | --- | --- | --- | --- | | Transport | stdio, local HTTP, hosted | SSE, WebSocket | Hosted HTTP, stdio | stdio, HTTP | Hosted HTTP | | Tools | 25+ (full profile) | 7 | Configurable | About 25 | 2 | | Crawl | Yes | Yes | Via Actors | No | No | | JavaScript pages | Yes | Yes | Depends on Actor | Yes | No | | License | MIT (MCP), AGPL-3.0 (core) | Apache-2.0 | MIT | Apache-2.0 | Hosted service | | Failed calls | Error-status pages cost 1 credit | Your infra | Depends on Actor | Your infra | Not charged | For the full ten-row rubric, see the [rubric guide](/guides/best-mcp-server-for-scraping). ## If a content API fits Live Web Data API appears in this comparison under the same rules as the other servers. Its MCP server does not render JavaScript, crawl or search. If your reason to switch is a smaller tool surface for reading public pages, [read the MCP docs](/go/docs-mcp?from=guide:firecrawl-mcp-alternatives) or [see the pricing page](/go/pricing?from=guide:firecrawl-mcp-alternatives). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. --- # Self-hosted scraping MCP server or hosted: what you trade Source: https://mcpscraper.net/guides/self-hosted-vs-hosted-scraping-mcp > Run a scraping MCP server yourself or use a hosted one? Licenses, operating effort, data handling and request-rate policy compared, with a checklist. A self-hosted scraping MCP server runs on your laptop, your server or your cluster. A hosted one runs on a provider's infrastructure and you connect over a remote URL. Both put web pages into your agent's context. The difference is who does the operating work, who sees the data and which license applies. ## What you run yourself Three open-source options cover most self-hosting questions, all checked against their docs on 2026-10-27: | Server | How it runs | Transport | License | | --- | --- | --- | --- | | Crawl4AI | Docker server with MCP endpoints | SSE at `/mcp/sse`, WebSocket at `/mcp/ws` | Apache-2.0, with an attribution clause in the README | | Playwright MCP | `npx` process or Docker image, launches a browser | stdio; HTTP with `--port` | Apache-2.0 | | Scrapling MCP | `scrapling-mcp` Python process | stdio; Streamable HTTP with `--http` | BSD-3-Clause | | Firecrawl (self-hosted) | Docker Compose, with the MCP server pointed at it | MCP server: stdio or local HTTP | Core AGPL-3.0, MCP server MIT | What you take on: - **Browsers.** Crawlers and browser servers start real browser processes. Memory, crashes and version upgrades are now your problem. - **Network exposure.** An HTTP transport listening on `0.0.0.0` without auth lets anyone on the network make your machine fetch pages. Scrapling's HTTP mode requires a token by default; keep it that way. Bind other servers to localhost unless you have a reason not to. - **Request-rate policy.** Requests leave from your own machines, so setting polite per-site rates and stopping on 403 and 429 responses is your job. Careless use can get your address rate limited, which is a fair response from the site. Keep request rates polite and read the [responsible scraping guide](/guides/responsible-scraping-with-mcp). - **Licenses.** Apache-2.0 and BSD-3-Clause are permissive. AGPL-3.0 has obligations when you offer modified software over a network. Crawl4AI's README asks for attribution. Have someone read the license before you ship a product on it. This is not legal advice. ## What a hosted server takes off your hands A hosted server is a URL and a key. Firecrawl, Apify, Jina and Live Web Data API all offer remote MCP endpoints over Streamable HTTP. You skip browsers, patching and capacity planning, and you accept the provider's acceptable-use policy. What you give up: - **Control over the code.** You cannot patch a parser or add a tool. - **Data locality.** The provider processes every URL and every page it returns. - **Cost predictability on your terms.** You pay in the provider's unit: credits, events or pages. The [pricing models guide](/guides/scraping-mcp-pricing-models) compares those units. - **Access to everything.** Reviewed sign-ups, restricted targets and acceptable-use rules limit what a hosted service will fetch. For most teams that is a feature, not a bug. ## Privacy and data handling Ask three questions of either option: 1. **Where are URLs and content logged, and for how long?** For self-hosted servers, check your own logs and the server's debug output. For hosted ones, read the privacy notice. 2. **Who can call the server?** A local stdio server is reachable only by the client that started it. A remote server is reachable by anyone with a key, so keys need rotation and per-environment scoping. 3. **What personal data could pass through?** Public pages can still contain names and contact details. Keep the agent away from pages that exist to profile people. ## Decision checklist Self-host when at least two of these are true: - Content or URLs must not reach a third party. - Someone on the team will own the server, with time set aside for it. - You need to change the code or add tools. - Your volume is high and steady enough that machines cost less than a per-page bill. Use a hosted server when at least two of these are true: - You want the agent reading pages today, not after an infrastructure project. - Your volume is spiky or small. - Nobody on the team has time to own a server, its updates and its request-rate policy. - A short list of tools matters more than full control. ## If a content API fits Live Web Data API is a hosted single-page content API. Its MCP server reads one public page per call and does not render JavaScript, crawl or search. If a hosted reader fits, [read the MCP docs](/go/docs-mcp?from=guide:self-hosted-vs-hosted-scraping-mcp). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. --- # Scraping MCP token usage: formats, tool count and failures Source: https://mcpscraper.net/guides/scraping-mcp-output-and-token-cost > Where a scraping MCP server spends your context window: output format, size caps, the number of registered tools and the wording of its errors. An agent with a scraping MCP server spends tokens in four places: the tool descriptions loaded into each session, the page content each call returns, the snapshots a browser server sends after each step, and the error messages when a fetch fails. Most "the agent ran out of context" problems come from one of these. This guide shows what to check in each and which controls the common servers document, as of 2026-10-27. ## Formats The same page can cost very different amounts depending on the format: - **Full HTML** keeps scripts, navigation, footers and attributes. It is useful when the markup itself is the data, and wasteful for reading. - **Markdown of the main content** keeps headings, lists, links and code blocks and drops the rest. It is the best default for reading, summarizing and citing. - **Plain text** drops structure too. It is fine for classification or a yes/no check. - **Accessibility snapshots** describe page elements and their roles so a browser server can click them. Playwright MCP uses them deliberately: they are what make reliable interaction possible. For reading, they carry more than you need. What the servers document: Scrapling's extraction returns Markdown, HTML, text or an AI snapshot, with main-content-only as the default. Jina's `read_url` returns Markdown. Live Web Data API's `get_page_contents` returns Markdown, text or HTML, with a `verbosity` setting that keeps only the main content by default. ## Size caps A cap stops one very long page from filling the window. Look for it in the tool parameters, not in the marketing page: | Server | Documented cap | | --- | --- | | Jina MCP | `max_tokens` on the endpoint caps `read_url`; `0` disables truncation | | Live Web Data API | `max_characters` per call, 500 to 100,000, default 8,000 | | Scrapling MCP | Element targeting with CSS selectors and regex search to return less | | Playwright MCP | No character cap; `--snapshot-mode none` drops snapshots from responses, and `browser_snapshot` takes a `depth` limit | A cap is a blunt tool. Combine it with main-content extraction so the characters you keep are the useful ones. ## Tool count and context cost The MCP specification has servers list their tools with a name, a description and an input schema. What the client does with that list differs. A client that passes every definition to the model up front spends context on every registered tool in every session, whether the model calls it or not. Claude Code defers MCP tool definitions by default through tool search: only tool names and server instructions load at session start, and a definition loads when the model needs it (checked 2026-10-28). A large tool surface costs less there, so read this row together with the client you use. | Server | Tools (2026-10-27) | Can you narrow it? | | --- | --- | --- | | Firecrawl MCP | Full profile of 25+ | Separate hosted endpoints, such as a search-only URL | | Playwright MCP | About 25 core tools | Optional groups stay off unless enabled with `--caps` | | Scrapling MCP | 16 | Not documented on the MCP page | | Jina MCP | 12 | `include_tools`, `exclude_tools`, tags | | Crawl4AI | 7 | Not documented on the self-hosting page | | Live Web Data API | 2 | Not needed | If two servers are connected, both lists load, as names only in a client that defers them. A common pattern is a small read-only server enabled by default and a browser server enabled only for interaction tasks. ## Failure semantics Failures cost tokens twice: once for the error text, and again when the model retries something that cannot succeed. Two properties help: 1. **Errors that say what to do.** "The page took too long; try again later or use another source" lets the model move on. A bare status code invites retries. 2. **Clear cost on failure.** Under Firecrawl's pricing, a page that returns an error status such as 403 or 404 still costs one credit. Live Web Data API charges nothing for failed or blocked requests and ends each such error with "No credits were charged." Self-hosted servers cost your compute either way. ## Checklist - Markdown of the main content is the default format. - Every page read has a cap. - Only the tools the job needs are registered. - Errors map to one clear next step. - You know what a failed call costs. - You tested one long page and one 404 before a long run. For the full rubric, see the [rubric guide](/guides/best-mcp-server-for-scraping). For the shapes of the bill behind these calls, read the [pricing models guide](/guides/scraping-mcp-pricing-models). ## If a content API fits Live Web Data API appears in these tables. Its MCP server reads one public page per call and does not render JavaScript, crawl or search. To see its tool parameters, [read the MCP docs](/go/docs-mcp?from=guide:scraping-mcp-output-and-token-cost). --- # Scraping MCP server pricing: credits, events and your own infra Source: https://mcpscraper.net/guides/scraping-mcp-pricing-models > The pricing shapes behind scraping MCP servers: credits per page, pay per event, pay per usage and self-hosted infrastructure, and what a failed call costs. Prices for scraping MCP servers change often, so this guide does not restate them. It explains the shapes behind the prices: what unit you are billed in, what happens when a call fails, and how to estimate a month before you commit. Every pricing fact below links to the vendor's own page and was checked on 2026-10-27. For current numbers, open the vendor's pricing page. ## Pricing shapes ### Credits per request Firecrawl bills in credits. Its pricing page lists one credit per page for scrape and crawl, one per map call, and extra credits for features such as JSON output, search results and browser interaction minutes. Plans differ in monthly credits, and its page states that only some plans carry unused credits forward. If the credit shape is the reason you are looking elsewhere, the [Firecrawl MCP alternatives guide](/guides/firecrawl-mcp-alternatives) lists options by reason to switch. **Good for:** steady workloads where you can count pages ahead of time. **Watch for:** feature add-ons that multiply the per-page cost. ### Pay per event and pay per usage Apify's MCP server exposes Actors as tools, and each Actor has its own pricing model. Apify's docs describe pay per event, where the Actor's author defines billable events such as a result or a start, and pay per usage, where you pay for the platform resources a run consumes. Pay per event Actors let you set a maximum charge per run. **Good for:** jobs a specific ready-made Actor already solves. **Watch for:** pay per usage is harder to estimate; Apify's own docs suggest testing on a small scope first. ### Credits per successful page Single-page content APIs often charge per page read. Live Web Data API charges credits per successful page, with more credits when a page sat behind bot protection or was rendered in a browser; failed or blocked requests cost nothing. The current plans are on the [pricing page](/go/pricing?from=guide:scraping-mcp-pricing-models). Jina's `read_url` works without a key at a lower rate limit, and a key raises the limits. **Good for:** agents that read known public pages. **Watch for:** these APIs do not crawl, and Live Web Data API's MCP tool does not render JavaScript, so jobs that need either belong elsewhere. ### Self-hosted infrastructure Playwright MCP, Crawl4AI and Scrapling cost nothing to license under their open-source terms. You pay for machines, browser memory, bandwidth and, mostly, the hours someone spends running them. The [hosted vs self-hosted guide](/guides/self-hosted-vs-hosted-scraping-mcp) covers that operating work. ## What a failed call costs Failure rules matter more than list prices for agents, because agents retry. Three cases cover it: | Rule | Example (2026-10-27) | Effect on an agent that retries | | --- | --- | --- | | Error-status pages are charged | Firecrawl: a 403 or 404 page is returned and costs 1 credit | Every retry of a dead URL costs money | | Failed calls are free | Live Web Data API: timeouts, blocked targets and pages without content cost 0 | Retries cost latency, not credits | | You pay for compute | Self-hosted servers | Retries cost machine time | Whatever the rule, give the agent errors it can act on so it stops retrying URLs that will never work. The [output and token cost guide](/guides/scraping-mcp-output-and-token-cost) covers error wording. ## Free tiers as an entry point Free tiers are good for the rubric's test step: one known page and one 404 on example.com, read through your real client. They are a poor basis for a production estimate, because rate limits on free access are usually lower and terms change. Treat any "free scraping MCP server" list as accurate only on its check date. ## Estimating your month Use four numbers: 1. **Pages per day** the agent will actually read, not the number it could. 2. **Failure rate**, from a test run against your real URL list. 3. **Units per page**, from the vendor's pricing page, including add-ons you will use. 4. **Retries per failure** your agent or client makes. Monthly units are roughly pages × 30 × units per page, plus failures × retries × the cost of a failed call. Run the same numbers for two shapes before you pick one. A server that is cheaper per page can cost more if it charges for every failed retry. ## If a content API fits Live Web Data API appears in these tables. Its MCP server reads one public page per call and does not render JavaScript, crawl or search. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. To see how its credits work, [read the MCP docs](/go/docs-mcp?from=guide:scraping-mcp-pricing-models). --- # Responsible scraping with MCP: a security and compliance review Source: https://mcpscraper.net/guides/responsible-scraping-with-mcp > How to approve a scraping MCP server: scope, robots.txt and site terms, permissions, logging and a review checklist. General information, not legal advice. Before a team standardizes on a scraping MCP server, someone has to approve it: what it can reach, who can call it, what it logs and how it behaves toward the sites it reads. This guide gives that reviewer a scope statement, a short section on robots.txt and site terms, and a checklist. It is general information, not legal advice. ## What is in and out of scope A scraping MCP server turns a model's decision into a request to a real website. That makes scope a policy question, not just a technical one. A reasonable default scope for agents: **In scope** - Public pages that anyone can open without logging in. - Documentation, reference pages, changelogs, public articles and your own sites. - Polite request rates, with a stop on errors. **Out of scope** - Content behind logins, paywalls or subscriptions, and private networks. - Ways around bot protection or access controls. - Collecting personal data or building profiles of identifiable people. - Bulk copying of a specific named website without checking its terms and any official API first. Hosted providers write similar limits into their acceptable-use policies. Live Web Data API's, for example, prohibits accessing non-public content and placing unreasonable load on websites. A self-hosted server has no such policy unless you write one. ## robots.txt and terms robots.txt is how site operators state which paths automated clients should avoid. RFC 9309 standardizes the format, and it is explicit that the rules are not a form of access authorization. They are still the clearest statement of what a site wants from automated clients, and following them is the polite default for an agent. Site terms of use are separate. They can restrict automated access even where robots.txt allows a path, and whether they bind you depends on the site and your jurisdiction. Your review should record: - whether the server checks robots.txt, and whether that can be turned off; - whether the provider honors opt-out and takedown requests; - who on your side reads the terms of sites the agent will use most. Record the provider's actual words, not your reading of them. Live Web Data API's terms, for example, put compliance with site terms on the customer, say Teneo Labs blocks a website for all customers once it has verified the operator's request, and say it may honour robots.txt directives but is not obliged to ([Terms](/terms), Section 8.6). A reviewer should treat that as "robots.txt handling is the customer's policy", not as a promise. This is general information, not legal advice. ## Fewer blocks on public pages, the honest way Agents that hammer a site get rate limited, and rate limits are a fair response. The way to see fewer blocks on public pages is the boring one: read fewer pages, cache what you read with a maximum age, space out requests to the same host, stop on 403 and 429 responses instead of retrying, and identify your use clearly where the server allows it. A server that advertises ways around bot protection is a compliance risk to flag in review, not a feature to rely on. ## Permissions and the MCP side The MCP specification's security best practices cover risks such as token passthrough and session handling. For a scraping server, three questions matter most: 1. **Who can call it?** A stdio server is reachable only by the client that started it. An HTTP server is reachable over the network; Scrapling's HTTP mode, for example, requires an auth token unless you disable it. Keep that default. 2. **What can the tools do?** The specification lets servers describe tools with annotations such as read-only hints. A read-only page reader is a smaller risk than a browser tool that can type into forms or run code. 3. **What does the content do to the agent?** Pages are untrusted input. A page can contain text that tries to instruct the model. Keep scraping tools separate from tools with side effects, and require confirmation before actions. ## Approving a server: checklist - The tools and their scope are listed and match the job. - Auth uses per-environment keys, tokens or OAuth, and keys rotate. - HTTP transports are bound to localhost or require a token. - There is an acceptable-use policy and a stated robots.txt position. - Logs of URLs and content follow your retention policy. - Rate limits per site are set in config, outside the model's control. - Logins, paywalls and personal data are out of scope by written policy. - Someone re-reviews the server when it ships a major release. The rubric's compliance row records these facts per server; see the [rubric guide](/guides/best-mcp-server-for-scraping). For where data goes when the server is hosted, read the [hosted vs self-hosted guide](/guides/self-hosted-vs-hosted-scraping-mcp). ## If a content API fits Live Web Data API is a hosted reader for public pages with a published acceptable-use policy. Sign-up is for businesses only; only Business, Enterprise and Custom sign-ups are reviewed before checkout. Its MCP server does not render JavaScript, and the product does not crawl or search. To see its tools and error behavior, [read the MCP docs](/go/docs-mcp?from=guide:responsible-scraping-with-mcp). --- # Apify MCP server vs Firecrawl MCP: Actors or one crawl API Source: https://mcpscraper.net/blog/apify-mcp-server-vs-firecrawl-mcp > Apify's MCP server turns Actors into tools; Firecrawl MCP exposes one crawl platform. Transport, tool surface, pricing shape and failure cost, checked 2026-10-27. Apify and Firecrawl are both crawl platforms with hosted MCP endpoints, so they often end up on the same shortlist. The difference is the model behind the tools. Apify's MCP server turns Actors, ready-made scrapers and automations from its store, into tools. Firecrawl's MCP server exposes one general crawl API. Rows below are from each project's docs, checked on 2026-10-27. ## Transport and auth Apify's server runs hosted over Streamable HTTP at mcp.apify.com, or locally over stdio via npx, with OAuth or a bearer token. Some Actors are only available through the hosted server. Firecrawl's server runs over stdio, as a local Streamable HTTP server or at a hosted URL, with an API key and an OAuth endpoint for hosted use. ## Tool surface This is where they differ in practice. Apify starts with a default set of tools and grows with every Actor you add; its `tools` parameter narrows the list to what the job needs. Firecrawl's full profile lists more than 25 tools for scraping, mapping, crawling, search, interaction, agents and monitors. On both, fewer registered tools means less context spent before the first page is read. ## Pricing shape and failed calls Apify Actors each carry their own pricing model. Apify's docs describe pay per event, where the Actor's author defines billable events, and pay per usage, where you pay for the platform resources a run uses; pay-per-event Actors let you set a maximum charge per run. Firecrawl bills credits per request, with extra credits for some features, and charges one credit for a page that returns an error status such as 403 or 404. The [pricing models guide](/guides/scraping-mcp-pricing-models) walks through estimating a month under each. ## Which one fits Choose Apify when a specific Actor already solves your job and you want to pay for its events. Choose Firecrawl when you want one general API for scrape, map and crawl. If you are moving away from Firecrawl for another reason, the [Firecrawl MCP alternatives guide](/guides/firecrawl-mcp-alternatives) sorts the options by reason, and the [rubric guide](/guides/best-mcp-server-for-scraping) has all ten rows for both servers. --- # Firecrawl MCP vs Crawl4AI MCP: hosted platform or your own crawler Source: https://mcpscraper.net/blog/firecrawl-mcp-vs-crawl4ai-mcp > Firecrawl MCP and Crawl4AI's MCP endpoints side by side on the rubric rows: transport, tools, license, hosting, failure cost and upkeep. Checked 2026-10-27. Both projects crawl websites and return Markdown, and both can run on your own machines. That is where the overlap ends. Firecrawl MCP is a thin MCP layer over a crawl platform that is usually hosted. Crawl4AI is an open-source crawler whose Docker server also speaks MCP. This post puts their MCP interfaces on the same rubric rows, checked against each project's docs on 2026-10-27. ## Transport and tool surface Firecrawl's MCP server runs over stdio, as a local Streamable HTTP server, or at a hosted URL with an API key. Its README lists a full profile of more than 25 tools: scrape, map, crawl, search, interaction, agents and monitors. In a client that loads every tool definition upfront, all of those descriptions reach the model in each session unless you use a narrower endpoint. Crawl4AI's Docker server exposes MCP over SSE at `/mcp/sse` and WebSocket at `/mcp/ws`. It registers seven tools: `md`, `html`, `screenshot`, `pdf`, `execute_js`, `crawl` and `ask`. That is a much smaller footprint in context, but clients that only speak stdio or Streamable HTTP need a bridge. ## License and hosting These are two separate questions. Firecrawl's MCP server is MIT, while Firecrawl's core is AGPL-3.0, and the README says the cloud version has features the self-hosted one lacks. Crawl4AI is Apache-2.0, with an attribution request in its README. If you plan to offer a modified crawler over a network, have someone read the AGPL obligations before choosing. This is not legal advice. ## What a failed call costs Firecrawl's pricing page says requests that return no result are not charged, but a page that responds with an error status such as 403 or 404 is returned and costs one credit. Crawl4AI has no bill: every call, failed or not, costs your own compute and the time someone spends keeping the server healthy. The [pricing models guide](/guides/scraping-mcp-pricing-models) shows how to estimate a month under each shape. ## Which one fits Choose Firecrawl MCP when you want map, crawl and search behind one hosted URL and do not want to run browsers. Choose Crawl4AI when crawling must stay on your infrastructure, when a permissive license matters, or when seven tools suit your context budget better. If neither crawling nor interaction is the job, a single-page reader may be enough; the [Firecrawl MCP alternatives guide](/guides/firecrawl-mcp-alternatives) sorts every option by reason to switch, and the [rubric guide](/guides/best-mcp-server-for-scraping) has all ten rows for both. --- # Playwright MCP vs Firecrawl MCP: browser control or crawl platform Source: https://mcpscraper.net/blog/playwright-mcp-vs-firecrawl-mcp > Playwright MCP drives a browser step by step; Firecrawl MCP scrapes, maps and crawls. The rubric rows that separate them, checked on 2026-10-27. People compare these two because both show up in "best MCP server for scraping" lists. They solve different jobs. Playwright MCP gives the model a real browser and lets it act one step at a time. Firecrawl MCP gives the model collection tools: scrape a page, map a site, crawl it, search. The rows below come from each project's docs, checked on 2026-10-27. ## The unit of work Playwright MCP works in browser sessions. The model navigates, reads an accessibility snapshot, clicks, types and reads the next snapshot. That is exactly right for form flows, UI tests and bug reproduction on your own app. Firecrawl MCP works in pages and sites. One call returns a page, a list of URLs or a crawl job, without the model steering a browser. ## Tool surface and context cost Both are large. Playwright MCP registers about 25 core tools, with more optional groups behind `--caps`. Firecrawl's README lists a full profile of more than 25 tools. The output differs more than the tool count: Playwright returns accessibility snapshots after actions, which carry element roles and references that reading does not need, while Firecrawl returns page content in the formats you ask for, with an option for main content only. The [output and token cost guide](/guides/scraping-mcp-output-and-token-cost) covers the controls each one documents. ## Hosting and cost Playwright MCP runs on your machine or in its Docker image under Apache-2.0. It is free to run; you pay for the machine and its browser memory. Firecrawl MCP is usually used against the hosted platform, billed in credits per request, and its pricing page states that a page returning an error status such as 403 or 404 still costs one credit. You can also point it at a self-hosted Firecrawl. ## Which one fits Pick Playwright MCP when the job is interaction. Pick Firecrawl MCP when the job is collecting many pages. If the job is reading a handful of known public pages, both carry more tools than you need; the [pick-by-job guide](/guides/choose-a-scraping-mcp-server-by-job) shows the lighter options, and the [rubric guide](/guides/best-mcp-server-for-scraping) has all ten rows for both servers. --- # Live Web Data API — full documentation # Overview Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network. ## What it does You send up to 10 URLs (and optionally a country). We fetch each page through a residential or mobile connection, extract the main content and return it in the format your agent or pipeline needs. Successful pages cost 1 credit; failures cost nothing. | | Value | | --- | --- | | Base URL | `https://livewebdata.com/api` | | Endpoints | `POST /v1/contents`, `GET /v1/account` | | Formats | markdown and text by default; html with `html: true` | | OpenAPI | [`/openapi.json`](https://mcpscraper.net/openapi.json) · [`/openapi.yaml`](https://mcpscraper.net/openapi.yaml) | | Fetch strategy (v1) | static fetch (`fast`), a retry through another connection (`retry`), or a real browser (`browser`) with `render` or `renderFallback` | ## Authentication Create a key in the dashboard under [API keys](https://mcpscraper.net/dashboard/keys) and send it as a bearer token. Keys start with `1337_live_` (test keys with `1337_test_`) and are shown only once. Keep them in an environment variable such as `LIVE_WEB_DATA_API_KEY` — never in client-side code or repositories. ```bash title="Headers" Authorization: Bearer 1337_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Content-Type: application/json ``` ## Quickstart Fetch a page as Markdown: ```bash title="cURL" curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com/article"]}' ``` The response contains one entry in `results` per successful page and one entry in `statuses` per requested URL. The page HTML is included only when you send `"html": true`: ```json title="Response (abridged)" { "requestId": "req_01J…", "results": [{ "id": "https://example.com/article", "url": "https://example.com/article", "title": "Article", "markdown": "# Article\n\n…", "text": "Article\n\n…" }], "statuses": [{ "id": "https://example.com/article", "status": "success", "source": "livecrawled", "fetchedAt": "2026-10-23T09:15:02.118Z", "httpStatusCode": 200, "strategy": "fast", "credits": 1, "error": null }], "searchTime": 1240 } ``` When `text` or `markdown` was cut at `text.maxCharacters`, `results[].textTruncated` is `true`. `results[].warnings` (absent when empty) flags pages that could only be partly retrieved (`PAGE_TRUNCATED`), probably need `render: true` (`THIN_CONTENT`), were converted as plain text (`EXTRACTION_FALLBACK`) or lost their `html` (and maybe part of their text) to the response's 24 MiB size limit (`RESPONSE_SIZE_LIMIT`). See the [API reference](https://mcpscraper.net/docs/api-reference) for every parameter, or try it without code in the dashboard [playground](https://mcpscraper.net/dashboard/playground). ## SDKs Thin, typed clients for JavaScript/TypeScript and Python with built-in retries and typed errors. See [SDKs](https://mcpscraper.net/docs/sdks). ```javascript title="JS SDK" import { Teneo } from "@teneo/contents"; const client = new Teneo({ apiKey: process.env.LIVE_WEB_DATA_API_KEY, baseUrl: "https://livewebdata.com/api", }); const res = await client.contents({ urls: ["https://example.com/article"] }); for (const s of res.statuses) console.log(s.id, s.status, s.error?.tag ?? ""); console.log(res.results[0]?.markdown); ``` ```python title="Python SDK" import os from teneo import Teneo client = Teneo( api_key=os.environ["LIVE_WEB_DATA_API_KEY"], base_url="https://livewebdata.com/api", ) res = client.contents( urls=["https://example.com/article"], ) for s in res["statuses"]: print(s["id"], s["status"], (s["error"] or {}).get("tag", "")) print(res["results"][0]["markdown"] if res["results"] else "") ``` ## Live fetching Every request is fetched live; there is no cache. ## Geotargeting Set `countryCode` to an ISO-3166-1 alpha-2 code (for example `de`, `jp`, `br`) to fetch the page as seen from that country. The default is `us`. ```bash title="cURL" curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/pricing" ], "countryCode": "de" }' ``` ```python title="Python" import requests response = requests.post( "https://livewebdata.com/api/v1/contents", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={ "urls": ["https://example.com/pricing"], "countryCode": "de" }, timeout=60, ) data = response.json() print(data["results"][0]["markdown"]) ``` ```javascript title="JavaScript" const response = await fetch("https://livewebdata.com/api/v1/contents", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ "urls": [ "https://example.com/pricing" ], "countryCode": "de" }), }); const data = await response.json(); console.log(data.results[0]?.markdown); ``` ## Multiple URLs Send up to 10 URLs in one request. They are processed in parallel; `statuses` has one entry per URL (matched by `id`) and credits are charged per successful URL. A restricted URL gets a per-URL `RESTRICTED_TARGET` status instead of failing the whole request. More than 10 URLs return `400 TOO_MANY_URLS`. Rate limits count URLs, not HTTP requests. ```bash title="cURL" curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/a", "https://example.com/b", "https://example.com/blocked" ] }' ``` ```python title="Python" import requests response = requests.post( "https://livewebdata.com/api/v1/contents", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={ "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"] }, timeout=60, ) data = response.json() print(data["results"][0]["markdown"]) ``` ```javascript title="JavaScript" const response = await fetch("https://livewebdata.com/api/v1/contents", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ "urls": [ "https://example.com/a", "https://example.com/b", "https://example.com/blocked" ] }), }); const data = await response.json(); console.log(data.results[0]?.markdown); ``` ## Restricted targets Some domain categories are blocked for legal and regulatory reasons. Requests are rejected with `451` and cost no credits. Repeated attempts lead to automatic key suspension. Report abuse to abuse@livewebdata.com. ## MCP for agents Agents can use the API as a tool over MCP with your existing key — two tools, `get_page_contents` and `get_account_status`. Setup for Claude Code, Cursor, VS Code and Claude Desktop: [MCP for agents](https://mcpscraper.net/docs/mcp). ## For AI coding agents Every docs page is also available as plain Markdown by appending `.md` to its URL (e.g. [`/docs/api-reference.md`](https://mcpscraper.net/docs/api-reference.md)). An index for LLMs lives at [`/llms.txt`](https://mcpscraper.net/llms.txt) and the full docs in one file at [`/llms-full.txt`](https://mcpscraper.net/llms-full.txt). --- # API reference Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome. ## Endpoint ```bash title="Request" POST https://livewebdata.com/api/v1/contents Authorization: Bearer Content-Type: application/json ``` Machine-readable specs: - OpenAPI 3.1 (JSON): https://mcpscraper.net/openapi.json - OpenAPI 3.1 (YAML): https://mcpscraper.net/openapi.yaml - Postman collection: https://mcpscraper.net/live-web-data-api.postman_collection.json - Interactive reference: https://mcpscraper.net/docs/api-reference/interactive ## Request parameters | Parameter | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | `urls` | `string[]` | yes | — | The pages to fetch, 1–10 http or https URLs of at most 8,192 characters each. Processed in parallel; one entry per URL in results/statuses. Duplicates (same URL, ignoring #fragment) are fetched once; repeated positions cost 0 credits. | | `countryCode` | `string` | no | "us" | ISO-3166-1 alpha-2 code. The page is fetched through a connection in this country. | | `text` | `boolean \| object` | no | true | Content options. Pass an object to set maxCharacters and verbosity. | | `text.maxCharacters` | `integer` | no | 100000 | Truncates text and markdown to this many characters (max 1,000,000); `results[].textTruncated` tells you when it did. The MCP tool defaults to 8,000 (`max_characters`). | | `text.verbosity` | `"compact" \| "full"` | no | "compact" | compact = main content only. full = whole body incl. navigation and footer. | | `html` | `boolean` | no | false | Include the page HTML as served in results[].html. A response carries at most 24 MiB of JSON: past that, the remaining results come without html (warning RESPONSE_SIZE_LIMIT). | | `livecrawlTimeout` | `integer (ms)` | no | 30000 (60000 with render) | Maximum time for a live fetch, 1–90000 ms. A lower value is raised to 30000, the time the browser fallback needs, unless you send renderFallback: false (then any value, default 10000). With render or renderFallback: true at least 30000. | | `render` | `boolean` | no | false | Render every URL in a real browser, as a normal visitor sees it: 3 credits, 5 when the page needed a retry through another connection. Cannot be combined with renderFallback. | | `renderFallback` | `boolean` | no | on | Fetch normally first and render in a real browser only when the first attempt did not return the page, shows no or very little visible text, or was blocked. On by default: omit it. Send false for a plain fetch only (never a browser). A normal success costs its normal 1 or 2 credits, a rendered one only the render credits; statuses[].fallbackUsed tells you when it rendered. Cannot be combined with render: true. | ```json title="Example body" { "urls": ["https://example.com/article"], "countryCode": "de", "text": { "maxCharacters": 50000, "verbosity": "compact" }, "livecrawlTimeout": 45000 } ``` Unknown fields are rejected with `400 INVALID_REQUEST`; the message names the field. So is a URL longer than 8,192 characters (`urls.0: Each URL may be at most 8,192 characters.`). ## Response | Field | Type | Description | | --- | --- | --- | | `requestId` | `string` | Id of this request, the same for every repeat with the same Idempotency-Key. Also returned as X-Request-Id. | | `results[]` | `object[]` | One entry per successfully fetched URL. | | `results[].url` | `string` | Final URL after redirects. | | `results[].title` | `string \| null` | Page title. | | `results[].author / publishedDate / image` | `string \| null` | Metadata when the page exposes it. | | `results[].favicon` | `string \| null` | Favicon URL. | | `results[].markdown` | `string` | Main content as Markdown. | | `results[].text` | `string` | Plain text, tags removed, whitespace normalized. | | `results[].html` | `string` | Page HTML as served. Only with html: true, and only while the response stays within its 24 MiB size limit (else warning RESPONSE_SIZE_LIMIT). | | `results[].textTruncated` | `boolean` | true when text or markdown was cut at text.maxCharacters, or at the response size limit (warning RESPONSE_SIZE_LIMIT). Present whenever text is enabled. | | `results[].warnings` | `{ code, message }[]` | Absent when empty. "PAGE_TRUNCATED" = only part of the page could be retrieved; content after that point is missing. "THIN_CONTENT" = very little text, the page probably needs render: true. "EXTRACTION_FALLBACK" = the page was too complex to convert in time; text is plain and markdown has no structure. "RESPONSE_SIZE_LIMIT" = the response reached its 24 MiB size limit: this result has no html, or no html and cut text/markdown. | | `statuses[]` | `object[]` | One entry per requested URL, success or error. | | `statuses[].status` | `"success" \| "error"` | Outcome for this URL. | | `statuses[].source` | `"livecrawled" \| null` | Always "livecrawled" on success: every request is fetched live. null on error. | | `statuses[].fetchedAt` | `string \| null` | When the page was fetched (ISO 8601). null on error. | | `statuses[].strategy` | `"fast" \| "retry" \| "browser"` | Fetch strategy. "fast" = standard page (1 credit); "retry" = the first attempt did not return the page, a retry through another connection did (2 credits); "browser" = rendered in a real browser, as a normal visitor sees it (3 credits, 5 when it needed a retry through another connection). | | `statuses[].credits` | `integer` | Credits charged for this URL: 1 standard, 2 retried, 3 rendered, 5 rendered + retried, 0 on error and for a short page (thin). | | `statuses[].fallbackUsed` | `boolean` | Present (true) only when the browser fallback ran for this URL. With strategy "browser" the content is that render (fast → render); otherwise rendering was tried and did not help. | | `statuses[].thin` | `boolean` | Present (true) only for a short page: fewer than 500 characters of visible text even after rendering. Delivered as a success at 0 credits, with only that visible text. | | `statuses[].error` | `{ tag, message } \| null` | Per-URL error, see crawl errors. | | `searchTime` | `integer (ms)` | Total processing time. | Browser fallback is on by default: a page the fast fetch cannot read (no or very little visible text, blocked, 401/403, a timeout) is rendered in a real browser and charged only the render credits. `statuses[].fallbackUsed` is `true` when that happened (`strategy: "browser"` = the content is the render). Send `renderFallback: false` for a fast fetch only. A page that still shows fewer than 500 characters of visible text after rendering is delivered as a success with `statuses[].thin: true` at 0 credits, holding only that visible text — it is not an error. `text.maxCharacters` (default 100,000, max 1,000,000) cuts `text` and `markdown`; `results[].textTruncated` is `true` when it did. A response carries at most 24 MiB of JSON: from the result that would pass it on, results come without `html`, and if that is not enough, `text` and `markdown` are cut (`textTruncated: true`), each such result flagged `RESPONSE_SIZE_LIMIT`. The request never fails because of it. `title` and `author` are at most 2,000 characters each; an `image` or `favicon` URL longer than 4,096 characters is left out (`favicon` then falls back to `/favicon.ico`). The MCP tool `get_page_contents` has its own, smaller default of 8,000 (`max_characters`), because its result goes into an agent's context. `results[].warnings` is present only when there is something to know about a page: | Code | Meaning | What to do | | --- | --- | --- | | `PAGE_TRUNCATED` | Only part of the page could be retrieved (size limit or an interrupted transfer); content after that point is missing. | Nothing to retry; the result is the part that arrived. | | `THIN_CONTENT` | Very little text next to scripts or an empty app root: the page probably loads its content with JavaScript. The URL is still charged as a success. | Retry with `render: true`. The default browser fallback renders a page with fewer than 500 characters of visible text by itself, not one with more. | | `EXTRACTION_FALLBACK` | The page structure was too complex to convert in time; `text` is plain and `markdown` carries no structure. | Use `text`. | | `RESPONSE_SIZE_LIMIT` | The response reached its 24 MiB size limit. This result and every later one come without `html`; when that was not enough, `text` and `markdown` were cut too (`textTruncated: true`). The URL is still a success and is charged. | Request fewer URLs at once, or without `html`. | ## Response headers | Header | Description | | --- | --- | | `X-Request-Id` | Same as requestId in the body. | | `X-Credits-Remaining` | Credit balance after this request. | | `X-RateLimit-Limit` | URLs you can send at once (bucket size, = rateLimit.burst in GET /v1/account). The bucket refills at rateLimit.requestsPerSecond URLs per second. | | `X-RateLimit-Remaining` | URLs you can send right now. | | `Retry-After` | Seconds to wait: before retrying a 429, or before repeating a 202 with the same Idempotency-Key. | | `Idempotency-Key` | The request's Idempotency-Key: yours, or one the API created when you sent none. On every answer (200, 202, errors) except a 400 for a malformed key and a 413 for a body over 10 MiB (refused unread). Repeat a 202 with it. | ## Errors ### Per-URL errors Returned inside `statuses[].error` with HTTP 200. These never cost credits. | Tag | Meaning | | --- | --- | | `CRAWL_NOT_FOUND` | The page returned 404/410. | | `SOURCE_NOT_AVAILABLE` | The target refused access (e.g. 403 or 429) after retries. | | `CRAWL_LIVECRAWL_TIMEOUT` | The live fetch did not finish within livecrawlTimeout. | | `CRAWL_NOT_RENDERED` | No usable content, e.g. the page needs JavaScript rendering (retry without renderFallback: false, or with render: true), or a rendered page shows no visible text. 0 credits. | | `UNSUPPORTED_CONTENT_TYPE` | The URL does not serve HTML or text. | | `CRAWL_UNKNOWN_ERROR` | Something else went wrong. Safe to retry. | | `SOURCE_BLOCKED` | Access denied by the site, or it answered with an interstitial page (HTTP 202) instead of the content. 0 credits. | | `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable; no credits were charged. | | `RATE_LIMITED` | Rendering limit for your plan reached; no credits were charged. | | `RESTRICTED_TARGET` | This URL is restricted. Other URLs in the same request are still processed. 0 credits. | ### Request errors Returned with a non-200 status and no results. | HTTP | Tag | Meaning | | --- | --- | --- | | 400 | `INVALID_REQUEST` | Invalid parameters. | | 400 | `TOO_MANY_URLS` | More URLs than allowed per request (10). | | 401 | `UNAUTHORIZED` | Missing or invalid API key. | | 402 | `INSUFFICIENT_CREDITS` | Not enough credits. Top up or upgrade. | | 403 | `KEY_SUSPENDED` | The key or account is suspended or not yet approved. | | 429 | `RATE_LIMITED` | Rate limit exceeded. Retry after the Retry-After header. | | 409 | `IDEMPOTENCY_CONFLICT` | The Idempotency-Key was already used with a different request, or its result is no longer available (results are kept 15 minutes). Use a new key. | | 413 | `PAYLOAD_TOO_LARGE` | The request body is larger than 10 MiB. Nothing was read or charged. | | 451 | `RESTRICTED_TARGET` | All requested URLs are restricted. No credits charged. | | 503 | `SERVICE_UNAVAILABLE` | Temporarily unavailable. Retry shortly. | ```json title="Error body" { "error": { "code": 402, "tag": "INSUFFICIENT_CREDITS", "message": "Not enough credits. Top up or upgrade your plan." }, "requestId": "req_01J…" } ``` ## Long requests and Idempotency-Key Most requests answer within seconds. A request still fetching after about 100 seconds (slow sites, rendering, a long `livecrawlTimeout`) answers `202` instead of making you wait on an open connection: ```json title="202 Accepted" { "status": "pending", "requestId": "req_01J…", "idempotencyKey": "lwda_3f1c2b7e-0d4a-4c1e-9a55-2b8f0e6c7d10", "retryAfter": 1, "message": "Still fetching. Send the same request again with the same Idempotency-Key header to get the result; it is not charged again." } ``` Send the **same request body** again with the header `Idempotency-Key: ` after `Retry-After` seconds. The repeat waits for the fetch and answers `200` with the result, or `202` again if it is still running. A repeat is never charged again. The SDKs and the MCP server do this for you. **Always send your own `Idempotency-Key`** (8–128 characters of `A–Z a–z 0–9 _ - : .`, for example a UUID) on the first request, and keep it until you have the answer. A connection can break after the API received the request: a network error, a timeout on your side, a deploy. Send the same request again with the same key, and you get that fetch (its result, or `202` while it still runs) instead of a new one, so you pay once. Without your own key, a cut connection is a lost answer: the retry has no key, so it is a new request and charged again. Every answer of `POST /v1/contents` (`200`, `202` and errors) carries the request's key in the `Idempotency-Key` response header: yours, or one the API created when you sent none. The only exceptions are the `400` for a malformed key and the `413` for a request body over 10 MiB, which is refused before it is read (nothing was charged, there is nothing to repeat). The SDKs create a key for each `contents()` call when you pass none, send it on every retry and put it on the error they raise (`idempotencyKey` / `idempotency_key`). - Results are kept for **15 minutes**. A result nobody collects stays charged; after 15 minutes a repeat answers `409 IDEMPOTENCY_CONFLICT`, and a new key fetches the pages again. - A key is bound to its request body for 24 hours: reusing it with a different body answers `409 IDEMPOTENCY_CONFLICT`. Use a new key for every new request. ## Rate limits Limits are per account and count **URLs per second** (a request with 3 URLs uses 3). When exceeded, the API returns `429` with a `Retry-After` header (seconds). Limits work as a bucket: `X-RateLimit-Limit` is how many URLs you can send at once (`rateLimit.burst` in `GET /v1/account`), `X-RateLimit-Remaining` how many right now, and the bucket refills at your plan's URLs per second (`rateLimit.requestsPerSecond`). | Plan | Price / month | Rate limit (URLs/s) | | --- | --- | --- | | Lite | $10 | 1 | | Starter | $100 | 2 | | Developer | $500 | 5 | | Business | $900 | 10 | | Enterprise | $2,250 | 20 | | Custom | Custom | Custom | ## Credits | Request | Credits | | --- | --- | | Standard page (static / server-rendered) | 1 | | Retried page (succeeded on a retry through another connection) | 2 | | JS-rendered page | 3 | | Rendered + retried page | 5 | | Short page (under 500 characters of visible text; delivered as that text, a success) | 0 | | Failed request (incl. access denied by the site) | 0 | - Plan credits reset monthly. - Top-up credits never expire. When your balance is too low for the URLs in a request, the API returns `402 INSUFFICIENT_CREDITS`. The current balance is returned in `X-Credits-Remaining` on every response. ## Account `GET https://livewebdata.com/api/v1/account` returns your plan, remaining credits and rate limit. It is free — call it before large jobs. ```json title="GET /v1/account" { "plan": "starter", "status": "active", "credits": { "plan": 81234, "topup": 10000, "total": 91234 }, "rateLimit": { "requestsPerSecond": 2, "burst": 4 }, "periodEndsAt": "2026-11-01T00:00:00Z" } ``` --- # Examples Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard. ## Basic fetch The smallest possible request — one URL, defaults for everything else. ```bash title="cURL" curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/article" ] }' ``` ```python title="Python" import requests response = requests.post( "https://livewebdata.com/api/v1/contents", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={ "urls": ["https://example.com/article"] }, timeout=60, ) data = response.json() print(data["results"][0]["markdown"]) ``` ```javascript title="JavaScript" const response = await fetch("https://livewebdata.com/api/v1/contents", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ "urls": [ "https://example.com/article" ] }), }); const data = await response.json(); console.log(data.results[0]?.markdown); ``` ## Geotargeting Fetch a page as seen from Japan. ```bash title="cURL" curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/pricing" ], "countryCode": "jp" }' ``` ```python title="Python" import requests response = requests.post( "https://livewebdata.com/api/v1/contents", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={ "urls": ["https://example.com/pricing"], "countryCode": "jp" }, timeout=60, ) data = response.json() print(data["results"][0]["markdown"]) ``` ```javascript title="JavaScript" const response = await fetch("https://livewebdata.com/api/v1/contents", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ "urls": [ "https://example.com/pricing" ], "countryCode": "jp" }), }); const data = await response.json(); console.log(data.results[0]?.markdown); ``` ## Text limits Return at most 5,000 characters of the full page body, including navigation and footer. ```bash title="cURL" curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/docs/getting-started" ], "text": { "maxCharacters": 5000, "verbosity": "full" } }' ``` ```python title="Python" import requests response = requests.post( "https://livewebdata.com/api/v1/contents", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={ "urls": ["https://example.com/docs/getting-started"], "text": { "maxCharacters": 5000, "verbosity": "full" } }, timeout=60, ) data = response.json() print(data["results"][0]["markdown"]) ``` ```javascript title="JavaScript" const response = await fetch("https://livewebdata.com/api/v1/contents", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ "urls": [ "https://example.com/docs/getting-started" ], "text": { "maxCharacters": 5000, "verbosity": "full" } }), }); const data = await response.json(); console.log(data.results[0]?.markdown); ``` ## Several URLs Fetch up to 10 pages in one request. Each URL gets its own status; only successful pages cost credits. ```bash title="cURL" curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/a", "https://example.com/b", "https://example.com/blocked" ] }' ``` ```python title="Python" import requests response = requests.post( "https://livewebdata.com/api/v1/contents", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={ "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"] }, timeout=60, ) data = response.json() print(data["results"][0]["markdown"]) ``` ```javascript title="JavaScript" const response = await fetch("https://livewebdata.com/api/v1/contents", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ "urls": [ "https://example.com/a", "https://example.com/b", "https://example.com/blocked" ] }), }); const data = await response.json(); console.log(data.results[0]?.markdown); ``` ## Whole jobs For URL lists, retries and scheduling see [Automate requests](https://mcpscraper.net/docs/automation) — including ready-made Python and Node scripts. --- # Automate requests Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled. ## From playground to code Every run in the [playground](https://mcpscraper.net/dashboard/playground) shows a **Use it in code** panel with cURL, Python, JavaScript and SDK snippets for exactly the request you just made. Start there, then move to the scripts below. Download all example scripts: https://mcpscraper.net/live-web-data-api-examples.zip | File | What it does | | --- | --- | | `examples/python/quickstart.py` | One request, prints title + first 500 chars of markdown | | `examples/python/batch_from_csv.py` | Reads `urls.csv` (`url`, optional `country`), fetches with concurrency = rate limit, retries with backoff, writes `results.jsonl` and prints a summary | | `examples/node/quickstart.mjs` | Same as the Python quickstart, native `fetch`, no dependencies | | `examples/node/batch_from_csv.mjs` | Same as the Python batch script, no dependencies | | `examples/urls.csv` | 10 sample URLs incl. a restricted, a missing and a timing-out URL to show error handling | ```bash title="Run the batch script" export LIVE_WEB_DATA_API_KEY=1337_live_… # never hard-code keys python examples/python/batch_from_csv.py examples/urls.csv node examples/node/batch_from_csv.mjs examples/urls.csv ``` ## Process a list of URLs The simplest loop: one request per URL, one after the other. ```python title="Python" import os, requests API = os.environ.get("LIVE_WEB_DATA_API_BASE_URL", "https://livewebdata.com/api") headers = {"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"} for url in ["https://example.com/a", "https://example.com/b"]: r = requests.post(f"{API}/v1/contents", json={"urls": [url]}, headers=headers, timeout=60) status = r.json()["statuses"][0] print(url, status["status"], status["error"] and status["error"]["tag"]) ``` ```javascript title="Node" const API = process.env.LIVE_WEB_DATA_API_BASE_URL ?? "https://livewebdata.com/api"; const headers = { Authorization: `Bearer ${process.env.LIVE_WEB_DATA_API_KEY}`, "Content-Type": "application/json" }; for (const url of ["https://example.com/a", "https://example.com/b"]) { const res = await fetch(`${API}/v1/contents`, { method: "POST", headers, body: JSON.stringify({ urls: [url] }) }); const { statuses } = await res.json(); console.log(url, statuses[0].status, statuses[0].error?.tag ?? ""); } ``` You can also send up to 10 URLs per request — see [Several URLs](https://mcpscraper.net/docs/examples#several-urls). ## Respect rate limits - Keep a steady pace of **your plan's URLs per second** (from `GET /v1/account` → `rateLimit.requestsPerSecond`). Bursts up to `rateLimit.burst` URLs (the bucket size, also sent as `X-RateLimit-Limit`) are fine; the bucket refills at that rate. - Watch `X-RateLimit-Remaining` on each response: it is how many URLs you can send right now. - On `429`, wait `Retry-After` seconds and retry the same request. ## Retries Retry `429`, `5xx` and per-URL `CRAWL_LIVECRAWL_TIMEOUT` up to **3 times** with exponential backoff and jitter (e.g. `0.5s · 2^n + random(0–0.25s)`). Do **not** retry: | Outcome | Why | | --- | --- | | `451 RESTRICTED_TARGET` | The domain is blocked; repeated attempts can suspend your key | | `400 INVALID_REQUEST` / `TOO_MANY_URLS` | Fix the request first | | `401` / `403` | Key invalid, revoked or suspended | | `402 INSUFFICIENT_CREDITS` | Stop the job cleanly, top up, resume | | `CRAWL_NOT_FOUND` | The page doesn't exist | **Send your own `Idempotency-Key`** with every `POST /v1/contents` (one per logical request, for example per URL of your list) and send the same key on every retry of that request. A connection cut after the API received the request (a network error, a timeout, a deploy) is then retried without a second charge: the repeat returns that fetch's result. A `202` (still fetching) is repeated the same way, after `Retry-After`. Use a **new** key for a new attempt at a URL that already has an answer, such as a per-URL `CRAWL_LIVECRAWL_TIMEOUT`: the same key returns the same answer. See [Long requests and Idempotency-Key](https://mcpscraper.net/docs/api-reference#long-requests-and-idempotency-key). The [SDKs](https://mcpscraper.net/docs/sdks) implement this policy for you. ## Check your balance first Call `GET /v1/account` before a large job and compare `credits.total` with the number of URLs. Stop cleanly when a request returns `402`. ```bash title="cURL" curl https://livewebdata.com/api/v1/account -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" ``` ## Scheduling Any scheduler works — the scripts are plain processes that exit when done. ```bash title="cron (daily at 06:00)" 0 6 * * * cd /opt/jobs && LIVE_WEB_DATA_API_KEY=… /usr/bin/python3 batch_from_csv.py urls.csv --out results.jsonl ``` ```yaml title=".github/workflows/fetch.yml" name: Fetch pages on: schedule: [{ cron: "0 6 * * *" }] workflow_dispatch: jobs: fetch: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: node examples/node/batch_from_csv.mjs examples/urls.csv --out results.jsonl env: LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }} ``` Other schedulers (Cloud Scheduler, Kubernetes CronJob, n8n, Airflow) run the same command. ## Keep your key safe - Store it in an environment variable (`LIVE_WEB_DATA_API_KEY`) or your platform's secret manager. - Never put it in client-side code, mobile apps, browser extensions or repositories. - Use one key per environment (e.g. production, staging, n8n) — you can label keys in the dashboard and see the label in the usage log. - Rotate a key immediately if it leaks: create a new one, then revoke the old one. --- # SDKs Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches. ## Install The SDKs live in the repository under `packages/` and are not published yet. ```bash title="JavaScript / TypeScript" cd packages/sdk-js && npm install && npm run build npm install ./packages/sdk-js # in your project ``` ```bash title="Python" pip install ./packages/sdk-python # requires httpx ``` ## JavaScript / TypeScript ```javascript title="JS SDK" import { Teneo } from "@teneo/contents"; const client = new Teneo({ apiKey: process.env.LIVE_WEB_DATA_API_KEY, baseUrl: "https://livewebdata.com/api" }); const page = await client.getPage("https://example.com/article", { country: "de", format: "markdown" }); console.log(page.title, page.credits); const res = await client.contents({ urls: ["https://example.com/a", "https://example.com/b"] }); const account = await client.account(); ``` ## Python ```python title="Python SDK" import os from teneo import Teneo client = Teneo(api_key=os.environ["LIVE_WEB_DATA_API_KEY"], base_url="https://livewebdata.com/api") page = client.get_page("https://example.com/article", country="de", format="markdown") print(page.title, page.credits) res = client.contents(urls=["https://example.com/a", "https://example.com/b"]) account = client.account() ``` Both clients read the key from `LIVE_WEB_DATA_API_KEY` and the base URL from `LIVE_WEB_DATA_API_BASE_URL` when you don't pass them. ## Methods | JS | Python | Description | | --- | --- | --- | | `contents(request, { idempotencyKey })` | `contents(urls=…, country_code=…, text=…, html=…, livecrawl_timeout=…, render=…, render_fallback=…, idempotency_key=…)` | Raw `POST /v1/contents` (1–10 URLs). `render_fallback=False` / `renderFallback: false` turns the default browser fallback off | | `getPage(url, opts)` | `get_page(url, country=…, format=…)` | One URL → `{ title, url, content, credits, source }`; raises if that URL failed. `format: "html"` requests the HTML (`html: true`) | | `account()` | `account()` | `GET /v1/account` | ## Errors | Error | When | | --- | --- | | `TeneoAuthError` | 401 / 403 — key invalid, revoked or suspended | | `TeneoInsufficientCreditsError` | 402 | | `TeneoRateLimitError` | 429 after all retries (`retryAfter` / `retry_after`) | | `TeneoIdempotencyError` | 409 — the idempotency key was used with a different request, or its result expired (15 minutes) | | `TeneoRestrictedTargetError` | 451 | | `TeneoConnectionError` | Network error or timeout after all retries, or a `2xx` answer cut while it was read. The API may have received the request: repeat it with the error's `idempotencyKey` / `idempotency_key` | | `TeneoPendingError` | Still `202` when `maxWaitMs` / `max_wait` ran out; `idempotencyKey` / `idempotency_key` collects the result later | | `TeneoPageError` | `getPage` / `get_page`: the URL failed (`tag`, e.g. `CRAWL_NOT_FOUND`) | | `TeneoAPIError` | Base class (status, tag, requestId, and `idempotencyKey` / `idempotency_key`: the key of a `contents()` call, else `null` / `None`) | ## Retries Both SDKs retry `429` (respecting `Retry-After`), `5xx`, network errors, a `2xx` answer cut while it is read and — for `getPage` — `CRAWL_LIVECRAWL_TIMEOUT`, up to 3 times with exponential backoff and jitter. `400`, `401`, `402`, `403`, `409` and `451` are never retried. Configure with `maxRetries` / `max_retries`. Every attempt of one `contents()` call sends the same `Idempotency-Key`, so a retry picks up the first fetch and is never charged twice (render requests included). A fetch that runs longer than about 100 seconds answers `202`; the SDKs repeat it after `Retry-After` until the result arrives, for up to 5 minutes (`maxWaitMs` / `max_wait`). These repeats do not count as retries. See [Long requests](https://mcpscraper.net/docs/api-reference#long-requests-and-idempotency-key). Without your own key, the SDKs create one per `contents()` call. When the retries run out, the error carries that key: call `contents()` again with the same request and `{ idempotencyKey: e.idempotencyKey }` / `idempotency_key=e.idempotency_key` within 15 minutes, and a fetch the API already received returns its result instead of being charged again. **Better: always pass your own key** (one per logical request, for example per row of your input), so a request cut by a network error, a timeout or a deploy can be retried without a second charge even after your process restarts. --- # MCP for agents Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key. ## What is MCP The Model Context Protocol (MCP) is an open standard that lets AI agents discover and call external tools. Our MCP server exposes the Contents API as two tools, so an agent can read any public web page on its own. ## Server & auth | | Value | | --- | --- | | Server URL | `https://mcpscraper.net/mcp` | | Transport | Streamable HTTP (stateless): JSON-RPC over `POST`; `GET` and `DELETE` answer `405` | | Auth | `Authorization: Bearer ` | | Keys | Same keys as the REST API — create one under [API keys](https://mcpscraper.net/dashboard/keys) | Missing or invalid keys are rejected with HTTP 401 (suspended accounts: 403, "API access for this account is suspended. Contact support.") before any tool runs. ## Client setup Replace `YOUR_KEY` with your API key. In the dashboard, the [Integrations](https://mcpscraper.net/dashboard/integrations) page fills in your key for you. ### Claude Code Run this in your terminal. The server is then available in every Claude Code session for this project. ```bash claude mcp add --transport http live-web-data https://mcpscraper.net/mcp \ --header "Authorization: Bearer YOUR_KEY" ``` Add `--scope user` to make it available in all projects. Check with `claude mcp list`. ### Cursor Add this to your Cursor MCP config, globally or per project. File: `~/.cursor/mcp.json or .cursor/mcp.json` ```json { "mcpServers": { "live-web-data": { "url": "https://mcpscraper.net/mcp", "headers": { "Authorization": "Bearer YOUR_KEY" } } } } ``` Prefer an env var: `"Bearer ${env:LIVE_WEB_DATA_API_KEY}"`. ### VS Code Add this to your workspace. VS Code asks for the key the first time the server starts and stores it securely. File: `.vscode/mcp.json` ```json { "inputs": [ { "type": "promptString", "id": "live-web-data-key", "description": "Live Web Data API key", "password": true } ], "servers": { "live-web-data": { "type": "http", "url": "https://mcpscraper.net/mcp", "headers": { "Authorization": "Bearer ${input:live-web-data-key}" } } } } ``` Uses the top-level key `servers` (not `mcpServers`). Paste `YOUR_KEY` when prompted. ### Claude Desktop Claude Desktop connects to remote servers through the mcp-remote bridge. Add this to your Claude Desktop config and restart the app. File: `claude_desktop_config.json` ```json { "mcpServers": { "live-web-data": { "command": "npx", "args": [ "-y", "mcp-remote", "https://mcpscraper.net/mcp", "--header", "Authorization:${AUTH_HEADER}", "--transport", "http-only" ], "env": { "AUTH_HEADER": "Bearer YOUR_KEY" } } } } ``` Requires Node.js. Alternative on paid plans: Settings → Connectors → Add custom connector with request header `authorization` = `Bearer …` (where available; the server must be publicly reachable). ### Other MCP clients Any MCP client that supports Streamable HTTP can connect directly with the URL and the Authorization header. For stdio-only clients, use the mcp-remote bridge: ```bash npx -y mcp-remote https://mcpscraper.net/mcp \ --header "Authorization: Bearer YOUR_KEY" --transport http-only ``` Server URL: `https://mcpscraper.net/mcp` · Transport: Streamable HTTP · Auth header: `Authorization: Bearer ` ### REST API Prefer plain HTTP? The same API key works with the REST endpoint. ```bash curl -X POST https://livewebdata.com/api/v1/contents \ -H "Authorization: Bearer YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com/article"]}' ``` ## get_page_contents Fetch a public web page live and return its main content as clean Markdown, plain text or HTML. Read-only, idempotent. Single URL per call. | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `url` | string (http/https, at most 8,192 characters) | required | The public URL to fetch. | | `country` | ISO-3166 alpha-2 | `"us"` | Fetch the page as seen from this country, e.g. 'de', 'gb', 'us'. | | `format` | `"markdown"` \| `"text"` \| `"html"` | `"markdown"` | Output format. Prefer markdown. The page HTML is fetched and returned only for `"html"`. | | `max_characters` | integer 500–100000 | `8000` | Maximum characters of content to return. Smaller than the REST API's default (`text.maxCharacters`, 100,000) because the result goes into the agent's context; raise it when the content is cut. | | `verbosity` | `"compact"` \| `"full"` | `"compact"` | compact = main content only, full = whole page body. | The result is one text item with a short header, followed by the content. The full REST-style JSON is also returned as `structuredContent` (with `results[].html` only for `format: "html"`). ```markdown title="Tool result" # Example article Source: https://example.com/article · country: us · live · credits used: 1 · credits left: 8,411 # Example article Main content as clean Markdown… ``` ## get_account_status Check the remaining credits, plan and rate limit for the connected account. Agents should call it before large batches. No parameters, free. Returns the same numbers as `GET /v1/account`. ```text title="Tool result" Plan: Starter · Credits left: 8,412 · Rate limit: 2 URL/s (burst 4) · Status: active ``` ## Errors Errors are returned with `isError: true` and one actionable sentence the agent can follow. When nothing was charged, the message ends with “No credits were charged.” | Case | Message to the agent | | --- | --- | | 451 restricted target | This website is restricted and cannot be retrieved. Do not retry; use another source. | | `CRAWL_NOT_RENDERED` | The page has no usable text: it needs JavaScript rendering, or shows no text even when rendered. Try another source or a different URL for the same information. | | `CRAWL_NOT_FOUND` | Page not found (404/410). | | `SOURCE_NOT_AVAILABLE` | The website refused this request. Retry once with a different country, otherwise use another source. | | `SOURCE_BLOCKED` | Access denied by the website. Retry later or with a different country, otherwise use another source. | | `CRAWL_LIVECRAWL_TIMEOUT` | The page timed out. Retry once with a longer timeout, otherwise use another source. | | `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable. Retry in a few minutes, otherwise use another source. | | `RATE_LIMITED` | The rendering limit for the user's plan is reached. Do not retry rendering now; use another source. | | 402 insufficient credits | The account is out of credits. Tell the user to top up at …/dashboard/billing. | | 429 rate limited | Rate limit reached. Wait N seconds before retrying. | | 401 / 403 | The API key is invalid or suspended. Tell the user to check their key at …/dashboard/keys. | | Still fetching after 5 minutes | The page did not finish loading within 5 minutes, and the fetch may still be charged. Use another source. | ## Credits MCP calls are billed exactly like REST calls: 1 credit per successful page, failed requests (including access denied by the site) are free. They appear in Usage with channel `mcp`. See [Credits](https://mcpscraper.net/docs/api-reference#credits). --- # No-code recipes Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap. ## Before you start You need an API key from [API keys](https://mcpscraper.net/dashboard/keys). Every recipe sends the same request: | Setting | Value | | --- | --- | | Method | `POST` | | URL | `https://livewebdata.com/api/v1/contents` | | Header | `Authorization: Bearer YOUR_KEY` | | Body (JSON) | `{"urls": ["https://example.com/article"], "countryCode": "us"}` | The response is JSON: page content in `results[0].markdown`, the outcome in `statuses[0].status` and `statuses[0].error.tag`. ## n8n > **Native integration coming soon.** Use the built-in **HTTP Request** node for now. _Screenshot coming soon: n8n HTTP Request node_ | Field (HTTP Request node) | Value | | --- | --- | | Method | `POST` | | URL | `https://livewebdata.com/api/v1/contents` | | Authentication | Generic Credential Type → **Bearer Auth** (token = your key) — or **Header Auth** with Name `Authorization`, Value `Bearer YOUR_KEY` | | Send Body | on | | Body Content Type | `JSON` | | Specify Body | `Using JSON` → `{"urls": ["{{ $json.url }}"]}` | | Options → Response → Response Format | `JSON` (default: autodetect) | **URL lists:** feed one item per URL into the node. Use the node's **Options → Batching** (*Items per Batch* = your rate limit, *Batch Interval (ms)* = 1000) — or a **Loop Over Items** node plus a **Wait** node (1 second) to stay within your rate limit. ## Make > **Native integration coming soon.** Use the **HTTP** app's **Make a request** module for now. _Screenshot coming soon: Make HTTP "Make a request" module_ | Field (HTTP → Make a request) | Value | | --- | --- | | Authentication type | `API key` (stores the key in a keychain) — or `No authentication` plus the header below | | URL | `https://livewebdata.com/api/v1/contents` | | Method | `POST` | | Headers | `Authorization` = `Bearer YOUR_KEY` | | Body content type | `application/JSON` | | Body input method | `JSON string` | | Body content | `{"urls": ["https://example.com/article"]}` | | Parse response | `Yes` | **URL lists:** use an **Iterator** over your URLs before the HTTP module and set the scenario's scheduling so the number of operations per second stays within your rate limit. ## Zapier > **Native integration coming soon.** Use **Webhooks by Zapier** → **Custom Request** for now (not available on the Free plan). _Screenshot coming soon: Zapier Webhooks by Zapier – Custom Request_ | Field (Custom Request) | Value | | --- | --- | | Method | `POST` | | URL | `https://livewebdata.com/api/v1/contents` | | Data Pass-Through? | `False` | | Data | `{"urls": ["https://example.com/article"]}` | | Headers | `Authorization` = `Bearer YOUR_KEY`, `Content-Type` = `application/json` | The JSON response is available as fields in later steps. ## Tips - Failed pages return HTTP 200 with `statuses[0].status = "error"` — add a filter/router on that field. - Keep the key in the tool's credential store, not in plain step fields, wherever possible. ---