Responsible scraping with MCP: a security and compliance review

Before a team standardizes on a scraping MCP server, someone has to approve it: what it can reach, who can call it, what it logs and how it behaves toward the sites it reads. This guide gives that reviewer a scope statement, a short section on robots.txt and site terms, and a checklist. It is general information, not legal advice.

What is in and out of scope

A scraping MCP server turns a model's decision into a request to a real website. That makes scope a policy question, not just a technical one. A reasonable default scope for agents:

In scope

• Public pages that anyone can open without logging in.
• Documentation, reference pages, changelogs, public articles and your own sites.
• Polite request rates, with a stop on errors.

Out of scope

• Content behind logins, paywalls or subscriptions, and private networks.
• Ways around bot protection or access controls.
• Collecting personal data or building profiles of identifiable people.
• Bulk copying of a specific named website without checking its terms and any official API first.

Hosted providers write similar limits into their acceptable-use policies. Live Web Data API's, for example, prohibits accessing non-public content and placing unreasonable load on websites. A self-hosted server has no such policy unless you write one.

robots.txt and terms

robots.txt is how site operators state which paths automated clients should avoid. RFC 9309 standardizes the format, and it is explicit that the rules are not a form of access authorization. They are still the clearest statement of what a site wants from automated clients, and following them is the polite default for an agent.

Site terms of use are separate. They can restrict automated access even where robots.txt allows a path, and whether they bind you depends on the site and your jurisdiction. Your review should record:

• whether the server checks robots.txt, and whether that can be turned off;
• whether the provider honors opt-out and takedown requests;
• who on your side reads the terms of sites the agent will use most.

Record the provider's actual words, not your reading of them. Live Web Data API's terms, for example, put compliance with site terms on the customer, say Teneo Labs blocks a website for all customers once it has verified the operator's request, and say it may honour robots.txt directives but is not obliged to (Terms, Section 8.6). A reviewer should treat that as "robots.txt handling is the customer's policy", not as a promise. This is general information, not legal advice.

Fewer blocks on public pages, the honest way

Agents that hammer a site get rate limited, and rate limits are a fair response. The way to see fewer blocks on public pages is the boring one: read fewer pages, cache what you read with a maximum age, space out requests to the same host, stop on 403 and 429 responses instead of retrying, and identify your use clearly where the server allows it. A server that advertises ways around bot protection is a compliance risk to flag in review, not a feature to rely on.

Permissions and the MCP side

The MCP specification's security best practices cover risks such as token passthrough and session handling. For a scraping server, three questions matter most:

1. Who can call it? A stdio server is reachable only by the client that started it. An HTTP server is reachable over the network; Scrapling's HTTP mode, for example, requires an auth token unless you disable it. Keep that default.
2. What can the tools do? The specification lets servers describe tools with annotations such as read-only hints. A read-only page reader is a smaller risk than a browser tool that can type into forms or run code.
3. What does the content do to the agent? Pages are untrusted input. A page can contain text that tries to instruct the model. Keep scraping tools separate from tools with side effects, and require confirmation before actions.

Approving a server: checklist

• The tools and their scope are listed and match the job.
• Auth uses per-environment keys, tokens or OAuth, and keys rotate.
• HTTP transports are bound to localhost or require a token.
• There is an acceptable-use policy and a stated robots.txt position.
• Logs of URLs and content follow your retention policy.
• Rate limits per site are set in config, outside the model's control.
• Logins, paywalls and personal data are out of scope by written policy.
• Someone re-reviews the server when it ships a major release.

The rubric's compliance row records these facts per server; see the rubric guide. For where data goes when the server is hosted, read the hosted vs self-hosted guide.

If a content API fits

Live Web Data API is a hosted reader for public pages with a published acceptable-use policy. Sign-up is for businesses only; only Business, Enterprise and Custom sign-ups are reviewed before checkout. Its MCP server does not render JavaScript, and the product does not crawl or search. To see its tools and error behavior, read the MCP docs.