Crawlbase MCP

What it does

Crawlbase's own MCP server puts its crawling infrastructure behind three tools an agent can call directly:

• crawl — fetch a URL and return raw HTML
• crawl_markdown — return clean, LLM-ready Markdown with ads and boilerplate stripped
• crawl_screenshot — capture the rendered page as an image

Async crawling with cloud storage is supported for larger jobs that should not block the agent.

Why route a fetch through it

An agent's built-in fetch tool works until the page is a JavaScript application, sits behind an anti-bot check, or serves different content by region — which describes a large share of the pages anyone actually wants to scrape. Crawlbase handles JavaScript rendering, proxy rotation and anti-bot evasion server-side, plus device emulation and geo-targeting, so the agent gets the rendered page instead of a challenge screen or an empty shell.

The crawl_markdown tool is the one to reach for by default in an agent context: converting server-side means the model receives readable prose rather than spending its context window on markup it has to parse itself.

Authentication

Two tokens, and the distinction matters for both cost and behaviour: CRAWLBASE_TOKEN is the normal token used for plain HTML requests, while CRAWLBASE_JS_TOKEN is the JavaScript token required for JS-rendered pages and screenshots. Both are issued free on signup. When running the server in HTTP mode you can pass them per request as X-Crawlbase-Token and X-Crawlbase-JS-Token headers instead of environment variables, which is the better fit for a multi-tenant deployment.

Practical notes

Published as @crawlbase/mcp on npm under the MIT licence; version 1.3.0. Crawlbase describes its platform as web data infrastructure used by more than 70,000 developers. Usage is metered by the Crawlbase account behind your tokens rather than by the MCP server itself, so check your plan's request limits before pointing a loop at it.