Web reader for agents · MCP

Agent Web

A URL in, clean LLM-ready markdown out — a free, agent-native web reader for AI agents and developers. Call it over plain HTTP, or install it as an MCP server and let an agent read pages as a tool. No account, no API key.

Open the live reader at agent-web.foomworks.workers.dev →

Running an OpenClaw agent? Agent Web is published as an installable skill on ClawHub — clawhub.ai/foomworks/agent-web.

What it does

1. 01

   Clean, LLM-ready markdown

   Give it the URL of a publicly reachable, server-rendered page; get back the title, a word count, and the readable content as markdown — the boilerplate stripped, ready to drop into a prompt or a RAG pipeline. No account, no API key.

2. 02

   Polite by design (the charter-clean line)

   It honors the origin's robots.txt for our user-agent, identifies itself honestly, and does a single read-only GET — it never spoofs a browser, rotates proxies, or bypasses anti-bot, CAPTCHA, paywalls or login walls. That restraint is the product, not a limitation: it is the clean, defensible subset of web reading.

3. 03

   Agent-native (MCP + A2A)

   Beyond the plain HTTP endpoint, it speaks the Model Context Protocol over Streamable HTTP, so an MCP-capable agent can install it and call the read_url / read_url_preview tools directly. Discovery manifests (mcp.json, an A2A agent card, and OpenAPI) are all served for automatic wiring.

What "polite reader" means

Plenty of tools will render JavaScript, rotate residential proxies, and push through CAPTCHAs and paywalls. Agent Web deliberately does none of that. It is built for the large, clean case — public, server-rendered pages you are allowed to read — and it draws a bright line at evasion:

• Honors robots.txt for our user-agent before fetching; a disallowed path is refused, not fetched.
• Identifies honestly with a descriptive User-Agent — never spoofs a browser to look human.
• Read-only GET of one caller-supplied URL. No crawl, no proxy rotation, no session replay.
• Never bypasses anti-bot, CAPTCHA, paywalls or login walls.
• Blocks private, loopback, link-local and internal hosts (SSRF protection); every redirect hop is re-validated.

The full policy is served at /policy.

Call it over HTTP

The simplest path — a plain GET from anything that speaks HTTP.

1 · Read a page as markdown

curl 'https://agent-web.foomworks.workers.dev/read?url=https://example.com'

You get back JSON with the resolved url, the title, a words count, and the page as markdown.

2 · Peek before pulling the whole thing

curl 'https://agent-web.foomworks.workers.dev/read/preview?url=https://example.com'

Returns just the first ~600 characters and a truncation flag — handy when an agent is deciding whether a page is worth the full read.

Install it as an MCP server

Agent Web speaks the Model Context Protocol over Streamable HTTP at POST /mcp, exposing two free tools — read_url and read_url_preview. Point an MCP-capable client at the endpoint and the tools appear automatically:

# List the tools the server offers
curl -s -X POST https://agent-web.foomworks.workers.dev/mcp \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

The server descriptor lives at /.well-known/mcp.json, with an A2A agent card at /.well-known/agent-card.json and an OpenAPI description at /openapi.json — so agents and tooling can wire it up without reading this page.

Endpoints

All free — no account, no API key, call from anything:

• GET /read?url= free Fetch a URL and return LLM-ready markdown — title, word count, and the markdown body.
• GET /read/preview?url= free The first ~600 characters of the markdown — a cheap look before pulling the full text.
• POST /mcp free MCP server (Streamable HTTP, JSON-RPC 2.0): the agent-native read_url / read_url_preview tools.
• GET /.well-known/mcp.json free MCP server descriptor for automatic discovery.
• GET /.well-known/agent-card.json free A2A agent card.
• GET /openapi.json free OpenAPI 3.1 description of the public surface.
• GET /policy free The acceptable-use policy — the polite-reader guarantees, in full.
• GET /health free Service status and reader configuration.
• GET /stats free Daily usage counters.

Agent Web is built and maintained by an autonomously operated software workshop under human oversight. It reads only public, server-rendered pages you are entitled to read, honoring each site's robots.txt; you are responsible for the URLs you pass and your right to use the content returned. Provided as-is. No affiliation with, or endorsement by, any site you read is implied.