Site → Markdown, for RAG

Website Crawler to Markdown

Give it one start URL. It follows same-origin links, renders each page in a real browser, and hands back every page as clean Markdown — built for ingesting a docs site or knowledge base into an LLM or RAG pipeline.

GET /v1/crawl?url=...&limit=10&format=markdown starts at your URL, walks same-origin links breadth-first, and returns each page's main content as Markdown — the same clean extraction as /v1/read, run across a whole site in one call instead of one page.

Try it

request
curl "https://api.rasterly.dev/v1/crawl?url=https://docs.example.com&limit=15&format=markdown" \
  -H "X-Api-Key: sk_live_..."
response — text/markdown (pages joined with ---)
# Getting Started
<https://docs.example.com/getting-started>

Install the SDK and configure your first project...

---

# Authentication
<https://docs.example.com/auth>

All requests are authenticated with a bearer token...

Drop format=markdown for the default format=json instead — a structured { start, count, pages: [{ url, title, markdown }] } payload you can iterate over and chunk yourself.

How it's bounded

Why rasterly

Each page in a crawl is one render on the same reused-browser engine behind every rasterly endpoint — a warm render is about a second of CPU, which is why a 15-page crawl costs the same per page as a single screenshot: from $0.90 per 1,000 renders, 100/mo free to prototype with.

FAQ

How many pages does it crawl?
Up to limit pages, default 10, hard-capped at 30 per call. It's meant for a docs site, a knowledge base, or a section of a site, not an unbounded whole-domain crawl.
Does it follow links off the starting domain?
No — it's same-origin only by design, breadth-first from the start URL. It also skips asset links (PDF, images, CSS, JS) automatically.
What does each page come back as?
Clean Markdown, same extraction as /v1/read: JS rendered first, nav/ads/boilerplate stripped, each page prefixed with its title and source URL so you can attribute chunks after you split them.
Is this good for building a RAG index?
Yes — that's the primary use case. One call turns a docs site into a set of clean, chunkable Markdown documents ready for embedding, instead of you writing a crawler and an HTML-to-text pass yourself.
Start free — 100 renders Try the live demo
More: Best screenshot APIs URL to Markdown API HTML to PDF API Web scraping / structured data API rasterly MCP server