Crawl the web as an agent would.
Headless Chromium, clean Markdown, and structured JSON — built for models that need the page, not the chrome.
100 pages free every month. No card required.
POST /v1/crawls
{ "url": "https://example.com" }
→ 202 status: queued
GET /v1/crawls/:id/pages
→ clean Markdown, one entry per page
GET /v1/crawls/:id/map
→ every URL discovered
GET /v1/crawls/:id/cleaned
→ typed areas, headings, site rollupRendered in Chromium
Playwright loads the page the way a browser does, so client-side content is there.
Robots and SSRF closed
Robots rules fail closed and private network targets are blocked at fetch time.
Markdown and a site map
Main content per page, plus a map of every URL the crawl discovered.
Typed JSON on request
Headings, areas, and a site-level rollup for models that want structure.
How it works
From a URL to something a model can read
One task API. Start a crawl, then read the results in whichever shape your agent needs.
Point at a URL
POST a seed with an Idempotency-Key. The crawl is queued and returns 202 immediately.
Read Markdown
Pull pages as clean Markdown with the sitemap and doors — main content, not a blob of chrome.
Ask for JSON
GET /cleaned returns typed areas, headings, and a site rollup once cleanup finishes.
API
Two calls to a clean site
Authenticate with a bearer key. Every crawl is tenant-private and idempotent.
curl -X POST https://crawler.lumen-labs.ai/v1/crawls \
-H "Authorization: Bearer $LUMEN_CRAWLER_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"url":"https://example.com"}'
curl https://crawler.lumen-labs.ai/v1/crawls/$ID/cleaned \
-H "Authorization: Bearer $LUMEN_CRAWLER_KEY"| Method | Endpoint | Returns |
|---|---|---|
| POST | /v1/crawls | Start a crawl from a seed URL |
| GET | /v1/crawls/:id | Status, counts, and stop reason |
| GET | /v1/crawls/:id/pages | Paginated pages as Markdown or cleaned JSON |
| GET | /v1/crawls/:id/map | Every URL discovered, as a map |
| GET | /v1/crawls/:id/cleaned | Site-level typed JSON rollup |
| POST | /v1/crawls/:id/cancel | Stop a running crawl |
Pricing
Pay for pages, not seats
Usage is counted in successfully fetched pages and resets each UTC month.
Give your agent the page, not the chrome.
Create an account, grab a key, and run your first crawl in a minute.