Overview

Crawl walks a site from a seed URL and streams every page as it is scraped. Use it when the set of URLs is not known in advance, not for one page (that is scrape), and not for content behind a login flow. A crawl always streams as newline-delimited JSON: a header frame, then one JSON object per page as it is fetched, and answers a job id immediately so you can follow along without holding the whole crawl in memory. The rate is 1 credit per page, the same rate as a direct scrape.

import { Outcrawl } from '@outcrawl/sdk';

const oc = new Outcrawl();
const crawl = await oc.crawl('https://example.com', { limit: 100 });
console.log(crawl.id); // store this to reach the crawl later
for await (const page of crawl) {
  console.log(page.url, page.markdown);
}

import asyncio
from outcrawl import Outcrawl

async def main():
    async with Outcrawl() as oc:
        crawl = await oc.crawl("https://example.com", limit=100)
        async for page in crawl:
            print(page["url"], page.get("markdown"))
        print(crawl.job["id"])  # store this to reach the crawl later

asyncio.run(main())

curl https://api.outcrawl.ai/v1/crawl \
  -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 100
  }'

A crawl is read once, as it streams. To read it again, or after the connection drops, use its id with the job lifecycle calls.

The MCP tool outcrawl_crawl folds the whole stream into one tool response. A model gets the complete set of pages back from a single call rather than reading a stream itself.

Job lifecycle

Poll status, read results and cancel a crawl by its id.

Welcome

SDKs

CLI

API

Scrape

Crawl

Search

Browsers

Agents

Monitors

Account

Billing

FAQ

Job lifecycle