> ## Documentation Index
> Fetch the complete documentation index at: https://outcrawl.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Walk a site from a seed URL, stream every page as it is fetched.

export const OutcrawlStyles = () => {
  if (typeof document !== 'undefined' && !document.getElementById('outcrawl-overrides')) {
    const s = document.createElement('style');
    s.id = 'outcrawl-overrides';
    s.textContent = `
      /* Match the product's radius grid (4 / 6 / 8 / 9999), not Mintlify's default. */
      .rounded-sm { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-md { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-lg { border-radius: 0.5rem !important; }   /* 8px, --r-panel */
      .rounded-l-sm { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-sm { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-md { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-md { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-lg { border-top-left-radius: 0.5rem !important; border-bottom-left-radius: 0.5rem !important; }
      .rounded-r-lg { border-top-right-radius: 0.5rem !important; border-bottom-right-radius: 0.5rem !important; }

      /* Cards: never pure white in light mode, matching the one-big-card surface tone. */
      .card { background-color: #fcfcfc !important; border-radius: 0.5rem !important; }
      html.dark .card { background-color: #141311 !important; }

      /* Docs hero box */
      .oc-hero { background-color: #fcfcfc; border: 1px solid #e5e2dc; }
      html.dark .oc-hero { background-color: #141311; border-color: #26241f; }
      html.dark .oc-hero h1 { color: #edeae4; }

      /* Thin, transparent-track, hide-until-hover scrollbar. */
      ::-webkit-scrollbar { width: 6px; height: 6px; background-color: transparent; }
      ::-webkit-scrollbar-track { background-color: transparent; }
      ::-webkit-scrollbar-thumb { background-color: rgba(155, 155, 155, 0.5); border-radius: 10px; transition: opacity 0.3s ease; opacity: 0; }
      ::-webkit-scrollbar-thumb:hover { background-color: rgba(155, 155, 155, 0.7); }
      *:hover::-webkit-scrollbar-thumb,
      *:focus::-webkit-scrollbar-thumb,
      *:active::-webkit-scrollbar-thumb { opacity: 1; }
      * { scrollbar-width: thin; scrollbar-color: rgba(155, 155, 155, 0.5) transparent; }
    `;
    document.head.appendChild(s);
  }
  return null;
};

<OutcrawlStyles />

Crawl walks a site from a seed URL and streams every page as it is scraped. Use it when the set of URLs is not known in advance, not for one page (that is [scrape](/docs/scrape/overview)), and not for content behind a login flow.

A crawl always streams as newline-delimited JSON: a header frame, then one JSON object per page as it is fetched, and answers a job id immediately so you can follow along without holding the whole crawl in memory.

The rate is **1 credit per page**, the same rate as a direct scrape.

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import { Outcrawl } from '@outcrawl/sdk';

  const oc = new Outcrawl();
  const crawl = await oc.crawl('https://example.com', { limit: 100 });
  console.log(crawl.id); // store this to reach the crawl later
  for await (const page of crawl) {
    console.log(page.url, page.markdown);
  }
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import asyncio
  from outcrawl import Outcrawl

  async def main():
      async with Outcrawl() as oc:
          crawl = await oc.crawl("https://example.com", limit=100)
          async for page in crawl:
              print(page["url"], page.get("markdown"))
          print(crawl.job["id"])  # store this to reach the crawl later

  asyncio.run(main())
  ```

  ```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  curl https://api.outcrawl.ai/v1/crawl \
    -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com",
      "limit": 100
    }'
  ```
</CodeGroup>

A crawl is read once, as it streams. To read it again, or after the connection drops, use its id with the [job lifecycle](/docs/crawl/jobs) calls.

<Note>
  The MCP tool `outcrawl_crawl` folds the whole stream into one tool response. A model gets the complete set of pages back from a single call rather than reading a stream itself.
</Note>

<Card title="Job lifecycle" icon="route" href="/docs/crawl/jobs">
  Poll status, read results and cancel a crawl by its id.
</Card>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.