> ## Documentation Index
> Fetch the complete documentation index at: https://outcrawl.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Keep a docs knowledge base in sync

> Crawl a docs site into embedded chunks, then re-embed only the pages that changed.

export const OutcrawlStyles = () => {
  if (typeof document !== 'undefined' && !document.getElementById('outcrawl-overrides')) {
    const s = document.createElement('style');
    s.id = 'outcrawl-overrides';
    s.textContent = `
      /* Match the product's radius grid (4 / 6 / 8 / 9999), not Mintlify's default. */
      .rounded-sm { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-md { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-lg { border-radius: 0.5rem !important; }   /* 8px, --r-panel */
      .rounded-l-sm { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-sm { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-md { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-md { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-lg { border-top-left-radius: 0.5rem !important; border-bottom-left-radius: 0.5rem !important; }
      .rounded-r-lg { border-top-right-radius: 0.5rem !important; border-bottom-right-radius: 0.5rem !important; }

      /* Cards: never pure white in light mode, matching the one-big-card surface tone. */
      .card { background-color: #fcfcfc !important; border-radius: 0.5rem !important; }
      html.dark .card { background-color: #141311 !important; }

      /* Docs hero box */
      .oc-hero { background-color: #fcfcfc; border: 1px solid #e5e2dc; }
      html.dark .oc-hero { background-color: #141311; border-color: #26241f; }
      html.dark .oc-hero h1 { color: #edeae4; }

      /* Thin, transparent-track, hide-until-hover scrollbar. */
      ::-webkit-scrollbar { width: 6px; height: 6px; background-color: transparent; }
      ::-webkit-scrollbar-track { background-color: transparent; }
      ::-webkit-scrollbar-thumb { background-color: rgba(155, 155, 155, 0.5); border-radius: 10px; transition: opacity 0.3s ease; opacity: 0; }
      ::-webkit-scrollbar-thumb:hover { background-color: rgba(155, 155, 155, 0.7); }
      *:hover::-webkit-scrollbar-thumb,
      *:focus::-webkit-scrollbar-thumb,
      *:active::-webkit-scrollbar-thumb { opacity: 1; }
      * { scrollbar-width: thin; scrollbar-color: rgba(155, 155, 155, 0.5) transparent; }
    `;
    document.head.appendChild(s);
  }
  return null;
};

<OutcrawlStyles />

A support assistant is only as good as the docs it retrieves, and docs change every week. Re-embedding a whole site every night wastes money, and never re-embedding it means the assistant quotes last quarter's settings. This program crawls a docs section into heading-aware chunks with metadata, stores them in Postgres with pgvector, and on every later run re-embeds only the pages whose content changed and removes the ones that are gone.

## What you need

* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* Postgres with the `pgvector` extension, in `DATABASE_URL`. The program creates its two tables.
* An embeddings key. The program uses OpenAI's `text-embedding-3-small` (`OPENAI_API_KEY`); any embedding model works if you change the vector size.
* `npm install @outcrawl/sdk openai pg` or `pip install outcrawl openai "psycopg[binary]"`.

## How it works

<Steps>
  <Step title="Crawl the section">
    A [crawl](/docs/crawl/overview) starts at the section root, stays inside `includePaths`, and streams each page as clean markdown with its title and fetch time. Nothing waits for the whole site.
  </Step>

  <Step title="Skip what has not changed">
    Each page's markdown is hashed and compared with the hash stored last time. Only new or changed pages are chunked and embedded. Pages that came back empty or behind an interstitial are skipped, so a bad fetch never replaces a good copy.
  </Step>

  <Step title="Chunk by heading">
    Chunks follow the page's own headings, up to about 1,500 characters, and keep the heading path (`Workflows > Triggers > schedule`) as metadata. Headings inside code blocks are ignored.
  </Step>

  <Step title="Write a page in one transaction">
    The page row and its chunks are replaced together, so a search never sees half of an update.
  </Step>

  <Step title="Remove what is gone, carefully">
    Pages missing from a crawl that finished below its `limit` are deleted with their chunks. A crawl that failed or hit the limit deletes nothing.
  </Step>
</Steps>

## The program

<CodeGroup>
  ````typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import { createHash } from 'node:crypto';
  import OpenAI from 'openai';
  import pg from 'pg';
  import { Outcrawl } from '@outcrawl/sdk';

  const SITE = 'https://docs.github.com/en/actions';
  const LIMIT = 1000; // a crawl that stops here may have missed pages, so nothing is deleted after it

  const oc = new Outcrawl();
  const openai = new OpenAI();
  const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });

  await db.query(`
    create extension if not exists vector;
    create table if not exists kb_pages (
      url text primary key, hash text not null, title text, crawled_at timestamptz not null);
    create table if not exists kb_chunks (
      id text primary key, url text not null references kb_pages (url) on delete cascade,
      heading text not null, content text not null, embedding vector(1536) not null);
  `);

  // Split on headings, keeping the heading path as metadata. Headings inside code blocks are not headings.
  function chunk(markdown: string, max = 1500): { heading: string; text: string }[] {
    const chunks: { heading: string; text: string }[] = [];
    let path: string[] = [];
    let lines: string[] = [];
    let inCode = false;
    const flush = () => {
      let piece = '';
      for (const paragraph of lines.join('\n').split(/\n{2,}/)) {
        if (piece !== '' && piece.length + paragraph.length > max) {
          chunks.push({ heading: path.join(' > '), text: piece.trim() });
          piece = '';
        }
        piece += `${paragraph}\n\n`;
      }
      if (piece.trim() !== '') chunks.push({ heading: path.join(' > '), text: piece.trim() });
      lines = [];
    };
    for (const line of markdown.split('\n')) {
      if (line.startsWith('```')) inCode = !inCode;
      const heading = inCode ? null : /^(#{1,3})\s+(.+)/.exec(line);
      if (heading !== null) {
        flush();
        path = [...path.slice(0, heading[1]!.length - 1), heading[2]!.trim()];
      }
      lines.push(line);
    }
    flush();
    return chunks;
  }

  const known = new Map(
    (await db.query<{ url: string; hash: string }>('select url, hash from kb_pages')).rows.map((r) => [r.url, r.hash]),
  );
  const seen = new Set<string>();
  let updated = 0;

  const crawl = await oc.crawl(SITE, {
    limit: LIMIT,
    includePaths: ['/en/actions/**'], // `**` spans path segments; `*` would stop at one
    formats: ['markdown'],
  });

  for await (const page of crawl) {
    // A page that failed or came back empty must not replace a good copy.
    if (page.statusCode !== 200 || page.markdown === undefined) continue;
    if (page.warnings?.some((w) => w.code === 'empty_capture' || w.code === 'blocked_interstitial')) continue;
    seen.add(page.url);

    const hash = createHash('sha256').update(page.markdown).digest('hex');
    if (known.get(page.url) === hash) continue; // unchanged since the last crawl

    const title = page.metadata.title ?? page.url;
    const chunks = chunk(page.markdown);
    if (chunks.length === 0) continue;
    const embedded = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: chunks.map((c) => `${title}\n${c.heading}\n\n${c.text}`),
    });

    const tx = await db.connect();
    try {
      await tx.query('begin');
      await tx.query(
        `insert into kb_pages (url, hash, title, crawled_at) values ($1, $2, $3, $4)
         on conflict (url) do update set hash = $2, title = $3, crawled_at = $4`,
        [page.url, hash, title, page.metadata.scrapedAt],
      );
      await tx.query('delete from kb_chunks where url = $1', [page.url]);
      for (const [i, c] of chunks.entries()) {
        await tx.query(
          'insert into kb_chunks (id, url, heading, content, embedding) values ($1, $2, $3, $4, $5)',
          [`${page.url}#${i}`, page.url, c.heading, c.text, JSON.stringify(embedded.data[i]!.embedding)],
        );
      }
      await tx.query('commit');
      updated += 1;
    } catch (err) {
      await tx.query('rollback');
      throw err;
    } finally {
      tx.release();
    }
  }

  // Pages the site no longer serves, removed only after a crawl that finished below its limit.
  let removed = 0;
  if (crawl.status === 'completed' && crawl.completed < LIMIT) {
    const gone = [...known.keys()].filter((url) => !seen.has(url));
    removed = (await db.query('delete from kb_pages where url = any($1)', [gone])).rowCount ?? 0;
  }

  console.log(`crawl ${crawl.id}: ${seen.size} pages read, ${updated} re-embedded, ${removed} removed`);
  await db.end();
  ````

  ````python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import asyncio
  import hashlib
  import json
  import os
  import re

  import psycopg
  from openai import AsyncOpenAI
  from outcrawl import Outcrawl

  SITE = "https://docs.github.com/en/actions"
  LIMIT = 1000  # a crawl that stops here may have missed pages, so nothing is deleted after it

  SCHEMA = """
  create extension if not exists vector;
  create table if not exists kb_pages (
    url text primary key, hash text not null, title text, crawled_at timestamptz not null);
  create table if not exists kb_chunks (
    id text primary key, url text not null references kb_pages (url) on delete cascade,
    heading text not null, content text not null, embedding vector(1536) not null);
  """


  def chunk(markdown: str, limit: int = 1500) -> list[dict]:
      """Split on headings, keeping the heading path as metadata. Headings inside code blocks are not headings."""
      chunks: list[dict] = []
      path: list[str] = []
      lines: list[str] = []
      in_code = False

      def flush() -> None:
          piece = ""
          for paragraph in re.split(r"\n{2,}", "\n".join(lines)):
              if piece and len(piece) + len(paragraph) > limit:
                  chunks.append({"heading": " > ".join(path), "text": piece.strip()})
                  piece = ""
              piece += paragraph + "\n\n"
          if piece.strip():
              chunks.append({"heading": " > ".join(path), "text": piece.strip()})
          lines.clear()

      for line in markdown.split("\n"):
          if line.startswith("```"):
              in_code = not in_code
          heading = None if in_code else re.match(r"^(#{1,3})\s+(.+)", line)
          if heading:
              flush()
              path[:] = [*path[: len(heading[1]) - 1], heading[2].strip()]
          lines.append(line)
      flush()
      return chunks


  async def main() -> None:
      llm = AsyncOpenAI()
      async with Outcrawl() as oc, await psycopg.AsyncConnection.connect(os.environ["DATABASE_URL"], autocommit=True) as db:
          await db.execute(SCHEMA)
          known = dict(await (await db.execute("select url, hash from kb_pages")).fetchall())
          seen: set[str] = set()
          updated = 0

          crawl = await oc.crawl(SITE, limit=LIMIT, includePaths=["/en/actions/**"], formats=["markdown"])
          async for page in crawl:
              # A page that failed or came back empty must not replace a good copy.
              markdown = page.get("markdown")
              if page["statusCode"] != 200 or not markdown:
                  continue
              if any(w["code"] in ("empty_capture", "blocked_interstitial") for w in page.get("warnings", [])):
                  continue
              seen.add(page["url"])

              digest = hashlib.sha256(markdown.encode()).hexdigest()
              if known.get(page["url"]) == digest:
                  continue  # unchanged since the last crawl

              title = page["metadata"].get("title") or page["url"]
              chunks = chunk(markdown)
              if not chunks:
                  continue
              embedded = await llm.embeddings.create(
                  model="text-embedding-3-small",
                  input=[f"{title}\n{c['heading']}\n\n{c['text']}" for c in chunks],
              )

              async with db.transaction():
                  await db.execute(
                      """insert into kb_pages (url, hash, title, crawled_at) values (%s, %s, %s, %s)
                         on conflict (url) do update set hash = excluded.hash, title = excluded.title,
                         crawled_at = excluded.crawled_at""",
                      (page["url"], digest, title, page["metadata"]["scrapedAt"]),
                  )
                  await db.execute("delete from kb_chunks where url = %s", (page["url"],))
                  for i, c in enumerate(chunks):
                      await db.execute(
                          "insert into kb_chunks (id, url, heading, content, embedding) values (%s, %s, %s, %s, %s)",
                          (f"{page['url']}#{i}", page["url"], c["heading"], c["text"], json.dumps(embedded.data[i].embedding)),
                      )
              updated += 1

          # Pages the site no longer serves, removed only after a crawl that finished below its limit.
          removed = 0
          job = crawl.job or {}
          if job.get("status") == "completed" and job.get("completed", LIMIT) < LIMIT:
              gone = [url for url in known if url not in seen]
              removed = (await db.execute("delete from kb_pages where url = any(%s)", (gone,))).rowcount

          print(f"crawl {job.get('id')}: {len(seen)} pages read, {updated} re-embedded, {removed} removed")


  asyncio.run(main())
  ````
</CodeGroup>

Run it nightly from any scheduler, for example `0 3 * * * cd /srv/kb && npx tsx kb.ts` (or `python kb.py`) in cron. The first run embeds everything; later runs embed only the difference.

## What you get back

The console, on a night when the docs team shipped two edits and retired a page:

```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
crawl crw_01K5ZT4N8Q: 412 pages read, 2 re-embedded, 1 removed
```

One row of `kb_chunks`:

```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
  "id": "https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#14",
  "url": "https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows",
  "heading": "Events that trigger workflows > schedule",
  "content": "## schedule\n\nThe schedule event allows you to trigger a workflow at a scheduled time. You can schedule a workflow to run at specific UTC times using POSIX cron syntax...",
  "embedding": "[0.0132, -0.0417, 0.0088, ...]"
}
```

Retrieval is one query: `select url, heading, content from kb_chunks order by embedding <=> $1 limit 8`, and every chunk carries the URL your assistant cites.

## What it costs

A crawl is 1 credit per page. A 400-page docs section is **400 credits per crawl**, whether or not the pages changed, because a page has to be read to know.

| Schedule | Crawls a month | Credits a month | Fits |
| - | - | - | - |
| Nightly | 30 | about 12,000 | Pro (50,000) |
| Twice a week | 9 | about 3,600 | Builder (10,000) |
| Weekly | 4 to 5 | about 1,800 | Builder (10,000) |

The saving is on your side: only changed pages are embedded and written, so a quiet night costs your embedding provider almost nothing.

## Make it yours

* Put a [monitor](/docs/monitors/overview) on the docs changelog. It calls a webhook endpoint of yours when the page changes, and your endpoint starts this crawl the moment something ships instead of waiting for night.
* Crawl several sections or sites in one run by looping over `[root, includePaths]` pairs; the stored hashes already work for any URL.
* Store `page.metadata.language` too and filter retrieval by the user's language.
* Replace pgvector with your own vector store: the upsert is the only part that changes.

## Related

<CardGroup cols={2}>
  <Card title="Crawl" icon="sitemap" href="/docs/crawl/overview">Limits, paths and streaming.</Card>
  <Card title="Crawl jobs" icon="history" href="/docs/crawl/jobs">Recover a crawl's pages if the stream drops.</Card>
  <Card title="Monitors" icon="eye" href="/docs/monitors/overview">A webhook when a page changes.</Card>
  <Card title="Pricing" icon="tag" href="/docs/billing/pricing">Every rate on this page.</Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.