Keep a docs knowledge base in sync

A support assistant is only as good as the docs it retrieves, and docs change every week. Re-embedding a whole site every night wastes money, and never re-embedding it means the assistant quotes last quarter’s settings. This program crawls a docs section into heading-aware chunks with metadata, stores them in Postgres with pgvector, and on every later run re-embeds only the pages whose content changed and removes the ones that are gone.

What you need

An Outcrawl API key in OUTCRAWL_API_KEY.
Postgres with the pgvector extension, in DATABASE_URL. The program creates its two tables.
An embeddings key. The program uses OpenAI’s text-embedding-3-small (OPENAI_API_KEY); any embedding model works if you change the vector size.
npm install @outcrawl/sdk openai pg or pip install outcrawl openai "psycopg[binary]".

How it works

Crawl the section

A crawl starts at the section root, stays inside includePaths, and streams each page as clean markdown with its title and fetch time. Nothing waits for the whole site.

Skip what has not changed

Each page’s markdown is hashed and compared with the hash stored last time. Only new or changed pages are chunked and embedded. Pages that came back empty or behind an interstitial are skipped, so a bad fetch never replaces a good copy.

Chunk by heading

Chunks follow the page’s own headings, up to about 1,500 characters, and keep the heading path (Workflows > Triggers > schedule) as metadata. Headings inside code blocks are ignored.

Write a page in one transaction

The page row and its chunks are replaced together, so a search never sees half of an update.

Remove what is gone, carefully

Pages missing from a crawl that finished below its limit are deleted with their chunks. A crawl that failed or hit the limit deletes nothing.

The program

import { createHash } from 'node:crypto';
import OpenAI from 'openai';
import pg from 'pg';
import { Outcrawl } from '@outcrawl/sdk';

const SITE = 'https://docs.github.com/en/actions';
const LIMIT = 1000; // a crawl that stops here may have missed pages, so nothing is deleted after it

const oc = new Outcrawl();
const openai = new OpenAI();
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });

await db.query(`
  create extension if not exists vector;
  create table if not exists kb_pages (
    url text primary key, hash text not null, title text, crawled_at timestamptz not null);
  create table if not exists kb_chunks (
    id text primary key, url text not null references kb_pages (url) on delete cascade,
    heading text not null, content text not null, embedding vector(1536) not null);
`);

// Split on headings, keeping the heading path as metadata. Headings inside code blocks are not headings.
function chunk(markdown: string, max = 1500): { heading: string; text: string }[] {
  const chunks: { heading: string; text: string }[] = [];
  let path: string[] = [];
  let lines: string[] = [];
  let inCode = false;
  const flush = () => {
    let piece = '';
    for (const paragraph of lines.join('\n').split(/\n{2,}/)) {
      if (piece !== '' && piece.length + paragraph.length > max) {
        chunks.push({ heading: path.join(' > '), text: piece.trim() });
        piece = '';
      }
      piece += `${paragraph}\n\n`;
    }
    if (piece.trim() !== '') chunks.push({ heading: path.join(' > '), text: piece.trim() });
    lines = [];
  };
  for (const line of markdown.split('\n')) {
    if (line.startsWith('```')) inCode = !inCode;
    const heading = inCode ? null : /^(#{1,3})\s+(.+)/.exec(line);
    if (heading !== null) {
      flush();
      path = [...path.slice(0, heading[1]!.length - 1), heading[2]!.trim()];
    }
    lines.push(line);
  }
  flush();
  return chunks;
}

const known = new Map(
  (await db.query<{ url: string; hash: string }>('select url, hash from kb_pages')).rows.map((r) => [r.url, r.hash]),
);
const seen = new Set<string>();
let updated = 0;

const crawl = await oc.crawl(SITE, {
  limit: LIMIT,
  includePaths: ['/en/actions/**'], // `**` spans path segments; `*` would stop at one
  formats: ['markdown'],
});

for await (const page of crawl) {
  // A page that failed or came back empty must not replace a good copy.
  if (page.statusCode !== 200 || page.markdown === undefined) continue;
  if (page.warnings?.some((w) => w.code === 'empty_capture' || w.code === 'blocked_interstitial')) continue;
  seen.add(page.url);

  const hash = createHash('sha256').update(page.markdown).digest('hex');
  if (known.get(page.url) === hash) continue; // unchanged since the last crawl

  const title = page.metadata.title ?? page.url;
  const chunks = chunk(page.markdown);
  if (chunks.length === 0) continue;
  const embedded = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: chunks.map((c) => `${title}\n${c.heading}\n\n${c.text}`),
  });

  const tx = await db.connect();
  try {
    await tx.query('begin');
    await tx.query(
      `insert into kb_pages (url, hash, title, crawled_at) values ($1, $2, $3, $4)
       on conflict (url) do update set hash = $2, title = $3, crawled_at = $4`,
      [page.url, hash, title, page.metadata.scrapedAt],
    );
    await tx.query('delete from kb_chunks where url = $1', [page.url]);
    for (const [i, c] of chunks.entries()) {
      await tx.query(
        'insert into kb_chunks (id, url, heading, content, embedding) values ($1, $2, $3, $4, $5)',
        [`${page.url}#${i}`, page.url, c.heading, c.text, JSON.stringify(embedded.data[i]!.embedding)],
      );
    }
    await tx.query('commit');
    updated += 1;
  } catch (err) {
    await tx.query('rollback');
    throw err;
  } finally {
    tx.release();
  }
}

// Pages the site no longer serves, removed only after a crawl that finished below its limit.
let removed = 0;
if (crawl.status === 'completed' && crawl.completed < LIMIT) {
  const gone = [...known.keys()].filter((url) => !seen.has(url));
  removed = (await db.query('delete from kb_pages where url = any($1)', [gone])).rowCount ?? 0;
}

console.log(`crawl ${crawl.id}: ${seen.size} pages read, ${updated} re-embedded, ${removed} removed`);
await db.end();

import asyncio
import hashlib
import json
import os
import re

import psycopg
from openai import AsyncOpenAI
from outcrawl import Outcrawl

SITE = "https://docs.github.com/en/actions"
LIMIT = 1000  # a crawl that stops here may have missed pages, so nothing is deleted after it

SCHEMA = """
create extension if not exists vector;
create table if not exists kb_pages (
  url text primary key, hash text not null, title text, crawled_at timestamptz not null);
create table if not exists kb_chunks (
  id text primary key, url text not null references kb_pages (url) on delete cascade,
  heading text not null, content text not null, embedding vector(1536) not null);
"""


def chunk(markdown: str, limit: int = 1500) -> list[dict]:
    """Split on headings, keeping the heading path as metadata. Headings inside code blocks are not headings."""
    chunks: list[dict] = []
    path: list[str] = []
    lines: list[str] = []
    in_code = False

    def flush() -> None:
        piece = ""
        for paragraph in re.split(r"\n{2,}", "\n".join(lines)):
            if piece and len(piece) + len(paragraph) > limit:
                chunks.append({"heading": " > ".join(path), "text": piece.strip()})
                piece = ""
            piece += paragraph + "\n\n"
        if piece.strip():
            chunks.append({"heading": " > ".join(path), "text": piece.strip()})
        lines.clear()

    for line in markdown.split("\n"):
        if line.startswith("```"):
            in_code = not in_code
        heading = None if in_code else re.match(r"^(#{1,3})\s+(.+)", line)
        if heading:
            flush()
            path[:] = [*path[: len(heading[1]) - 1], heading[2].strip()]
        lines.append(line)
    flush()
    return chunks


async def main() -> None:
    llm = AsyncOpenAI()
    async with Outcrawl() as oc, await psycopg.AsyncConnection.connect(os.environ["DATABASE_URL"], autocommit=True) as db:
        await db.execute(SCHEMA)
        known = dict(await (await db.execute("select url, hash from kb_pages")).fetchall())
        seen: set[str] = set()
        updated = 0

        crawl = await oc.crawl(SITE, limit=LIMIT, includePaths=["/en/actions/**"], formats=["markdown"])
        async for page in crawl:
            # A page that failed or came back empty must not replace a good copy.
            markdown = page.get("markdown")
            if page["statusCode"] != 200 or not markdown:
                continue
            if any(w["code"] in ("empty_capture", "blocked_interstitial") for w in page.get("warnings", [])):
                continue
            seen.add(page["url"])

            digest = hashlib.sha256(markdown.encode()).hexdigest()
            if known.get(page["url"]) == digest:
                continue  # unchanged since the last crawl

            title = page["metadata"].get("title") or page["url"]
            chunks = chunk(markdown)
            if not chunks:
                continue
            embedded = await llm.embeddings.create(
                model="text-embedding-3-small",
                input=[f"{title}\n{c['heading']}\n\n{c['text']}" for c in chunks],
            )

            async with db.transaction():
                await db.execute(
                    """insert into kb_pages (url, hash, title, crawled_at) values (%s, %s, %s, %s)
                       on conflict (url) do update set hash = excluded.hash, title = excluded.title,
                       crawled_at = excluded.crawled_at""",
                    (page["url"], digest, title, page["metadata"]["scrapedAt"]),
                )
                await db.execute("delete from kb_chunks where url = %s", (page["url"],))
                for i, c in enumerate(chunks):
                    await db.execute(
                        "insert into kb_chunks (id, url, heading, content, embedding) values (%s, %s, %s, %s, %s)",
                        (f"{page['url']}#{i}", page["url"], c["heading"], c["text"], json.dumps(embedded.data[i].embedding)),
                    )
            updated += 1

        # Pages the site no longer serves, removed only after a crawl that finished below its limit.
        removed = 0
        job = crawl.job or {}
        if job.get("status") == "completed" and job.get("completed", LIMIT) < LIMIT:
            gone = [url for url in known if url not in seen]
            removed = (await db.execute("delete from kb_pages where url = any(%s)", (gone,))).rowcount

        print(f"crawl {job.get('id')}: {len(seen)} pages read, {updated} re-embedded, {removed} removed")


asyncio.run(main())

Run it nightly from any scheduler, for example 0 3 * * * cd /srv/kb && npx tsx kb.ts (or python kb.py) in cron. The first run embeds everything; later runs embed only the difference.

What you get back

The console, on a night when the docs team shipped two edits and retired a page:

crawl crw_01K5ZT4N8Q: 412 pages read, 2 re-embedded, 1 removed

One row of kb_chunks:

{
  "id": "https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#14",
  "url": "https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows",
  "heading": "Events that trigger workflows > schedule",
  "content": "## schedule\n\nThe schedule event allows you to trigger a workflow at a scheduled time. You can schedule a workflow to run at specific UTC times using POSIX cron syntax...",
  "embedding": "[0.0132, -0.0417, 0.0088, ...]"
}

Retrieval is one query: select url, heading, content from kb_chunks order by embedding <=> $1 limit 8, and every chunk carries the URL your assistant cites.

What it costs

A crawl is 1 credit per page. A 400-page docs section is 400 credits per crawl, whether or not the pages changed, because a page has to be read to know.

Schedule	Crawls a month	Credits a month	Fits
Nightly	30	about 12,000	Pro (50,000)
Twice a week	9	about 3,600	Builder (10,000)
Weekly	4 to 5	about 1,800	Builder (10,000)

The saving is on your side: only changed pages are embedded and written, so a quiet night costs your embedding provider almost nothing.

Make it yours

Put a monitor on the docs changelog. It calls a webhook endpoint of yours when the page changes, and your endpoint starts this crawl the moment something ships instead of waiting for night.
Crawl several sections or sites in one run by looping over [root, includePaths] pairs; the stored hashes already work for any URL.
Store page.metadata.language too and filter retrieval by the user’s language.
Replace pgvector with your own vector store: the upsert is the only part that changes.

Crawl

Limits, paths and streaming.

Crawl jobs

Recover a crawl’s pages if the stream drops.

Monitors

A webhook when a page changes.

Pricing

Every rate on this page.

Agent Examples

Scale Examples

Scheduled Examples

Build Examples

Keep a docs knowledge base in sync

What you need

How it works

The program

What you get back

What it costs

Make it yours

Crawl

Crawl jobs

Monitors

Pricing

​What you need

​How it works

​The program

​What you get back

​What it costs

​Make it yours

​Related

Crawl

Crawl jobs

Monitors

Pricing

What you need

How it works

The program

What you get back

What it costs

Make it yours

Related