Job lifecycle - Outcrawl

A crawl’s id, from the first frame of its stream, is what every call on this page needs: crawl.id in TypeScript, crawl.job["id"] in Python once the first frame has arrived. Store it before you need it, especially if the connection that started the crawl might drop.

Status

import { Outcrawl } from '@outcrawl/sdk';

const oc = new Outcrawl();
const job = await oc.crawl.get('<id>');
console.log(job.status, job.discovered, job.completed, job.credits, job.failures);

import asyncio
from outcrawl import Outcrawl

async def main():
    async with Outcrawl() as oc:
        job = await oc.crawl.get("<id>")
        print(job["status"], job["discovered"], job["completed"], job["credits"], job["failures"])

asyncio.run(main())

curl https://api.outcrawl.ai/v1/crawl/<id> \
  -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"

Status, pages discovered, pages completed, spend so far, and the per-URL failures. Safe to poll, it is a row read and not a browser.

Results

import { Outcrawl } from '@outcrawl/sdk';

const oc = new Outcrawl();
let cursor: string | undefined;
do {
  const batch = await oc.crawl.results('<id>', { cursor });
  for (const record of batch) {
    console.log(record.page.url, record.page.markdown);
  }
  cursor = batch.cursor;
} while (cursor !== undefined);

import asyncio
from outcrawl import Outcrawl

async def main():
    async with Outcrawl() as oc:
        cursor = None
        while True:
            batch = await oc.crawl.results("<id>", cursor=cursor)
            for record in batch:
                print(record["page"]["url"], record["page"].get("markdown"))
            cursor = batch.cursor
            if cursor is None:
                break

asyncio.run(main())

curl "https://api.outcrawl.ai/v1/crawl/<id>/results?cursor=<cursor>" \
  -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"

The pages a crawl has delivered, oldest first, paginated with a cursor. Pages land during the crawl, so this answers before the crawl has finished, and a cancelled crawl still delivers what it had already fetched.

Cancel

import { Outcrawl } from '@outcrawl/sdk';

const oc = new Outcrawl();
const job = await oc.crawl.cancel('<id>');
console.log(job.status, job.completed);

import asyncio
from outcrawl import Outcrawl

async def main():
    async with Outcrawl() as oc:
        job = await oc.crawl.cancel("<id>")
        print(job["status"], job["completed"])

asyncio.run(main())

curl -X POST https://api.outcrawl.ai/v1/crawl/<id>/cancel \
  -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"

Stops a crawl. It settles with the pages already reached. Pages already fetched stay billed, the limit you set on submit is the spending protection, not a refund argued afterward.

If the connection that started a streaming crawl drops, the crawl itself is cancelled. These three calls are how you read the pages you already paid for afterward.