# API keys
Source: https://outcrawl.ai/docs/account/api-keys
One of two bearer credentials: a long-lived key for your own applications.
An API key is a long-lived credential for your own applications and scripts. It is not the only bearer credential the API accepts, an OAuth access token from signing in also works, see [API authentication](/docs/api/authentication) for both and when to use which.
```
ock_YOUR_KEY_ID.YOUR_SECRET
```
Generate and revoke keys from **[https://outcrawl.ai/console/keys](https://outcrawl.ai/console/keys)**. A new key is shown once, at creation, copy it immediately.
Set the key once in the environment:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
export OUTCRAWL_API_KEY=ock_YOUR_KEY_ID.YOUR_SECRET
```
Then call the API with it:
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY
const doc = await oc.scrape('https://example.com');
console.log(doc.markdown);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc: # reads OUTCRAWL_API_KEY
doc = await oc.scrape("https://example.com")
print(doc.markdown)
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/scrape \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
Both official SDKs read `OUTCRAWL_API_KEY` from the environment. The CLI reads it too, and it takes priority over a stored `outcrawl login` session, which is what makes it the right choice for CI and any headless environment where nobody can click through a sign-in page.
# Authentication
Source: https://outcrawl.ai/docs/account/authentication
Sign in to the console and manage account access.
This page covers signing in to the console at **[https://outcrawl.ai/console](https://outcrawl.ai/console)**. For authenticating REST API and MCP requests, see [API authentication](/docs/api/authentication), a different credential for a different purpose.
## Sign-in methods
One click, no password to set.
One click, no password to set.
A 6-digit code sent to your inbox.
A link in your inbox that opens a confirm page; only the click signs you in.
The magic link is deliberately not a one-step sign-in: it opens a confirm page rather than authenticating on load, so a mail scanner opening the link on your behalf cannot use it up before you do. The 6-digit code is the same underlying check without that extra click.
These four methods are also how you sign in when the CLI's `outcrawl login` or an MCP client sends you to a browser to approve a code: if you are not already signed in, you pick one of the four first, then approve.
## Account access
The console is where an account's settings live. See [settings](/docs/account/settings) for workspace-level configuration, and [API keys](/docs/account/api-keys) for the credential your own applications and scripts use.
# Integrations
Source: https://outcrawl.ai/docs/account/integrations
Connect an external MCP server so a run can call its tools mid-task.
Connect an MCP server, Gmail, Slack, and similar HTTPS endpoints, so an agent run can call its tools mid-task. Connecting one gives an agent reach outside the browser, so this step is human-only: it has no MCP tool. A run only reaches an integration it is explicitly given at submit time; connecting a server to the workspace is not the same as giving a specific run access to it.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const integration = await oc.integrations.connect({
name: 'slack',
transport: { kind: 'http', url: 'https://mcp.slack.example' },
auth: { kind: 'bearer', secret: 'slack_token' },
tools: null, // every tool the server offers
});
const integrations = await oc.integrations.list();
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
integration = await oc.integrations.connect(
name="slack",
transport={"kind": "http", "url": "https://mcp.slack.example"},
auth={"kind": "bearer", "secret": "slack_token"},
tools=None, # every tool the server offers
)
integrations = await oc.integrations.list()
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/integrations \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"name": "slack", "transport": {"kind": "http", "url": "https://mcp.slack.example"}, "auth": {"kind": "bearer", "secret": "slack_token"}, "tools": null}'
curl https://api.outcrawl.ai/v1/integrations -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
`transport.url` must be a public HTTPS address. `auth.kind` is `bearer`, `header` (with a `header` name) or `none`, and `tools` is a list of tool names, or `null` for every tool the server offers.
`auth.secret` is a name from your [secrets](/docs/account/secrets), never a raw token. `integrations.list` is the blast radius: every connected server, its tools, and the name it authenticates with, everything an agent could reach outside the browser if a run named it. `integrations.delete` disconnects a server; the underlying credential is untouched, it lives in secrets and is deleted there separately.
# Profiles
Source: https://outcrawl.ai/docs/account/profiles
Saved logins and a pinned exit, kept across sessions.
A profile keeps a pinned exit and persistent storage, cookies and logins that survive across sessions. Pass one to a browser session or an agent run to keep a login rather than signing in fresh every time.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const profile = await oc.profiles.create({ label: 'acme-user-42' });
const profiles = await oc.profiles.list();
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
profile = await oc.profiles.create(label="acme-user-42")
profiles = await oc.profiles.list()
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/profiles \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"label": "acme-user-42"}'
curl https://api.outcrawl.ai/v1/profiles -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
## Rules
* **Leased while open.** A second concurrent use of the same profile is refused rather than silently forking its cookies.
* **Delete is irreversible.** It removes the profile's stored cookies and site data permanently.
* Filter `profiles.list` by a label glob to find one quickly across a large account.
A profile is what keeps a browser session's login intact.
# Rules
Source: https://outcrawl.ai/docs/account/rules
Plain-language restrictions on what an agent may do, classified as enforced or guidance.
Rules are plain-language sentences a workspace sets to restrict what an agent may do. Setting one returns its classification on the same call: **enforced**, a gate that blocks the action, or **guidance**, only present in the prompt. You always know which kind you actually have, rather than assuming.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const result = await oc.rules.set([
'Never submit a payment form without a human taking control first.',
]);
for (const rule of result.rules) {
console.log(rule.class, rule.unenforceable); // enforced or guidance, and why not enforced
}
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
result = await oc.rules.set([
"Never submit a payment form without a human taking control first.",
])
for rule in result.rules:
print(rule["class"], rule["unenforceable"]) # enforced or guidance, and why not enforced
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X PUT https://api.outcrawl.ai/v1/rules \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"rules": ["Never submit a payment form without a human taking control first."]}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const current = await oc.rules.get();
console.log(current.enforced, current.guidance); // how many of each
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
current = await oc.rules.get()
print(current.enforced, current.guidance) # how many of each
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/rules -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Read them back with `rules.get`, and again the class is the answer, not the text: a rule you thought was a hard gate might come back as guidance, with the reason it could not be compiled into one.
Unlike secrets, rules **does** carry an MCP tool. The worst an assistant can do with this surface is narrow its own permissions, the opposite risk a credential poses, so an assistant reading `rules.get` can also tell a person "you think that is enforced and it is not", which is the whole point of the split.
# Secrets
Source: https://outcrawl.ai/docs/account/secrets
Store a credential under a name an agent can reference but never read.
Store a credential, a card, an environment variable, under a name a run can reference. No surface, API, SDK, CLI or console, ever reads a stored value back. A run reaches it only by naming it; the substitution happens below the model, which never sees the raw value.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const secret = await oc.secrets.create({ handle: 'card_amex', kind: 'card', value: '...' });
const secrets = await oc.secrets.list(); // names and a card's last four, never a value
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
secret = await oc.secrets.create(handle="card_amex", kind="card", value="...")
secrets = await oc.secrets.list() # names and a card's last four, never a value
asyncio.run(main())
```
```bash CLI theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl secrets create card_amex --kind card
# Prompts for the value, hidden, when this runs in a terminal.
echo -n "$CARD_NUMBER" | outcrawl secrets create card_amex --kind card
# Reads the value from stdin instead, when the command is piped.
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/secrets \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"handle": "card_amex", "kind": "card", "value": "..."}'
curl https://api.outcrawl.ai/v1/secrets -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
This has no MCP tool, deliberately. An assistant must never manage a credential itself; a model that could create, list or delete one is one convincing page away from moving somebody's money. A secret is created from the CLI or an SDK, straight to the platform; the console at [https://outcrawl.ai/console/secrets](https://outcrawl.ai/console/secrets) lists what is stored and revokes it, and neither surface ever holds a value. An agent still uses one, at runtime, by naming it, the one it was given at submit time.
`secrets.list` returns names and, for a card, its last four digits, and nothing else. `secrets.delete` takes effect the next time a run would type the value, not at the end of the run: a run that was given a deleted secret finds it gone and stops, naming it.
# Settings
Source: https://outcrawl.ai/docs/account/settings
General, Organization, Members and Danger zone: what each workspace setting does.
Workspace settings live at **[https://outcrawl.ai/console/settings](https://outcrawl.ai/console/settings)**, in four sections. Each is its own page; the tabs at the top move between them.
## General
Name and logo. The logo is resized to 128px in your browser before it saves, and shows next to the workspace's name in the switcher.
## Organization
Who this workspace bills as. A workspace starts as a **personal workspace**, invoices name you. **Switch to organization** to bill a company instead: a legal name, a billing address and a tax ID, all attached to this workspace's own Stripe customer. Visible and editable to the workspace's owner and admins only.
## Members
Who is on this workspace, and what they can do. See [key concepts](/docs/getting-started/concepts#workspaces-and-organizations) for the three roles. Invite by email, change a role, or remove someone: removing a member revokes their key for this workspace and any agent or app they had signed in with it, immediately.
## Danger zone
Two actions that cannot be undone, both available to the workspace's owner only.
### Transfer ownership
Pick another member and confirm once. They become the owner; you become an admin, in the same instant, on the same workspace. Ownership can only move to someone already on the workspace, this does not invite anyone.
### Delete
Deletes the workspace, its members, its invitations and its plan. Before you can, the console shows exactly what is lost: every member who loses access, every key that stops working, every pending invitation withdrawn, the plan, and the credit balance forfeited. Deletion is refused while either is true:
* **this is your only workspace.** A replacement workspace would not start with the Free plan's credits (see [key concepts](/docs/getting-started/concepts#workspaces-and-organizations)), so there is nowhere for the account to land;
* **a paid plan is active and not already set to cancel.** Cancel it from Billing first, or wait until it is scheduled to end.
Once neither blocks it, type the workspace's name to confirm. Deletion happens in order: every member's key and sign-in for this workspace is revoked first, a plan already winding down is stopped immediately with no refund, then the workspace's rows are removed, and anyone who was on it lands on whichever of their other workspaces they opened first.
# Downloads
Source: https://outcrawl.ai/docs/agents/downloads
Get every file a run or a browser session downloaded.
Every file the browser downloads is kept for you: in an agent run, and in a browser you drive yourself with `oc.browser()`. An invoice PDF, a CSV export, a statement: list the run's downloads and fetch the bytes.
* **What is kept:** every file the browser downloads, up to 200 MB each.
* **How long:** 7 days from when it was downloaded. `expiresAt` says when.
* **What it costs:** nothing extra. Keeping, listing and fetching files is free.
* **The link:** each file comes with a `url` that works for one hour and needs no API key. Anyone who has the link can fetch the file until it expires, so treat it like a password: share it only with whoever should get the file. List again for a fresh one.
A file shows up as soon as its download finishes. A run waits for downloads still in progress before it ends, so a list taken right after the run ends has all of them.
## List a run's downloads
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { mkdir } from 'node:fs/promises';
import { Outcrawl, saveDownload } from '@outcrawl/sdk';
const oc = new Outcrawl();
const downloads = await oc.agent.downloads('');
await mkdir('./files', { recursive: true });
for (const download of downloads) {
const path = await saveDownload(download, './files');
console.log(download.name, download.bytes, download.sourceUrl, path);
}
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from pathlib import Path
from outcrawl import Outcrawl, save_download
async def main():
async with Outcrawl() as oc:
downloads = await oc.agent.downloads("")
Path("./files").mkdir(exist_ok=True)
for download in downloads:
path = await save_download(download, "./files")
print(download["name"], download["bytes"], download["sourceUrl"], path)
asyncio.run(main())
```
```bash CLI theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
mkdir -p files
outcrawl agent downloads --save ./files
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/agent//downloads \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
On a job you already hold, `job.downloads()` does the same in both SDKs. Without `--save`, the CLI prints the list and saves nothing.
Over MCP, an assistant calls `outcrawl_agent_downloads` for a run and `outcrawl_session_downloads` for a browser session.
## Save a file
`saveDownload(download, dir)` in TypeScript and `save_download(download, directory)` in Python write one file into a folder that already exists, and answer the path they wrote. The CLI's `--save` does the same for every file in the list.
* The file is named after `name`. Characters a file name cannot hold, control characters and text direction marks become `_`. Long names are cut to 230 bytes. A name Windows keeps for a device, such as `CON` or `NUL.txt`, gets `_` in front.
* An existing file is never overwritten. If the name is taken, the first 8 characters of the download's `id` go before the extension, for example `invoice-8f0c1a2b.pdf`.
* The bytes are written to disk as they arrive, so a 200 MB file is never held in memory.
* If fewer bytes arrive than `bytes` says, the partial file is deleted and the call fails. List again and retry.
## Downloads in a browser session
A browser you drive with `oc.browser()` keeps its downloads too. Call `browser.downloads()` while it is open, or `oc.sessions.downloads(sessionId)` at any time in the next 7 days. See [sessions](/docs/browsers/sessions#downloads) for an example.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const downloads = await oc.sessions.downloads('');
for (const download of downloads) console.log(download.name, download.url);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
downloads = await oc.sessions.downloads("")
for download in downloads:
print(download["name"], download["url"])
asyncio.run(main())
```
```bash CLI theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl sessions downloads --save ./files
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/sessions//downloads \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
A session from `oc.sessions.get(id)` in TypeScript also carries `session.downloads()`.
### Playwright's own download event
With `oc.browser()`, Playwright's own `download` event works as usual. `download.saveAs()`, `download.path()` and `download.createReadStream()` give you the file: each one waits until the file is kept, for up to 2 minutes after it finishes downloading, and then fetches it.
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const pw = page.raw as unknown as Page; // import type { Page } from 'playwright'
const downloading = pw.waitForEvent('download');
await page.click('#export-csv');
await (await downloading).saveAs('./export.csv');
```
`page.raw.on('download', ...)` works the same way, on popups too. If a file was not kept, for example because it is over 200 MB, `download.failure()` says why and `saveAs()` fails with the same reason.
Connecting Playwright or Puppeteer to the browser without the TypeScript SDK, from Python or any other language? The browser runs on Outcrawl's machines, so Playwright's `saveAs()` and `path()` look for the file on your disk and find nothing. List the session's downloads instead, and match the file by its `sourceUrl` and `name`.
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import save_download
async def save_kept(oc, session_id, download, directory="."):
"""download is Playwright's Download. session_id is the session's id, as oc.sessions.list() shows it."""
await download.failure() # returns once the browser has finished the download
for _ in range(60):
kept = await oc.sessions.downloads(session_id, limit=1000)
found = [d for d in kept if d["sourceUrl"] == download.url and d["name"] == download.suggested_filename]
if found:
newest = max(found, key=lambda d: d["createdAt"])
return await save_download(newest, directory)
await asyncio.sleep(2)
raise RuntimeError(f"{download.suggested_filename} is not in the session's downloads")
```
If the same URL was downloaded more than once, the newest `createdAt` is the latest copy.
## What a list holds
`GET /v1/agent/{id}/downloads` and `GET /v1/sessions/{id}/downloads` answer the same shape:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"data": [
{
"id": "8f0c1a2b-5d1e-4c33-9a70-2e6b1d0f4c11",
"name": "invoice-2026-10.pdf",
"contentType": "application/pdf",
"bytes": 48213,
"sourceUrl": "https://vendor.example/invoices/123.pdf",
"createdAt": "2026-10-19T10:00:00.000Z",
"expiresAt": "2026-10-26T10:00:00.000Z",
"url": "https://api.outcrawl.ai/dl/"
}
],
"usage": {}
}
```
| Field | Meaning |
| - | - |
| `id` | The download's id. It stays the same every time you list. |
| `name` | The file name the site gave it. |
| `contentType` | The file's type, such as `application/pdf`. |
| `bytes` | Its size. |
| `sourceUrl` | The URL it was downloaded from. |
| `createdAt` | When the download finished. |
| `expiresAt` | When it is deleted, 7 days after `createdAt`. |
| `url` | A link to the bytes. It works for one hour and needs no API key. |
`data` is empty when nothing was downloaded. A run or session that does not exist, or belongs to another account, answers `404 not_found`.
The list comes a page at a time. When there is another page, the answer carries a `cursor`: pass it back as `?cursor=` to read the next one. `?limit=` sets the page size, from 1 to 1000. In the SDKs the list is an array with a `cursor` beside it, so `oc.agent.downloads(id, { cursor: downloads.cursor })` in TypeScript and `oc.agent.downloads(run_id, cursor=downloads.cursor)` in Python read the next page.
## Fetch the bytes
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -OJ "https://api.outcrawl.ai/dl/"
```
Send the link without your API key. It answers the bytes as `application/octet-stream`, with `Content-Length` and a `Content-Disposition` that carries the file name, so `curl -OJ` saves it under that name. The file's real type is `contentType` in the list. The link is a bearer link: it works for whoever holds it, for one hour, even if your API key is revoked or the request comes from outside your key's allowed IPs. After an hour, or once the file has been deleted, it answers `404`. List again for a fresh link while the file is still kept.
Anyone holding a link can fetch the file for that hour. Treat it like the file itself.
## Related
Hand a file to a run so it can upload it.
Status, events and results of a run.
# Files
Source: https://outcrawl.ai/docs/agents/files
Attach a file to a run.
Hand a file to a run so it can attach it to a page, at submit time, mid-run, or before its first step. The call answers a file id the agent uses to attach it; you never send the bytes twice and the model never sees them directly. The file is destroyed when the run ends, on every ending.
To get a file the run downloaded, such as an invoice PDF, use [downloads](/docs/agents/downloads). Use [`agent.results`](/docs/agents/monitoring) for the data a run produced.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { readFile } from 'node:fs/promises';
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const bytes = await readFile('./resume.pdf');
const file = await oc.agent.job('').addFile({ name: 'resume.pdf', bytes, contentType: 'application/pdf' });
console.log(file.id);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from pathlib import Path
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
content = Path("./resume.pdf").read_bytes()
file = await oc.agent.job("").add_file(name="resume.pdf", content=content, content_type="application/pdf")
print(file["id"])
asyncio.run(main())
```
```bash CLI theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl agent files add ./resume.pdf
```
```json MCP (outcrawl_agent_file_add) theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"runId": "",
"name": "resume.pdf",
"content": "",
"contentType": "application/pdf"
}
```
MCP carries the file's bytes as base64 directly in the tool call, it has no other way to move bytes. The CLI takes a file path and reads it for you; the TypeScript SDK takes a raw `Uint8Array` and the Python SDK takes `bytes`, and both encode it for you. All four reach the same endpoint, `POST https://api.outcrawl.ai/v1/agent//files`.
# Human in the loop
Source: https://outcrawl.ai/docs/agents/human-in-the-loop
Take the wheel of a live run, and hand it back.
Nothing sends a running agent new instructions. A person acts on a live run in two ways: answering a question the run asked, and control, taking the browser and handing it back.
## Control
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const run = oc.agent.job('');
let state = await run.control('take'); // the run holds at its next step boundary
while (state.pause?.reason !== 'control') {
await new Promise((wake) => setTimeout(wake, 2000));
state = await run.status();
}
console.log(state.pause.viewerUrl); // open it, press take control in the player
// ... do the step yourself ...
await run.control('release'); // the agent resumes on whatever page you left it on
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
run = oc.agent.job("")
state = await run.control("take") # the run holds at its next step boundary
while (state.get("pause") or {}).get("reason") != "control":
await asyncio.sleep(2)
state = await run.status()
print(state["pause"].get("viewerUrl")) # open it, press take control in the player
# ... do the step yourself ...
await run.control("release") # the agent resumes on whatever page you left it on
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/agent//control \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"action": "take"}'
```
Asks for the wheel of a live run. The run finishes the step it is on, then holds with the browser still open, and its status becomes `paused`. `pause.viewerUrl` on the run is the player for that session: open it and press **take control** there to drive the page yourself. It is absent when the run was submitted with `record: false`. Hand the wheel back with `control('release')` and the agent resumes from whatever page you left it on, not from where it was.
While you hold the wheel, whether after `control('take')` or by driving in the live viewer, the run bills browser time (1 credit a minute) and proxy usage only, with no model usage, until you hand it back. See [pricing](/docs/billing/pricing).
Use it for a step you would rather do yourself, a login you will not automate, a form only you should complete, not to stop a run (that is [`agent.cancel`](/docs/agents/overview#cancel)).
## Answer
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
await oc.agent.answer('', 'the blue one');
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
await oc.agent.answer("", "the blue one")
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/agent//answer \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"answer": "the blue one"}'
```
When a run needs something only you know, it parks: its status becomes `paused` with `pause.reason` set to `question`, `pause.question` holding what it asked, `pause.options` the replies it suggests (any answer is accepted) and `pause.expiresAt` when it stops waiting. Answer it and the run carries on from where it stopped. A question nobody answers by `pause.expiresAt` is answered "nobody replied" and the run decides what to do without it.
A run that can pay may ask two structured questions, in `pause.ask`. An `approval` question (a purchase above your workspace's approval amount) is answered with `{"approve": true}` or `{"approve": false}`; one nobody answers in time fails that purchase with `approval_timed_out` and nothing is bought. A `secret` question (a card, a billing profile or a security code) is never answered here: send the value as a run secret named `pause.ask.for`, and an answer carrying it is refused with `secret_in_answer`.
Control is the only way to redirect what a live run does. Nothing sends a running agent new instructions mid-task, cancel and resubmit for that.
# Monitoring a run
Source: https://outcrawl.ai/docs/agents/monitoring
Status, the event log, results, and agent-run webhooks.
## Status
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const run = await oc.agent.get('');
console.log(run.status, run.goals, run.credits, run.data);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
run = await oc.agent.get("")
print(run["status"], run.get("goals"), run["credits"], run.get("data"))
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/agent/ \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Where a run has got to: status, the plan it stated, spend so far, and its answer once it has one. Safe to poll often, it is a row read, not a browser. A job you submitted in the same process has the same call as `job.status()`.
## Events
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const events = oc.agent.events('', { after: 0, level: 'step' });
for await (const record of events) {
console.log(record.seq, record.kind, record.value);
}
console.log(events.cursor); // pass back as `after` to resume from here
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
events = oc.agent.events("", after=0, level="step")
async for record in events:
print(record["seq"], record["kind"], record["value"])
print(events.cursor) # pass back as `after` to resume from here
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl "https://api.outcrawl.ai/v1/agent//events?after=&level=step" \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Everything a run has done, in order: steps, findings, dead ends, plan changes. Ask for the verbosity you want with `level`: `status`, `step` (the default) or `debug`. Resume from the `after` cursor you last saw rather than re-reading from the start. Over HTTP this streams as newline-delimited JSON; the SDKs page through it for you and keep going until the run ends.
## Results
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
let cursor: string | undefined;
do {
const page = await oc.agent.results('', { cursor });
for (const record of page) {
console.log(record.kind, record.value);
}
cursor = page.cursor;
} while (cursor !== undefined);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
cursor = None
while True:
page = await oc.agent.results("", cursor=cursor)
for record in page:
print(record["kind"], record["value"])
cursor = page.cursor
if cursor is None:
break
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl "https://api.outcrawl.ai/v1/agent//results?cursor=" \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
The records a run has appended, oldest first. Records land during the run, so this answers before it finishes, and a cancelled run still delivers what it had already produced.
## Webhooks
A run can notify a bare webhook URL when it settles, and unlike a monitor, this one retries.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const job = oc.agent({
task: 'Report the title of the top story on Hacker News.',
caps: { budget: '16.00' },
webhook: 'https://yourapp.example/hooks/outcrawl',
});
console.log(await job.id());
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
job = oc.agent(
task="Report the title of the top story on Hacker News.",
caps={"budget": "16.00"},
webhook="https://yourapp.example/hooks/outcrawl",
)
print(await job.id())
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/agent \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"task": "Report the title of the top story on Hacker News.",
"caps": { "budget": "16.00" },
"webhook": "https://yourapp.example/hooks/outcrawl"
}'
```
```text Signature header theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
x-outcrawl-signature: t=1758000000,v1=
```
Verify the signature before trusting a payload: the header carries a Unix timestamp and an HMAC, in the same `t=...,v1=...` shape Stripe's webhooks use.
An agent run can only notify a **bare webhook URL**, never Discord or Slack directly, those two are monitor-only targets. In exchange, a run's webhook **does retry**: up to 8 attempts, backing off 1 minute, 3 minutes, 9 minutes, 27 minutes, then three more at 1 hour each, about 3 hours 40 minutes in total, before it gives up after the eighth refusal. A monitor's notification, by contrast, is sent once. Either way, a run's own events stay readable through `agent.events` and `agent.results` regardless of whether the webhook ever lands.
# Overview
Source: https://outcrawl.ai/docs/agents/overview
Hand over a natural-language task, it finishes in a real browser.
Submit a natural-language browser task and get an id back in milliseconds. The run is not held open on that call, hours are normal: read status separately, read results separately, cancel separately. In the SDKs those are `oc.agent.get(id)`, `oc.agent.results(id)` and `oc.agent.cancel(id)`, with the same names in Python. Use agents when a page needs decisions or interaction, not for plain reading, where [scrape](/docs/scrape/overview) is far cheaper.
An agent run is billed for the AI model's usage, browser time (1 credit per minute the run is open, paused time included) and proxy usage (2,000 credits per GB), in credits. A typical run costs **about 15 credits per minute** all in; that is an estimate, and the actual cost depends on the task. Driving a browser yourself with the SDK, without an agent, pays only browser time and proxy usage. See [pricing](/docs/billing/pricing).
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const job = oc.agent({
task: 'Report the title of the top story on Hacker News.',
caps: { budget: '16.00' },
});
console.log(await job.id()); // store this to reach the run later
const run = await job;
console.log(run.status, run.data);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
job = oc.agent(task="Report the title of the top story on Hacker News.", caps={"budget": "16.00"})
print(await job.id()) # store this to reach the run later
run = await job
print(run["status"], run.get("data"))
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/agent \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"task": "Report the title of the top story on Hacker News.",
"caps": { "budget": "16.00" }
}'
```
`caps.budget` is required, a hard ceiling on what the run is billed. A budget at or below 1 credit is refused before the run starts: a run spends model usage, browser time and proxy usage from its first step, so it would stop before reading anything. `caps.steps` and `caps.duration` are optional additional ceilings. This example task cost 5.0–7.7 credits on 2026-10-03 (n=33), so its 16.00 ceiling is twice the most it took; size yours from what your own tasks spend.
## Typed results
Give a run a JSON Schema and its answer comes back shaped to it instead of free text:
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const typed = await oc.agent({
task: 'Report the title of the top story on Hacker News.',
caps: { steps: 20, budget: '16.00', duration: '5m' },
schema: { type: 'object', properties: { title: { type: 'string' } }, required: ['title'] },
});
if (typed.status === 'completed') console.log(typed.data); // { title: '...' }
else console.log(typed.status, typed.stopReason); // capped, partial or failed: data may be absent
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
typed = await oc.agent(
task="Report the title of the top story on Hacker News.",
caps={"steps": 20, "budget": "16.00", "duration": "5m"},
schema={"type": "object", "properties": {"title": {"type": "string"}}, "required": ["title"]},
)
if typed["status"] == "completed":
print(typed["data"]) # {'title': '...'}
else:
print(typed["status"], typed.get("stopReason")) # capped, partial or failed: data may be absent
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/agent \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"task": "Report the title of the top story on Hacker News.",
"caps": { "steps": 20, "budget": "16.00", "duration": "5m" },
"schema": { "type": "object", "properties": { "title": { "type": "string" } }, "required": ["title"] }
}'
```
## Cancel
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const run = await oc.agent.cancel('');
console.log(run.status);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
run = await oc.agent.cancel("")
print(run["status"])
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/agent//cancel \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Stops a run and answers it as it stands, so cancelling a run that already finished is not an error. Work already done stays billed: `caps.budget` is the spending limit, cancel is the stop button.
Status, the event log, results, and webhooks.
Take the wheel of a live run, and hand it back.
Attach a file to a run.
Get every file a run downloaded.
# Answer the question a paused run is waiting on, and it carries on from where it stopped.
Source: https://outcrawl.ai/docs/api-reference/agent/answer-the-question-a-paused-run-is-waiting-on-and-it-carries-on-from-where-it-stopped
/api-reference/openapi.json post /v1/agent/{id}/answer
Answer the question a paused run is waiting on, and it carries on from where it stopped. A run parks when it needs something only you know (which of two options, a detail you did not give it) and reports the question on its status and its event stream. Use it to unblock a run whose status is paused; not to send a run new instructions, which no route does, and not to take over the browser yourself, which is outcrawl_agent_control.
# Every file the run's browser downloaded, kept 7 days, up to 200 MB each: name, content type, size, the URL it came fr...
Source: https://outcrawl.ai/docs/api-reference/agent/every-file-the-runs-browser-downloaded-kept-7-days-up-to-200-mb-each:-name-content-type-size-the-url-it-came-fr
/api-reference/openapi.json get /v1/agent/{id}/downloads
Every file the run's browser downloaded, kept 7 days, up to 200 MB each: name, content type, size, the URL it came from, and a link to the bytes that works for an hour with no API key. Use it to get a file the run downloaded, such as an invoice PDF or a CSV export; not for the run's extracted data, which is outcrawl_agent_results.
# Everything a run has done, in order, from a cursor: steps, findings, dead ends, plan changes.
Source: https://outcrawl.ai/docs/api-reference/agent/everything-a-run-has-done-in-order-from-a-cursor:-steps-findings-dead-ends-plan-changes
/api-reference/openapi.json get /v1/agent/{id}/events
Everything a run has done, in order, from a cursor: steps, findings, dead ends, plan changes. Ask for the verbosity you want with `level`, and resume with the `after` sequence you last saw rather than re-reading from the start. Use it to watch or audit a run; not to get its answer, which is outcrawl_agent_status.
# Get a single-use upload URL and token (at most five minutes, dead when the run ends) that let a browser add ONE tempo...
Source: https://outcrawl.ai/docs/api-reference/agent/get-a-single-use-upload-url-and-token-at-most-five-minutes-dead-when-the-run-ends-that-let-a-browser-add-one-tempo
/api-reference/openapi.json post /v1/agent/{id}/secret-intents
Get a single-use upload URL and token (at most five minutes, dead when the run ends) that let a browser add ONE temporary secret to ONE live run without the value passing through your server: a security code, a whole card, a billing profile or a text value. Name your checkout page’s origin when you mint it to send from your own page; without one, only the Outcrawl console may send it. Not for a value your server holds, which is agent.secrets.add.
# Give a live run one temporary secret, held only in the memory of the machine running it and dropped when the run ends...
Source: https://outcrawl.ai/docs/api-reference/agent/give-a-live-run-one-temporary-secret-held-only-in-the-memory-of-the-machine-running-it-and-dropped-when-the-run-ends
/api-reference/openapi.json post /v1/agent/{id}/secrets
Give a live run one temporary secret, held only in the memory of the machine running it and dropped when the run ends: a text value, a whole card (which may carry its security code), a billing profile (kind details), or one card field such as a stored card’s security code (kind card_field). Answers 204. Use it to answer a run asking for a security code, or to hand a running checkout a card or billing details for this run only; not to store a credential, which is secrets.create.
# Hand a file to a run so it can attach it to a page: at submit time, mid-run, or before its first step.
Source: https://outcrawl.ai/docs/api-reference/agent/hand-a-file-to-a-run-so-it-can-attach-it-to-a-page:-at-submit-time-mid-run-or-before-its-first-step
/api-reference/openapi.json post /v1/agent/{id}/files
Hand a file to a run so it can attach it to a page: at submit time, mid-run, or before its first step. Answers a HANDLE the agent uses with attachFile; you never send the bytes twice and the model never sees them. The file is destroyed when the run ends, on every ending. Use it for something the task needs to upload; not to fetch a file the run downloaded, which is outcrawl_agent_downloads.
# List the purchases one run opened: merchant, amount, the card by handle and last four, and how each ended (completed,...
Source: https://outcrawl.ai/docs/api-reference/agent/list-the-purchases-one-run-opened:-merchant-amount-the-card-by-handle-and-last-four-and-how-each-ended-completed
/api-reference/openapi.json get /v1/agent/{id}/purchases
List the purchases one run opened: merchant, amount, the card by handle and last four, and how each ended (completed, declined, abandoned, unknown, or still reserved). Use to reconcile a run's checkout; not for the run's credits, which are on agent.get.
# Stop a run.
Source: https://outcrawl.ai/docs/api-reference/agent/stop-a-run
/api-reference/openapi.json post /v1/agent/{id}/cancel
Stop a run. It settles with whatever it had reached and everything it already appended stays yours; work already performed is still billed, because caps are the protection and not a refund argued afterwards. Use it to stop spending; not to delete the output.
# Submit a natural-language browser task as a JOB and get an id back in milliseconds.
Source: https://outcrawl.ai/docs/api-reference/agent/submit-a-natural-language-browser-task-as-a-job-and-get-an-id-back-in-milliseconds
/api-reference/openapi.json post /v1/agent
Submit a natural-language browser task as a JOB and get an id back in milliseconds. The run is not held open on this call and hours are normal: read outcrawl_agent_status for the answer, outcrawl_agent_results for the records it appended, and outcrawl_agent_cancel to stop it. Use when the page needs decisions or interaction; not for plain reading, where scrape is far cheaper.
# Take the wheel of a live run, or hand it back.
Source: https://outcrawl.ai/docs/api-reference/agent/take-the-wheel-of-a-live-run-or-hand-it-back
/api-reference/openapi.json post /v1/agent/{id}/control
Take the wheel of a live run, or hand it back. Taking it PAUSES the agent where it stands with the browser still open, and answers the URL a person drives it at; handing it back resumes the same run on whatever page you left it on. Use it when a run needs a human for one step (a login you will not automate, a form only you can fill); not to stop a run, which is outcrawl_agent_cancel, and not to answer a question the agent asked, which is outcrawl_agent_answer.
# The records a run appended, a page at a time, oldest first.
Source: https://outcrawl.ai/docs/api-reference/agent/the-records-a-run-appended-a-page-at-a-time-oldest-first
/api-reference/openapi.json get /v1/agent/{id}/results
The records a run appended, a page at a time, oldest first. Records land DURING the run, so this answers before the run has finished and a killed run still delivers what it bought. Use it for the data; not for whether the run is done, which is outcrawl_agent_status.
# Where one run has got to: status, the plan it stated, what it has spent, and its answer once it has one.
Source: https://outcrawl.ai/docs/api-reference/agent/where-one-run-has-got-to:-status-the-plan-it-stated-what-it-has-spent-and-its-answer-once-it-has-one
/api-reference/openapi.json get /v1/agent/{id}
Where one run has got to: status, the plan it stated, what it has spent, and its answer once it has one. Safe to call as often as you like: it is a row read, not a browser. Use it to find out whether a submitted run has finished; not to fetch the records it produced, which is outcrawl_agent_results.
# Your agent runs, newest first, by status and age: id, task, status, steps, records, credits and replay URL, without t...
Source: https://outcrawl.ai/docs/api-reference/agent/your-agent-runs-newest-first-by-status-and-age:-id-task-status-steps-records-credits-and-replay-url-without-t
/api-reference/openapi.json get /v1/agent
Your agent runs, newest first, by status and age: id, task, status, steps, records, credits and replay URL, without the answer. Use to find a run you did not keep the id of; not to read one run's answer, which is outcrawl_agent_status.
# Stop a crawl.
Source: https://outcrawl.ai/docs/api-reference/crawl/stop-a-crawl
/api-reference/openapi.json post /v1/crawl/{id}/cancel
Stop a crawl. It settles with the pages it had reached and everything already delivered stays yours; pages already fetched are still billed, because the `limit` is the protection and not a refund argued afterwards. Use it to stop spending; not to delete the pages.
# The pages a crawl has delivered, a page at a time, oldest first.
Source: https://outcrawl.ai/docs/api-reference/crawl/the-pages-a-crawl-has-delivered-a-page-at-a-time-oldest-first
/api-reference/openapi.json get /v1/crawl/{id}/results
The pages a crawl has delivered, a page at a time, oldest first. Pages land DURING the crawl, so this answers before it has finished, and a cancelled crawl still delivers what it bought. Resume with the `cursor` you last saw. Use it for the content; not for whether the crawl is done, which is /v1/crawl/{id}.
# Walk a site from a seed URL and stream every page as it is scraped.
Source: https://outcrawl.ai/docs/api-reference/crawl/walk-a-site-from-a-seed-url-and-stream-every-page-as-it-is-scraped
/api-reference/openapi.json post /v1/crawl
Walk a site from a seed URL and stream every page as it is scraped. Use when the set of URLs is unknown; not for one page, and not for content behind a login flow.
# Where one crawl has got to: status, pages discovered, pages completed, what it has spent, and the per-URL failures.
Source: https://outcrawl.ai/docs/api-reference/crawl/where-one-crawl-has-got-to:-status-pages-discovered-pages-completed-what-it-has-spent-and-the-per-url-failures
/api-reference/openapi.json get /v1/crawl/{id}
Where one crawl has got to: status, pages discovered, pages completed, what it has spent, and the per-URL failures. A row read and not a browser, so it is safe to poll. Use it to learn whether a crawl is still running, or what it reached after a connection dropped; not to fetch the pages, which is /v1/crawl/{id}/results.
# What this account has been granted, what it has spent against the grant, and what is left.
Source: https://outcrawl.ai/docs/api-reference/credits/what-this-account-has-been-granted-what-it-has-spent-against-the-grant-and-what-is-left
/api-reference/openapi.json get /v1/credits
What this account has been granted, what it has spent against the grant, and what is left. Use to see a promotional or student balance and when it lapses; not for the monthly plan allowance, and not for per-call cost, which every response already carries.
# Change a connected server's tool list or the secret handle it authenticates with, without disconnecting it.
Source: https://outcrawl.ai/docs/api-reference/integrations/change-a-connected-servers-tool-list-or-the-secret-handle-it-authenticates-with-without-disconnecting-it
/api-reference/openapi.json patch /v1/integrations/{id}
Change a connected server's tool list or the secret handle it authenticates with, without disconnecting it. A new secret is checked against the server first and refused if it does not work.
# Check an MCP server before connecting it: calls its tool list with the credential a secret handle names, and answers ...
Source: https://outcrawl.ai/docs/api-reference/integrations/check-an-mcp-server-before-connecting-it:-calls-its-tool-list-with-the-credential-a-secret-handle-names-and-answers-
/api-reference/openapi.json post /v1/integrations/test
Check an MCP server before connecting it: calls its tool list with the credential a secret handle names, and answers ok with the tools it offers, or not ok with the reason (not_mcp, unauthorized, unreachable, missing_secret) and what to do. Stores nothing and grants nothing.
# Connect an MCP server (Gmail, Slack, Twilio) so a run can call its tools mid-task.
Source: https://outcrawl.ai/docs/api-reference/integrations/connect-an-mcp-server-gmail-slack-twilio-so-a-run-can-call-its-tools-mid-task
/api-reference/openapi.json post /v1/integrations
Connect an MCP server (Gmail, Slack, Twilio) so a run can call its tools mid-task. HTTPS endpoints only, authenticated by a secret handle, and no run reaches it unless the run names it. Use to give an agent a way past a wall the browser cannot climb; not to store the credential itself, which is a secret and has no tool.
# Disconnect an MCP server.
Source: https://outcrawl.ai/docs/api-reference/integrations/disconnect-an-mcp-server
/api-reference/openapi.json delete /v1/integrations/{id}
Disconnect an MCP server. Takes effect on the next call: a run already holding a grant finds the connector gone and is told so. The credential is untouched: it lives in the secrets store and is deleted there.
# Everything an agent can reach outside the browser: each connected server, the tools it exposes, and the secret handle...
Source: https://outcrawl.ai/docs/api-reference/integrations/everything-an-agent-can-reach-outside-the-browser:-each-connected-server-the-tools-it-exposes-and-the-secret-handle
/api-reference/openapi.json get /v1/integrations
Everything an agent can reach outside the browser: each connected server, the tools it exposes, and the secret handle it authenticates with. This is the blast radius, and a customer who cannot see it cannot consent to it.
# Delete a monitor and stop its schedule.
Source: https://outcrawl.ai/docs/api-reference/monitors/delete-a-monitor-and-stop-its-schedule
/api-reference/openapi.json delete /v1/monitors/{id}
Delete a monitor and stop its schedule. Snapshots are discarded.
# List monitors with their last check status.
Source: https://outcrawl.ai/docs/api-reference/monitors/list-monitors-with-their-last-check-status
/api-reference/openapi.json get /v1/monitors
List monitors with their last check status.
# Watch a URL on a schedule and notify on real change via Discord, Slack or webhook.
Source: https://outcrawl.ai/docs/api-reference/monitors/watch-a-url-on-a-schedule-and-notify-on-real-change-via-discord-slack-or-webhook
/api-reference/openapi.json post /v1/monitors
Watch a URL on a schedule and notify on real change via Discord, Slack or webhook. Use for recurring checks; not for a one-off comparison.
# Create a profile with a pinned exit and persistent storage.
Source: https://outcrawl.ai/docs/api-reference/profiles/create-a-profile-with-a-pinned-exit-and-persistent-storage
/api-reference/openapi.json post /v1/profiles
Create a profile with a pinned exit and persistent storage. Use to keep a login across sessions.
# Delete a profile and its stored cookies and site data.
Source: https://outcrawl.ai/docs/api-reference/profiles/delete-a-profile-and-its-stored-cookies-and-site-data
/api-reference/openapi.json delete /v1/profiles/{id}
Delete a profile and its stored cookies and site data. Irreversible.
# Fetch one profile: its label, pinned exit and creation time.
Source: https://outcrawl.ai/docs/api-reference/profiles/fetch-one-profile:-its-label-pinned-exit-and-creation-time
/api-reference/openapi.json get /v1/profiles/{id}
Fetch one profile: its label, pinned exit and creation time.
# List profiles, optionally filtered by a label glob.
Source: https://outcrawl.ai/docs/api-reference/profiles/list-profiles-optionally-filtered-by-a-label-glob
/api-reference/openapi.json get /v1/profiles
List profiles, optionally filtered by a label glob.
# List every purchase your runs made, newest first, filtered by card, merchant, status or time.
Source: https://outcrawl.ai/docs/api-reference/purchases/list-every-purchase-your-runs-made-newest-first-filtered-by-card-merchant-status-or-time
/api-reference/openapi.json get /v1/purchases
List every purchase your runs made, newest first, filtered by card, merchant, status or time. Use to reconcile spending across runs; not one run's purchases, which is agent.purchases.
# Replace the workspace rules with these plain-language sentences and get the classification back on the same call.
Source: https://outcrawl.ai/docs/api-reference/rules/replace-the-workspace-rules-with-these-plain-language-sentences-and-get-the-classification-back-on-the-same-call
/api-reference/openapi.json put /v1/rules
Replace the workspace rules with these plain-language sentences and get the classification back on the same call. Use to forbid something in your own words; a rule that could not be compiled into a gate comes back as guidance with the reason.
# The workspace rules and, for each one, whether it is ENFORCED as a gate that blocks a dispatch or is only GUIDANCE in...
Source: https://outcrawl.ai/docs/api-reference/rules/the-workspace-rules-and-for-each-one-whether-it-is-enforced-as-a-gate-that-blocks-a-dispatch-or-is-only-guidance-in
/api-reference/openapi.json get /v1/rules
The workspace rules and, for each one, whether it is ENFORCED as a gate that blocks a dispatch or is only GUIDANCE in the prompt. Use it to check that a rule you wrote is actually enforced; the class is the answer, not the text.
# Fetch one URL and return clean content in the requested formats.
Source: https://outcrawl.ai/docs/api-reference/scrape/fetch-one-url-and-return-clean-content-in-the-requested-formats
/api-reference/openapi.json post /v1/scrape
Fetch one URL and return clean content in the requested formats. Use for a single known page; not for following links, and not for tasks that need decisions.
# Web search, optionally with every hit rendered through our scrape pipeline.
Source: https://outcrawl.ai/docs/api-reference/search/web-search-optionally-with-every-hit-rendered-through-our-scrape-pipeline
/api-reference/openapi.json post /v1/search
Web search, optionally with every hit rendered through our scrape pipeline. Use to find pages; not to read one you already have a URL for.
# Delete a credential and the record of which runs used it.
Source: https://outcrawl.ai/docs/api-reference/secrets/delete-a-credential-and-the-record-of-which-runs-used-it
/api-reference/openapi.json delete /v1/secrets/{id}
Delete a credential and the record of which runs used it. Takes effect on the next fill: a run holding a grant finds nothing to resolve.
# Get a single-use, five-minute upload URL and token that let a browser store ONE secret under ONE name without the val...
Source: https://outcrawl.ai/docs/api-reference/secrets/get-a-single-use-five-minute-upload-url-and-token-that-let-a-browser-store-one-secret-under-one-name-without-the-val
/api-reference/openapi.json post /v1/secrets/intents
Get a single-use, five-minute upload URL and token that let a browser store ONE secret under ONE name without the value passing through your server. Use when a person types a credential into your own web page; not to store a value you already hold, which is secrets.create.
# List stored credentials by handle, with the last four digits of a card and nothing else.
Source: https://outcrawl.ai/docs/api-reference/secrets/list-stored-credentials-by-handle-with-the-last-four-digits-of-a-card-and-nothing-else
/api-reference/openapi.json get /v1/secrets
List stored credentials by handle, with the last four digits of a card and nothing else. No route returns a value, and that absence is the point.
# Store a credential under a handle an agent can reference but never read.
Source: https://outcrawl.ai/docs/api-reference/secrets/store-a-credential-under-a-handle-an-agent-can-reference-but-never-read
/api-reference/openapi.json post /v1/secrets
Store a credential under a handle an agent can reference but never read. Use to save a card or an environment variable for a run to use; not to read one back, which nothing can do.
# Store up to 500 credentials in one call, with one result per item and never a value.
Source: https://outcrawl.ai/docs/api-reference/secrets/store-up-to-500-credentials-in-one-call-with-one-result-per-item-and-never-a-value
/api-reference/openapi.json post /v1/secrets/batch
Store up to 500 credentials in one call, with one result per item and never a value. Use to import a wallet of cards or payment details, retry-safe per handle (onConflict skip or replace); not for one secret, which is secrets.create.
# A session's network as a HAR 1.2 file that opens in a browser's developer tools, with stored bodies inline.
Source: https://outcrawl.ai/docs/api-reference/sessions/a-sessions-network-as-a-har-12-file-that-opens-in-a-browsers-developer-tools-with-stored-bodies-inline
/api-reference/openapi.json get /v1/sessions/{id}/har
A session's network as a HAR 1.2 file that opens in a browser's developer tools, with stored bodies inline. Use to hand traffic to a tool that reads HAR; not for reading in code, which is sessions.network.
# Choose who may watch a session besides your own API keys: `private` (your organization, the default), `link` (anyone ...
Source: https://outcrawl.ai/docs/api-reference/sessions/choose-who-may-watch-a-session-besides-your-own-api-keys:-`private`-your-organization-the-default-`link`-anyone-
/api-reference/openapi.json post /v1/sessions/{id}/share
Choose who may watch a session besides your own API keys: `private` (your organization, the default), `link` (anyone holding the returned share URL) or `people` (named Outcrawl users, by email). Setting `private` revokes sharing and rotates the link.
# Every event a session recorded, in order, as a stream: tabs, navigations, console, network phases, pointer, video chu...
Source: https://outcrawl.ai/docs/api-reference/sessions/every-event-a-session-recorded-in-order-as-a-stream:-tabs-navigations-console-network-phases-pointer-video-chu
/api-reference/openapi.json get /v1/sessions/{id}/events
Every event a session recorded, in order, as a stream: tabs, navigations, console, network phases, pointer, video chunks and agent marks, filtered by `kind` and `tab`, resumable with `after`, and kept open on a live session with `follow`. Use to replay or audit a session programmatically; not for a summary of what went wrong, which is the digest.
# Every file a browser session you drove over /connect downloaded, kept 7 days, up to 200 MB each: name, content type, ...
Source: https://outcrawl.ai/docs/api-reference/sessions/every-file-a-browser-session-you-drove-over-connect-downloaded-kept-7-days-up-to-200-mb-each:-name-content-type-
/api-reference/openapi.json get /v1/sessions/{id}/downloads
Every file a browser session you drove over /connect downloaded, kept 7 days, up to 200 MB each: name, content type, size, the URL it came from, and a link to the bytes that works for an hour with no API key. Use it to get a file your own script downloaded; not for an agent run's files, which is outcrawl_agent_downloads.
# Every request a session made, as a stream of folded rows with headers, timing, protocol, remote address, initiator an...
Source: https://outcrawl.ai/docs/api-reference/sessions/every-request-a-session-made-as-a-stream-of-folded-rows-with-headers-timing-protocol-remote-address-initiator-an
/api-reference/openapi.json get /v1/sessions/{id}/network
Every request a session made, as a stream of folded rows with headers, timing, protocol, remote address, initiator and sizes; filter by url, method, status, type, tab or duration, opt into stored bodies with `bodies`, and follow a live session with `follow`. Use to debug a session's traffic; not for a summary, which is the digest, and not for one request, which is sessions.request.
# Export a session as a portable bundle that plays offline.
Source: https://outcrawl.ai/docs/api-reference/sessions/export-a-session-as-a-portable-bundle-that-plays-offline
/api-reference/openapi.json post /v1/sessions/{id}/export
Export a session as a portable bundle that plays offline.
# Fetch one session with its replay URL, share URL, tabs, duration, page count and charges.
Source: https://outcrawl.ai/docs/api-reference/sessions/fetch-one-session-with-its-replay-url-share-url-tabs-duration-page-count-and-charges
/api-reference/openapi.json get /v1/sessions/{id}
Fetch one session with its replay URL, share URL, tabs, duration, page count and charges. This is how a charge is disputed: watch what it bought. Use for one session you hold the id of; not for its requests, which is outcrawl_session_network.
# Links to a session's video, one per recorded stretch of each tab, with its time range and size, plus the tabs.
Source: https://outcrawl.ai/docs/api-reference/sessions/links-to-a-sessions-video-one-per-recorded-stretch-of-each-tab-with-its-time-range-and-size-plus-the-tabs
/api-reference/openapi.json get /v1/sessions/{id}/videos
Links to a session's video, one per recorded stretch of each tab, with its time range and size, plus the tabs. Fetch a link with the same API key.
# One request of a session in full, by the `requestId` the digest or the network stream named: request and response hea...
Source: https://outcrawl.ai/docs/api-reference/sessions/one-request-of-a-session-in-full-by-the-`requestid`-the-digest-or-the-network-stream-named:-request-and-response-hea
/api-reference/openapi.json get /v1/sessions/{id}/request
One request of a session in full, by the `requestId` the digest or the network stream named: request and response headers, timing, protocol, remote address, initiator, redirect, sizes and its stored body when there is one. Not for listing requests.
# Query past agent runs and browser sessions by status, outcome, task or age, each with its replay URL, duration, charg...
Source: https://outcrawl.ai/docs/api-reference/sessions/query-past-agent-runs-and-browser-sessions-by-status-outcome-task-or-age-each-with-its-replay-url-duration-charg
/api-reference/openapi.json get /v1/sessions
Query past agent runs and browser sessions by status, outcome, task or age, each with its replay URL, duration, charges and who may watch it. Use to find the session behind a charge or a failure; not for what happened inside one, which is outcrawl_session_network.
# The console lines a session's pages produced, a page at a time, oldest first, filtered by level and tab, with the scr...
Source: https://outcrawl.ai/docs/api-reference/sessions/the-console-lines-a-sessions-pages-produced-a-page-at-a-time-oldest-first-filtered-by-level-and-tab-with-the-scr
/api-reference/openapi.json get /v1/sessions/{id}/console
The console lines a session's pages produced, a page at a time, oldest first, filtered by level and tab, with the script URL and line when the browser knew them.
# What went wrong in a session, compactly: failed, blocked and 4xx/5xx requests, the slowest ones, the site's own XHR/f...
Source: https://outcrawl.ai/docs/api-reference/sessions/what-went-wrong-in-a-session-compactly:-failed-blocked-and-4xx5xx-requests-the-slowest-ones-the-sites-own-xhrf
/api-reference/openapi.json get /v1/sessions/{id}/digest
What went wrong in a session, compactly: failed, blocked and 4xx/5xx requests, the slowest ones, the site's own XHR/fetch calls, per-origin totals, console errors and the tabs, every list capped and no headers or bodies. Use first when a run failed or a page misbehaved; then fetch one request in full with outcrawl_session_request.
# Aggregate the same usage numbers already attached to each call.
Source: https://outcrawl.ai/docs/api-reference/usage/aggregate-the-same-usage-numbers-already-attached-to-each-call
/api-reference/openapi.json get /v1/usage
Aggregate the same usage numbers already attached to each call. Nothing appears here that was not visible on the call that caused it.
# Connect a card provider account with a credential your server holds: the provider's test runs first and the credentia...
Source: https://outcrawl.ai/docs/api-reference/wallet/connect-a-card-provider-account-with-a-credential-your-server-holds:-the-providers-test-runs-first-and-the-credentia
/api-reference/openapi.json post /v1/wallet/connections
Connect a card provider account with a credential your server holds: the provider's test runs first and the credential is sealed only if it passes (422 provider_test_failed, nothing stored, if not). The same handle again rotates the credential, replacing the old one only if the new one passes. Runs then mint one card per run from it.
# Disconnect a card provider account: every card a run made from it that is still open is closed first, then the creden...
Source: https://outcrawl.ai/docs/api-reference/wallet/disconnect-a-card-provider-account:-every-card-a-run-made-from-it-that-is-still-open-is-closed-first-then-the-creden
/api-reference/openapi.json delete /v1/wallet/connections/{id}
Disconnect a card provider account: every card a run made from it that is still open is closed first, then the credential is deleted. A run using it fails its next fill with a sentence naming the connection.
# Get a single-use upload URL and token (at most five minutes) that let a browser connect ONE card provider under ONE h...
Source: https://outcrawl.ai/docs/api-reference/wallet/get-a-single-use-upload-url-and-token-at-most-five-minutes-that-let-a-browser-connect-one-card-provider-under-one-h
/api-reference/openapi.json post /v1/wallet/connections/intents
Get a single-use upload URL and token (at most five minutes) that let a browser connect ONE card provider under ONE handle without the credential passing through your server. Name your page's origin to send from your own page; without one, only the Outcrawl console may send it. Not for a credential your server holds, which is wallet.connections.create.
# List the card provider accounts this workspace has connected, newest first: handle, provider, account name, status (a...
Source: https://outcrawl.ai/docs/api-reference/wallet/list-the-card-provider-accounts-this-workspace-has-connected-newest-first:-handle-provider-account-name-status-a
/api-reference/openapi.json get /v1/wallet/connections
List the card provider accounts this workspace has connected, newest first: handle, provider, account name, status (active, failing, revoked, pending), warnings and cards made this period. Never a credential, and no route returns one.
# List the five card providers a workspace can connect (Privacy.com, Brex, Slash, Stripe Issuing, Airwallex): regions, ...
Source: https://outcrawl.ai/docs/api-reference/wallet/list-the-five-card-providers-a-workspace-can-connect-privacycom-brex-slash-stripe-issuing-airwallex:-regions-
/api-reference/openapi.json get /v1/wallet/providers
List the five card providers a workspace can connect (Privacy.com, Brex, Slash, Stripe Issuing, Airwallex): regions, currencies, the credential fields each needs, what its cards can do, and whether this deployment can connect it now. Use before connecting one; not your connected accounts, which is wallet.connections.list.
# Read one connected card provider account by id: its status and the reason for it, the account name, warnings from the...
Source: https://outcrawl.ai/docs/api-reference/wallet/read-one-connected-card-provider-account-by-id:-its-status-and-the-reason-for-it-the-account-name-warnings-from-the
/api-reference/openapi.json get /v1/wallet/connections/{id}
Read one connected card provider account by id: its status and the reason for it, the account name, warnings from the last test, and when it was last used and tested. Never a credential.
# Read the workspace wallet settings: approvalAbove, the purchase amount above which a run pauses for a person's yes.
Source: https://outcrawl.ai/docs/api-reference/wallet/read-the-workspace-wallet-settings:-approvalabove-the-purchase-amount-above-which-a-run-pauses-for-a-persons-yes
/api-reference/openapi.json get /v1/wallet/settings
Read the workspace wallet settings: approvalAbove, the purchase amount above which a run pauses for a person's yes. Null means off, the default.
# Run the provider's read-only test again on a connected account and get ok, the account name, warnings and the reason ...
Source: https://outcrawl.ai/docs/api-reference/wallet/run-the-providers-read-only-test-again-on-a-connected-account-and-get-ok-the-account-name-warnings-and-the-reason-
/api-reference/openapi.json post /v1/wallet/connections/{id}/test
Run the provider's read-only test again on a connected account and get ok, the account name, warnings and the reason it failed. Answers 200 either way, and the connection's status follows the result.
# Set approvalAbove { amount, currency }, or null to turn it off: a purchase within a run's spend cap and above this am...
Source: https://outcrawl.ai/docs/api-reference/wallet/set-approvalabove--or-null-to-turn-it-off:-a-purchase-within-a-runs-spend-cap-and-above-this-am
/api-reference/openapi.json put /v1/wallet/settings
Set approvalAbove { amount, currency }, or null to turn it off: a purchase within a run's spend cap and above this amount pauses the run until a person answers approve true or false. It never lets a purchase past the cap.
# Authentication
Source: https://outcrawl.ai/docs/api/authentication
Two bearer credentials: an API key, or an OAuth access token.
Every request carries one bearer credential: an API key, or an OAuth access token. An API key works on `https://api.outcrawl.ai/v1` and on the MCP server alike. An OAuth token works where it was issued for: the token `outcrawl login` gets calls the API, and the token an MCP client gets calls the MCP server. This page is about the HTTP header itself; for signing in to the console, see [console authentication](/docs/account/authentication).
| | API key (`ock_...`) | OAuth access token (`oat_...`) |
| - | - | - |
| Where it comes from | Console → API keys | Signing in: `outcrawl login`, or an MCP client's first use |
| Lifetime | Long-lived, until revoked | 1 hour, refreshes for 30 days while in use |
| Best for | Your own applications, scripts, CI, anything headless | An interactive CLI or MCP client signed in as you |
| Works on | The API and the MCP server | The one it was issued for: the API (CLI) or the MCP server (MCP clients) |
| Scope | The workspace it was created in | The one workspace you picked at sign-in |
| Stops working when | You delete it in the console | It expires, you sign out (`outcrawl logout`), or you are removed from the workspace; removal cuts it off at once |
```bash API key theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/scrape \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
```bash OAuth access token theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/scrape \
-H "Authorization: Bearer oat_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
```
Both SDKs and the curl examples throughout these docs use an API key, the simplest path for code you write yourself. Use `OUTCRAWL_API_KEY` for CI and anything headless; it also takes priority over a stored CLI login, so a script always gets a predictable credential regardless of who is signed in on that machine.
## In the SDKs
Both SDKs send this header for you. Construct the client with no key and it reads `OUTCRAWL_API_KEY` from the environment; pass one in (`apiKey` in TypeScript, `api_key` in Python) and that key is used instead. With neither, the constructor throws a `TypeError` that names the variable, before any request is made.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY
const withKey = new Outcrawl({ apiKey: 'ock_YOUR_KEY_ID.YOUR_SECRET' }); // or pass the key in
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc: # reads OUTCRAWL_API_KEY
...
async with Outcrawl(api_key="ock_YOUR_KEY_ID.YOUR_SECRET") as oc: # or pass the key in
...
asyncio.run(main())
```
# Errors
Source: https://outcrawl.ai/docs/api/errors
The closed 31-code error union.
Every failed request answers the same envelope:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"code": "quota_exceeded",
"message": "...",
"details": {}
}
```
Branch on `code`. It is a closed set of 31 values; `message` is fixed per code and is never meant to be parsed.
| Code | Meaning |
| - | - |
| `unauthorized` | The key was missing, invalid, or not on the allowlist |
| `bad_request` | Malformed or missing input, fixable and retryable after a fix |
| `not_found` | No such resource, or none this key can see |
| `method_not_allowed` | Wrong HTTP method for the route |
| `plan_required` | The key is valid but names no plan this deployment can serve |
| `quota_exceeded` | The account is over a ceiling its plan sets (requests per minute, or monthly credits) |
| `concurrency_limit` | The account is holding every browser its plan allows at once |
| `cap_exceeded` | The request itself is over a ceiling the caller set on that one call |
| `capability_unavailable` | This deployment has nothing configured to serve the capability |
| `no_capacity` | No worker is free right now |
| `exit_unavailable` | No exit could be resolved for the requested geography |
| `tenant_scope` | A connection touched a target outside its own tenant |
| `profile_in_use` | The named profile is leased by another open session |
| `profile_not_found` | No such profile, or it belongs to another account |
| `page_unreachable` | No response arrived at all, resolution or connection failed |
| `page_lost` | The tab stopped existing before the page could be delivered |
| `document_unrepresentable` | The bytes arrived and cannot be turned into text |
| `decode_unavailable` | A stored recording cannot be decoded right now |
| `extraction_failed` | A schema'd extraction did not produce a matching document |
| `answer_lost` | The work finished but the answer never reached the edge |
| `deadline_exceeded` | The caller's own timeout was exceeded by work already done |
| `lab_arm_forbidden` | An agent run asked for an internal experiment this account is not enrolled in; only Outcrawl's own test runs send that, so drop the header |
| `search_refused` | Every search engine tried declined the search; nothing was charged, and a retry goes out from another exit |
| `integration_check_failed` | Connecting or updating an integration ran its check against the MCP server and it failed (422); `details.reason` is `not_mcp`, `unauthorized`, `unreachable` or `missing_secret`, and nothing was stored |
| `integration_changed` | The integration was replaced under the same name between your read and your update (409); nothing was stored, read it again |
| `secret_intent_invalid` | A `PUT` to a secret intent's URL carried a token that does not open it (401); nothing was stored |
| `secret_intent_spent` | The secret intent was already used, expired after five minutes, or its key was revoked (410); mint a new intent |
| `replay_access_denied` | This credential may not read this recording (403); `details.reason` says why, and `operator_key` means a key issued by Outcrawl staff, which never opens a customer's recordings |
| `replay_control_denied` | Someone watching a shared recording asked to drive it (403); only the owning account's key or team can take control |
| `replay_control_cross_site` | A request to drive a live session came from another site with only the player's cookie (403); use the player's own controls or send a key in a header |
| `internal` | Something broke on our side |
## What is retryable
Some of the above resolve on their own and are safe to retry unchanged: `no_capacity`, `decode_unavailable`, `extraction_failed` and `answer_lost` are all cases where the identical call commonly succeeds moments later. The rest are not: retrying a `bad_request` or a `quota_exceeded` without changing anything spends money to fail the same way again.
# Overview
Source: https://outcrawl.ai/docs/api/overview
How the Outcrawl API works.
One base URL, one credential, every capability behind it. Unlike a platform with several surfaces and several keys, Outcrawl is one coherent API.
```
https://api.outcrawl.ai/v1
```
Every request is authenticated (see [authentication](/docs/api/authentication)) and every response carries a `usage` object stating exactly what that call cost, on the result that caused it, nothing appears on a bill that was not already visible on the request that produced it.
## Two framings
| Framing | Shape | Used by |
| - | - | - |
| `json` | One object | Everything by default |
| `ndjson` | A header frame, then zero or more item frames, one JSON object per line | `crawl` (always), `agent.events` (always) |
`agent` answers `json` unless the request sets `stream: true`. `sessions.get` answers `json` unless it sets `live: true`. Every other capability's framing is fixed.
## Errors
A failed request answers `{"code": "...", "message": "...", "details": {...}}`. Branch on `code`, never on `message`. See the [full 31-code table](/docs/api/errors).
## The capabilities
One URL in, clean content out
Stream a whole site from a seed
Results you can act on
Hand over a task
Live CDP control, TypeScript only
Watch a page, notify on change
# Credits
Source: https://outcrawl.ai/docs/billing/credits
The metering unit behind every rate, where your credits come from, and the order they are spent in.
A credit is the unit every rate on the [pricing page](/docs/billing/pricing) is quoted in. Every scrape, crawl, search, browser minute, agent run and monitor check draws from one spendable figure, and that figure is the one **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)** shows as credits left: a request is refused only when it reaches zero.
## Where credits come from
* **Your plan's allowance.** 1,000 on Free, 10,000 on Builder, 50,000 on Pro, 250,000 on Scale. On a paid plan it is added once for each billing period you pay for, on the day the invoice is paid, and lasts until that period ends, plus 24 hours: the renewal is charged in that time, so the next period's credits arrive before the last period's run out. On Free it renews on the 1st of each month (UTC). Unused allowance does not roll over.
* **Credits you buy.** A top-up, by hand or automatic (below). Bought credits never expire.
* **Credits we grant.** A promotional, student or support grant. A grant may carry its own expiry, shown beside it on the billing page.
## The order they are spent in
Spending always draws from whatever expires soonest, so nothing lapses unused while something longer-lived is spent instead. In practice:
1. this period's plan allowance first,
2. then grants with an expiry, the soonest-expiring first (a grant that ends before your period does is spent before the allowance),
3. then credits that never expire: everything you bought.
So a top-up is only touched once the current period's allowance is used up, and whatever is left of it carries into the next period untouched.
## Buying credits
Add credits from **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)** on any plan, whether you are paying for one or not. A paid plan buys at its own price per credit: $49.00 per 10,000 on Builder, $39.80 on Pro, $31.96 on Scale. Free and paused workspaces buy at the Builder rate. A purchase is 100 to 1,000,000 credits and at least $0.50, and the credits are added the moment the payment succeeds.
## Auto top-up
Auto top-up buys credits for you before you run out. A workspace owner or admin sets it up on **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)** with three numbers:
* **Threshold:** charge when spendable credits drop below this. Suggested: a fifth of your plan's monthly credits.
* **Amount:** how many credits to buy each time, at the same price as buying them by hand. Suggested: one month of your plan's credits.
* **Monthly cap:** the most auto top-up may charge in one calendar month (UTC), counting a charge still in progress. Suggested: four top-ups' worth. It must cover at least one.
Spendable credits are checked every minute. Below the threshold, your default card is charged for the amount, and the credits are added as soon as the payment succeeds; they never expire. Only one auto top-up runs at a time for a workspace, and each one is charged at most once, however often the check runs. Once the next one would take the month past the cap, auto top-up waits for the next month instead. While your plan's renewal is being charged (up to two hours after its invoice is created), auto top-up waits too: the new period's credits are about to arrive.
It needs a default card on the workspace. If a charge is declined, or your bank asks to confirm it (which an automatic charge cannot do), auto top-up turns itself off, the billing page says why, and every workspace owner gets an email. The same happens if a single top-up comes to cost more than the monthly cap, for example after a pause or downgrade moves the workspace to a higher price per credit. Nothing is retried on the same card until someone turns it back on. Saving the settings never charges by itself: if credits are already below the new threshold, the first top-up follows within a minute.
## Refunds and disputes
A refund takes back the credits the refunded payment added, in proportion to the amount refunded, rounded in your favour. Refunding a top-up removes that share of its credits; refunding a plan invoice removes that share of its period's allowance, unless the period has already ended. A disputed payment's credits are removed while the dispute is open and added back if it is resolved in your favour.
The share taken back comes out of credits still on the balance first: what is left of the credits that payment added (the refunded period's allowance, for a plan invoice), then your other credits, purchased credits included. Only what the balance cannot cover is owed, and the next credits that arrive, from a renewal, a purchase or a grant, pay that first.
A refund of an auto top-up, or a dispute on any payment, also turns auto top-up off and emails the workspace owners, so the card is not charged again for the credits just taken back.
## Reading your balance
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const credits = await oc.credits();
console.log(credits.granted, credits.spent, credits.balance); // decimal strings
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
credits = await oc.credits()
print(credits.granted, credits.spent, credits.balance) # decimal strings
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/credits -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
`credits` returns four decimal strings for the credits recorded on your account: paid plan periods, purchases and grants. `granted` is what is still live, `expired` is what lapsed unused (shown on its own rather than dropped), `spent` is everything metered, and `balance` is what remains of the live credits after any amount owed. Free's monthly allowance is not a recorded grant, so it is not counted there; the credits left on the billing page include it. Per-call cost is on every response already, see [usage](/docs/billing/usage).
## Changing plan mid month
An upgrade starts a new monthly billing period on the day you make it. It charges the new plan's
full month today, less a credit for the unused time on your current plan, and your billing date
moves to today. That charge adds the same share of the new plan's monthly allowance, at the new
plan's rate: the unused time on the old plan comes back to you as money, so it is not paid for
again in credits. Whatever is left of the old plan's credits stays usable until the date that
period would have ended, and is spent first. A full month's allowance comes only with a full
month's invoice: the first payment on a new subscription, every renewal after it, and resuming a
paused plan with a charge. A downgrade takes effect at the end of the period you already paid for
and charges nothing when it lands; the lower plan's allowance arrives with its first renewal.
Each invoice adds credits at most once. Retrying a payment, or Stripe redelivering the same
webhook, never adds them twice for the same invoice.
# Overview
Source: https://outcrawl.ai/docs/billing/overview
One prepaid credit balance across every capability.
One credit balance, shared across scrape, crawl, search, browsers, agents and monitors. No separate proxy, browser or additional credit type, one number, one bill. Your plan's allowance is spent first each billing period; credits you buy never expire and are spent after it.
Upgrade, downgrade, pause or cancel from **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)**, where you can also buy credits on any plan, set up auto top-up, see every invoice and manage the card on file. See [pricing](/docs/billing/pricing) for what changing, pausing and cancelling do.
Plans and the rate for every metered kind.
Where credits come from, the order they are spent in, top-ups and auto top-up.
The aggregate of what every call has already told you it cost.
Add a card, buy credits, and let auto top-up use it.
# Payment methods
Source: https://outcrawl.ai/docs/billing/payment-methods
Add a card, buy credits, and let auto top-up use it.
Upgrade to a paid plan from **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)**. Choose a plan and a short form collects a billing address, an optional tax ID, and a card, all without leaving the page. Review the plan, its monthly price and the credits it includes, then confirm. If your card's issuer asks for extra verification, that happens in the same window; nothing here ever redirects you away from the console.
Add, remove or change the default card from the same page, on any plan, Free included. Workspace owners and admins manage cards.
[Auto top-up](/docs/billing/credits#auto-top-up) charges the default card. A card whose bank asks to confirm every payment works when you [buy credits](/docs/billing/credits#buying-credits) by hand, where you confirm it in the same window, but not for auto top-up, which charges while nobody is there to confirm: auto top-up turns itself off and emails the workspace owners rather than retrying it.
# Pricing
Source: https://outcrawl.ai/docs/billing/pricing
Plans and the rate for every metered kind.
## Plans
| | Free | Builder | Pro | Scale | Enterprise |
| - | - | - | - | - | - |
| Price | \$0 | \$49/mo | \$199/mo | \$799/mo | custom |
| Credits | 1,000/mo | 10,000/mo | 50,000/mo | 250,000/mo | custom |
| Concurrent browsers | 2 | 5 | 25 | 100 | custom |
| Requests/minute | 120 | 300 | 1,500 | 6,000 | custom |
A paid plan's credits are added once for each billing period you pay for and stay usable until 24 hours after that period ends, while the renewal is charged; Free's renew on the 1st of each month (UTC). Unused plan credits do not roll over. See [credits](/docs/billing/credits) for the order credits are spent in.
## Buying more credits
Buy credits from **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)** on any plan, or turn on [auto top-up](/docs/billing/credits#auto-top-up) to buy a set amount automatically when you run low, up to a monthly cap you choose. A paid plan buys at its own price per credit:
| | Free | Builder | Pro | Scale |
| - | - | - | - | - |
| Per 10,000 credits | \$49.00 (Builder rate) | \$49.00 | \$39.80 | \$31.96 |
A paused plan buys at the Builder rate too. Bought credits never expire, and are spent only after the current period's plan credits.
## Rates
| Kind | Rate |
| - | - |
| Scrape, crawl, search (per page rendered) | 1 credit / page |
| Search results | 5 credits / 10 results |
| Browser session time | 1 credit / minute |
| Proxy usage, browsers and agent runs | 2,000 credits / GB |
| Agent run | model usage + browser time + proxy usage, **about 15 credits / minute** (estimate) |
| Monitor check | 1 credit / check, plus model usage when a check is judged |
## Agent runs
An agent run is billed for the AI model's usage, browser time and proxy usage, in credits. A typical run costs **about 15 credits per minute** all in, and the middle half of runs land between 10 and 25 a minute. Model usage is converted at your plan's price per credit, so on Pro and Scale, where a credit costs less, the same minute is more credits: about 20 on Pro and 25 on Scale. These figures are estimates: the actual cost depends on the task. Most of a run's cost is the model's usage, so a longer run costs more, and most of the difference between a cheap minute and an expensive one is how much page data it moves.
* **Model usage** is what the model actually used on the run's steps, converted to credits as it happens.
* **Browser time** is 1 credit a minute for as long as the run is open, including time it is paused, waiting for you, or under your control.
* **Proxy usage** is 2,000 credits per GB (decimal; 2 credits per MB).
A paused run, or one you are driving yourself, bills browser time and proxy usage only. `caps.budget` is a hard ceiling on all three together. Drive a browser yourself with the SDK and you pay browser time and proxy usage alone.
## Monitors
Every check costs 1 credit, whether or not the page changed. A monitor with `judge: true` also bills the model's usage on each changed page it judges.
Captcha and challenge solving is bundled into the above and is never a separate line item. One credit balance, no separate proxy or browser credits.
## Changing, pausing and cancelling
Upgrade, downgrade, pause or cancel from **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)**. An upgrade applies
immediately and starts a new monthly billing period today: you pay the new plan's full month now,
less a credit for the unused time on your current plan, and the plan renews on this date from
then on. The billing page shows the exact amount and the new renewal date before you confirm. A
downgrade takes effect at the end of the period you already paid for, and nothing is charged
until then.
Cancelling walks through a reason, an offer of the plan one step down, a pause of 1 to 3 months,
and a 30%-off, 3-month discount, before the plan actually ends. A plan can pause once every 12
months, and the discount can be taken once. A paused plan charges nothing and its allowance does
not renew; the workspace keeps the credits it already has. Your first workspace runs on Free's
limits while paused and can buy credits at the Builder rate. Any other workspace has no Free plan
to fall back to, so it has no plan while paused and runs nothing until the plan returns; credits it
holds wait for it.
On the chosen date, billing resumes. The plan returns with the next renewal, charged on your usual
billing date, or at once if you pay the restart the billing page shows: one month today, which
restarts the monthly period from today. The billing page shows that amount before anything is
charged. Resuming early is one click: it charges nothing if the pause has not yet skipped a
renewal, and otherwise it is that same restart. A cancelled plan keeps running, at full
access, until the end of the period already paid for, and can be resumed with one click any time
before then.
## Enterprise
* Custom credits and concurrency
* Dedicated capacity
* 99.9% monthly uptime SLA
* Priority support during business hours
* Best-effort incident response
* Service credits for qualifying SLA breaches
* Enterprise security and networking
* Forward-deployed engineers, where applicable
Response times are negotiated and priced per contract.
# Usage
Source: https://outcrawl.ai/docs/billing/usage
The aggregate of what every call already told you it cost.
`usage` aggregates the same numbers already attached to each call. Nothing appears here that was not already visible on the response that caused it, this is a rollup, not a separate source of truth.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const usage = await oc.usage({ from: '2026-09-01', to: '2026-09-30', groupBy: 'day' });
console.log(usage.total);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
# `from` is a Python keyword, so the date range goes in as a dict.
usage = await oc.usage(**{"from": "2026-09-01", "to": "2026-09-30", "groupBy": "day"})
print(usage.total)
asyncio.run(main())
```
```bash CLI theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl usage
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl "https://api.outcrawl.ai/v1/usage?from=2026-09-01&to=2026-09-30&groupBy=day" \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
`from` and `to` are required, as ISO dates or instants; `groupBy` is `hour`, `day` (the default), `month` or `capability`. The CLI defaults to the current calendar month when you give neither.
See [credits](/docs/billing/credits) for the balance this usage draws down against, and **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)** for what plan is funding it and what has been paid.
# Human input model
Source: https://outcrawl.ai/docs/browsers/human-input
Why click, fill and type behave the way they do by default.
Your agent gets the page, every time. That claim rests on a specific, measured decision: `click()`, `fill()`, `type()`, `hover()`, `press()` and `selectOption()` all go through a measured human input model by default, rather than the exact, instantaneous input a scripted browser normally produces.
| | Measured human input | Unmodified scripted default |
| - | - | - |
| Click landing offset | Mean 8.1px, sd 15.0 | 0.0px, sd 0.0, exact centre every time |
| Key flight (time between keystrokes) | Mean 127ms, sd 89.4 | Fixed sd 34.3 |
| Sub-movements per stroke | Median 3 | 30, with per-sample tremor |
The ordering is deliberate: the failure worth guarding against is forgetting to opt into careful input, not forgetting to opt out of it. A page that accepts input correctly the first time is one where forms submit, sessions hold, and logins stick, which is the whole reliability claim this product sells.
`page.goto()` and `page.title()` are not modelled this way, deliberately, navigation has no human analogue.
## The escape hatch
`page.raw.*` reaches Playwright's own unmodified input directly, for the setup steps where speed genuinely matters more than realism. Every call through it is recorded against the session's own replay, so a run that later behaves unexpectedly is not a mystery to whoever reviews it afterward, the choice to use it is visible on review rather than hidden.
This page describes measured input behaviour and why it matters for pages working correctly. The product does not name what it runs underneath; the claim is that the page loads, the form accepts the input, and the session holds.
# Overview
Source: https://outcrawl.ai/docs/browsers/overview
A real browser your agent drives directly, over plain CDP.
Live browser control is **TypeScript-SDK only**. Not Python, not the CLI, not MCP. It needs a CDP connection held open in the caller's own process, so it is the one capability that cannot be a REST endpoint; the API, MCP and CLI reach the same underlying page work through [agents](/docs/agents/overview) and [scrape](/docs/scrape/overview) instead, which run the loop on our side. That is a difference in kind, not a missing feature.
The rate is **1 credit per minute** of session time, plus proxy usage at 2,000 credits per GB. An agent driving the browser adds its model's usage on top, see [pricing](/docs/billing/pricing).
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const profile = await oc.profiles.create({ label: 'demo-user-42' }); // minted once, reused by id
const browser = await oc.browser({ profile: profile.id }); // its cookies and logins persist
const page = await browser.newPage();
await page.goto('https://practicetestautomation.com/practice-test-login/'); // a public demo login
await page.fill('#username', 'student'); // human input model, the default
await page.fill('#password', 'Password123');
await page.click('#submit');
await page.raw.waitForURL('**/logged-in-successfully/');
console.log(await page.title()); // "Logged In Successfully | Practice Test Automation"
await browser.close();
```
`BrowserOptions` is exactly `{ profile, proxy, record }`. `profile` is a profile from [account/profiles](/docs/account/profiles), pass one to keep a login across sessions. `record` is on by default, every session gets a replay, see [sessions](/docs/browsers/sessions).
**You bring no infrastructure.** There are no proxies to buy or browsers to run yourself. `oc.browser()` allocates a session on the fleet and connects to it over CDP, nothing launches locally, and every request already leaves from a residential exit. You name a geography, not an exit list.
Why `click`, `fill` and `type` behave the way they do by default.
Every browser run produces a replay you can watch or export.
# Sessions
Source: https://outcrawl.ai/docs/browsers/sessions
Every browser-driven run produces a replay you can watch or export.
Every browser session, whether driven live or by an agent, produces a session with a replay. The three replay calls below have no MCP tool or CLI command: a replay is something a person watches in a browser to check a charge, not something a model or a script can act on. [Downloads](#downloads) are different, and every client reaches them.
## Get one session
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const session = await oc.sessions.get('');
console.log(session.replayUrl, session.duration, session.pages, session.usage);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
session = await oc.sessions.get("")
print(session.replay_url, session.duration, session.pages, session.usage)
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/sessions/ \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Returns the replay URL, duration in milliseconds, page count and usage. This is how a charge gets disputed: watch what it bought. To follow a session that is still running instead of waiting for it to end, iterate `session.live()` in either SDK, or add `?live=true` to the request.
A replay never autoplays. A person presses play.
## List sessions
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const sessions = await oc.sessions.list({ status: 'completed', since: '7d' });
for (const session of sessions) {
console.log(session.id, session.outcome, session.replayUrl);
}
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
sessions = await oc.sessions.list(status="completed", since="7d")
for session in sessions:
print(session["id"], session.get("outcome"), session.get("replayUrl"))
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl "https://api.outcrawl.ai/v1/sessions?status=completed&since=7d" \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Query past sessions by status, outcome, task or age.
## Export a session
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const bundle = await oc.sessions.export('', './session.outcrawl');
console.log(bundle.url, bundle.bytes, bundle.expiresAt);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
bundle = await oc.sessions.export("")
print(bundle["url"], bundle["bytes"], bundle["expiresAt"])
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/sessions//export \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Exports a session as a portable bundle that plays back offline, for attaching to a ticket or an audit trail. The answer carries a download `url` and when it expires. The TypeScript SDK also downloads the bundle when you pass a file path; in Python, download it from `url`.
## Downloads
Every file a browser you drive with `oc.browser()` downloads is kept for 7 days, up to 200 MB each. Click a download in your script, then list the session's downloads. A file shows up as soon as its download finishes.
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { setTimeout as sleep } from 'node:timers/promises';
import { Outcrawl, saveDownload } from '@outcrawl/sdk';
const oc = new Outcrawl();
const browser = await oc.browser();
const page = await browser.newPage();
await page.goto('https://example.com/reports');
await page.click('#export-csv');
let downloads = await browser.downloads();
while (downloads.length === 0) {
await sleep(1000); // the file is listed once it has finished downloading
downloads = await browser.downloads();
}
console.log(await saveDownload(downloads[0], '.'));
await browser.close();
```
After the browser closes, `oc.sessions.downloads(browser.sessionId)` lists the same files, and so does `outcrawl sessions downloads ` or `await oc.sessions.downloads(session_id)` in Python. Each file carries a link that works for an hour with no API key. See [downloads](/docs/agents/downloads) for the full shape, paging and how files are saved.
Playwright's own `download` event works too: on a page from `oc.browser()`, `download.saveAs(path)` waits for the file to be kept and writes it to `path`. A Playwright or Puppeteer script that connects without the TypeScript SDK does not get this, because the file is never on your disk; see [downloads](/docs/agents/downloads#playwrights-own-download-event) for how to fetch it instead.
# Agent
Source: https://outcrawl.ai/docs/cli/agent
Submit, watch and manage an agent run from the terminal.
Agents are the richest command family in the CLI, matching the richest family in the registry: eight routes behind one task.
## Submit a run
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl agent "find the cheapest flight to Lisbon"
outcrawl agent "find the cheapest flight from New York to Lisbon" --budget 20.00 --max-steps 40
```
`--budget` is a hard ceiling on what the run is billed. Without it, the CLI applies a sane default rather than refusing outright; name your own once a task is worth more than that. The second search cost 7.4–9.2 credits on 2026-10-03 (n=3), so 20.00 is more than twice that. Every run starts fresh: there is no command that continues a previous run, so write each task to stand on its own. Submitting returns an id in milliseconds, the run itself can take hours, so the next commands are how you follow it.
## Watch it
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl agent status # where it has got to, and what it has spent
outcrawl agent events # the full step-by-step log, streamed as it happens
outcrawl agent results # what it has produced so far, paginated
```
`status` is safe to poll as often as you like, it is a row read, not a browser. `events` streams, so it is the command for watching a run live rather than sampling it.
## Manage it
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl agent cancel # stop it; work already done stays billed
outcrawl agent control # take the wheel, the browser stays open
outcrawl agent control --release # hand it back, it resumes where you left it
outcrawl agent answer --answer "" # answer the question a paused run asked
```
`control` is the way to step in for something you would rather do yourself, a login you will not automate, a form only you should fill: the browser stays open and `control --release` hands it back on whatever page you left it on. No command sends a running agent new instructions; cancel and resubmit for that.
`answer` is for a run that stopped to ask: when it needs something only you know, its status is `paused` and `pause.question` in `agent status --json` holds what it asked. Answer it and the run carries on from where it stopped; left unanswered, the question expires and the run carries on without it. A card, a billing profile or a security code is never an answer: send it with `agent secrets add`.
## Attach a file
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl agent files add
```
Hands a file to the run so it can attach it to a page. The file is destroyed when the run ends, on every ending.
## Get the files it downloaded
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl agent downloads # name, size, type, expiry and link of each file
outcrawl agent downloads --save ./files # save every file into a folder that already exists
outcrawl sessions downloads --save ./files # the same for a browser you drove
```
Every file the run's browser downloaded is kept for 7 days, up to 200 MB each. `--save` prints `saved ` for each file and never overwrites one already there. Without it, the command saves nothing. See [downloads](/docs/agents/downloads).
## Scripting: the contract every command keeps
Every command in this CLI, not only `agent`, keeps the same contract whether a person runs it or a script does, unchanged by the colour, charts and prompts a terminal now gets ([Reference](/docs/cli/reference)):
* **`--json` on any command** puts exactly the API's response body on stdout, one line, nothing else mixed in: no banner, no colour, no "next step" hint.
* **Four exit codes**: `0` worked, `1` the call was made and did not work, `2` the invocation was wrong and nothing was sent, `3` the API refused (auth, plan or tenant scope).
* **Nothing ever prompts without a real terminal.** A missing id, a `secrets create` value with nothing to read it from: each is a usage error on stderr with the exit code above, never a hang waiting on input that cannot arrive.
* **`NO_COLOR`**, set to anything, turns off every colour code even when stdout is a real terminal.
* **A secret is piped, not typed, in a script:**
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
echo "4111111111111111" | outcrawl secrets create card_amex_personal --kind card --json
```
`outcrawl agent` is no exception: run from CI with `--json`, it behaves exactly as it always has.
# Installation
Source: https://outcrawl.ai/docs/cli/installation
Install the Outcrawl CLI on macOS, Linux or Windows.
## macOS and Linux
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -fsSL https://get.outcrawl.ai/cli/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
outcrawl login
```
Installs the current release directly: a signed static binary, nothing else to set up. It goes to `/usr/local/bin` when you can write there, and to `~/.local/bin` otherwise; a stock Mac does not have `~/.local/bin` on its PATH, so the second line makes `outcrawl` findable in this shell. The installer prints the same line when it is needed; add it to your shell profile to keep it.
Or with Homebrew, which puts it on your PATH for you:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
brew install outcrawlai/tap/outcrawl
```
## Windows
```powershell Scoop theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
scoop bucket add outcrawl https://github.com/outcrawlai/scoop-bucket
scoop install outcrawl
```
Or download the platform archive directly:
`GET https://get.outcrawl.ai/cli/manifest.json` lists the current version and a sha256 digest for every platform, including `windows_amd64` and `windows_arm64`.
The path pattern is `https://get.outcrawl.ai/cli//outcrawl__windows_.zip`, built from the version the manifest just gave you.
Hash the download against the manifest's digest before extracting. This is the same three-hash discipline `outcrawl upgrade` runs on every update: publish against download, the archive's own contents, and the installed binary, each checked separately.
## Sign in
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl login
```
Opens a browser with a short code already filled in; approve it, pick a workspace, and the CLI is signed in. This is the RFC 8628 device grant, the same standard flow GitHub's and AWS's own CLIs use: no password or key touches your terminal. The session stays signed in until you run `outcrawl logout`, or you are removed from the workspace, either of which cuts it off at once.
## CI and headless environments
Nothing to click through on a server. Use an API key instead, either the environment variable or the config file, and it takes priority over a stored login:
```bash Environment variable theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
export OUTCRAWL_API_KEY=ock_YOUR_KEY_ID.YOUR_SECRET
```
```json Config file theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"apiKey": "ock_YOUR_KEY_ID.YOUR_SECRET",
"apiUrl": "https://api.outcrawl.ai"
}
```
The config file lives at `$XDG_CONFIG_HOME/outcrawl/config.json` (`%APPDATA%\outcrawl\config.json` on Windows). Precedence is flag, then environment variable, then a stored login or the file, then default. `OUTCRAWL_API_URL` overrides the default base URL if you need to.
`--verbose` never prints the key itself, and scrubs it from its own error messages too, so a mistyped flag never leaks a credential into your terminal history.
## Staying current
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl upgrade # verify and install the published release
outcrawl upgrade --check # report what is published, change nothing
outcrawl upgrade --force # re-install even when the version already matches
```
A background check runs at most once every 24 hours and never blocks a command; set `OUTCRAWL_NO_VERSION_CHECK=1` to turn it off entirely.
# CLI
Source: https://outcrawl.ai/docs/cli/overview
Scrape, crawl, search, agents and monitors, all from your terminal.
The Outcrawl CLI is a single static binary. No runtime to install first, no Node, no Python.
**Get started in seconds:**
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -fsSL https://get.outcrawl.ai/cli/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
outcrawl login
outcrawl scrape https://example.com
outcrawl agent "find the cheapest flight to Lisbon"
```
`outcrawl login` opens a browser to sign in and approve the device once; after that the CLI stays signed in. Running in CI or anywhere headless? Set `OUTCRAWL_API_KEY` instead, see [installation](/docs/cli/installation).
Install on macOS, Linux or Windows, sign in or configure a key.
Submit, watch and manage an agent run from the terminal.
Every command, generated from the same registry as the API.
## Or just run outcrawl
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl
```
Typed at a terminal, this opens a place to work, in the manner of Claude Code. Your workspace, plan and the credits you have left sit at the top, everything you have done this session fills the middle, and a prompt waits at the bottom.
* **Type a task** and press enter, and an agent run starts. Its steps appear as it takes them (the pages it opens, what it reads), then its answer and what it cost. `esc` asks before it stops the run; work already done is billed.
* **`/scrape `** shows the page as text you can read in the terminal.
* **`/search `** numbers the results, and **`/scrape 2`** reads the second one.
* **`/crawl `** lists pages as they arrive; `esc` stops the crawl.
* **`/budget 5`** caps the agent runs you start next at 5 credits; `/budget off` clears it.
* **`/control`** gives you the live browser of a running agent run, to take over a step yourself.
* **`/runs`**, **`/usage`** and **`/account`** show your recent runs, this month's spend by day and by capability, and who is signed in with which keys.
`tab` completes a command, the arrow keys bring back earlier prompts, `/help` lists every command and key, and `q` on an empty prompt quits. Set `NO_COLOR` and it draws in plain monochrome.
The credits at the top refresh after each action finishes. Every figure is one your account already reports; the app never works one out for itself.
Every other invocation is unchanged. Pipe `outcrawl scrape` from a script or CI and stdout is still exactly the API's body; without a real terminal on both stdin and stdout, a bare `outcrawl` still prints its command list to stderr and exits 2, exactly as it always has.
## The commands are the registry's, not the binary's
There is no hand-written command list. Every command comes from the same 46-row capability registry that drives the REST API and the MCP server: 44 of the 46 rows carry a CLI command, and a new capability does not compile into the binary until it has one. The two that carry none are `sessions.export` and `secrets.intent`.
# Reference
Source: https://outcrawl.ai/docs/cli/reference
Complete command reference for the Outcrawl CLI.
Every command below comes from the registry's own `cliCommand` column, 44 of 46 rows. Nothing here is hand-maintained separately from the API it calls.
| Command | Description |
| - | - |
| **Sign-in** | |
| `outcrawl login` | Sign in with a browser, device grant, no key needed |
| `outcrawl logout` | End the stored session |
| **Scrape, crawl, search** | |
| `outcrawl scrape ` | Fetch one URL, return clean content |
| `outcrawl crawl ` | Walk a site from a seed, stream every page |
| `outcrawl crawl status ` | Where a crawl has got to |
| `outcrawl crawl results ` | Pages delivered so far |
| `outcrawl crawl cancel ` | Stop a crawl |
| `outcrawl search ` | Web search, optionally rendered |
| **Agent** | |
| `outcrawl agent ""` | Submit a task, get an id back |
| `outcrawl agent list` | Your runs, newest first |
| `outcrawl agent status ` | Status, plan, spend |
| `outcrawl agent events ` | The step-by-step log, streamed |
| `outcrawl agent results ` | What it has produced |
| `outcrawl agent cancel ` | Stop it |
| `outcrawl agent control [--release]` | Take the wheel, or hand it back with `--release` |
| `outcrawl agent answer --answer ""` | Answer the question a paused run asked |
| `outcrawl agent files add ` | Attach a file to a run |
| `outcrawl agent downloads [--save ]` | Every file a run downloaded |
| **Monitors** | |
| `outcrawl monitors create` | Watch a URL on a schedule |
| `outcrawl monitors ls` | List monitors with their last check |
| `outcrawl monitors rm ` | Delete a monitor |
| **Profiles** | |
| `outcrawl profiles create` | Create a profile that keeps logins |
| `outcrawl profiles ls` | List profiles |
| `outcrawl profiles get ` | Fetch one profile |
| `outcrawl profiles rm ` | Delete a profile |
| **Secrets** | |
| `outcrawl secrets create` | Store a credential under a handle |
| `outcrawl secrets ls` | List handles, never values |
| `outcrawl secrets rm ` | Delete a credential |
| **Rules** | |
| `outcrawl rules get` | The workspace rules, enforced or guidance |
| `outcrawl rules set` | Replace the rules, get the classification back |
| **Integrations** | |
| `outcrawl integrations connect ` | Connect an MCP server for an agent to call |
| `outcrawl integrations test` | Check an MCP server works, storing nothing |
| `outcrawl integrations ls` | Everything an agent can reach outside the browser |
| `outcrawl integrations update ` | Change a connector's tools or secret in place |
| `outcrawl integrations rm ` | Disconnect a server |
| **Sessions** | |
| `outcrawl sessions list` | Past sessions, newest first |
| `outcrawl sessions get ` | One session: replay, share, tabs, cost |
| `outcrawl sessions events ` | Stream a session's recorded events |
| `outcrawl sessions network ` | Stream every request a session made |
| `outcrawl sessions digest ` | What went wrong in a session, briefly |
| `outcrawl sessions request ` | One request in full, with its body |
| `outcrawl sessions console ` | A session's console lines |
| `outcrawl sessions har ` | Save a session's network as a HAR file |
| `outcrawl sessions videos ` | Links to a session's video |
| `outcrawl sessions downloads [--save ]` | Every file a browser session downloaded |
| `outcrawl sessions share ` | Choose who may watch a session |
| **Usage and credits** | |
| `outcrawl usage` | Aggregate usage |
| `outcrawl credits` | Granted, spent, remaining |
| **Other** | |
| `outcrawl upgrade [--check\|--force]` | Verify and install the published release |
Two rows carry no CLI command: `sessions.export` and `secrets.intent`. Export a recording from the SDK or the API; a secret intent is minted by your own app server for a page a person types into.
## `usage` date ranges
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl usage # this calendar month, the default
outcrawl usage --period 7d # 24h | 7d | 30d | this-month | last-month
outcrawl usage --from 2026-09-01 --to 2026-09-30 # --from and --to, always together
```
`--period` and `--from`/`--to` are alternatives: combining them is refused, and `--from` without `--to` (or the reverse) is refused the same way it always has been. Give neither and the range defaults to the current calendar month, in UTC.
## `secrets create` values
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl secrets create card_amex_personal --kind card
```
Prompts for the value, hidden, when run at a terminal, and reads it from stdin instead when the command is piped.
`--value` still works, for a script that already passes it, but the value never reaches output. Using it prints a warning to stderr, once, every time (D19): a credential does not belong on a command line a shell history file can read.
## Tips
| Exit code | Meaning |
| - | - |
| `0` | Worked |
| `1` | The call was made and did not work |
| `2` | The invocation was wrong, nothing was sent |
| `3` | The API refused: auth, plan, or tenant scope |
Set `OUTCRAWL_TIMING=1` for a per-phase timing breakdown on stderr (startup, parse, config, DNS, connect, TLS, request, server, body, render). Set `OUTCRAWL_NO_VERSION_CHECK=1` to disable the background update check entirely. Run `outcrawl --help` for detailed usage on any command.
# Job lifecycle
Source: https://outcrawl.ai/docs/crawl/jobs
Poll status, read results and cancel a crawl by its id.
A crawl's id, from the first frame of its stream, is what every call on this page needs: `crawl.id` in TypeScript, `crawl.job["id"]` in Python once the first frame has arrived. Store it before you need it, especially if the connection that started the crawl might drop.
```mermaid theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
flowchart LR
A[submit] --> B[status]
B --> C[results]
B --> D[cancel]
C --> B
```
## Status
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const job = await oc.crawl.get('');
console.log(job.status, job.discovered, job.completed, job.credits, job.failures);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
job = await oc.crawl.get("")
print(job["status"], job["discovered"], job["completed"], job["credits"], job["failures"])
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/crawl/ \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Status, pages discovered, pages completed, spend so far, and the per-URL failures. Safe to poll, it is a row read and not a browser.
## Results
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
let cursor: string | undefined;
do {
const batch = await oc.crawl.results('', { cursor });
for (const record of batch) {
console.log(record.page.url, record.page.markdown);
}
cursor = batch.cursor;
} while (cursor !== undefined);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
cursor = None
while True:
batch = await oc.crawl.results("", cursor=cursor)
for record in batch:
print(record["page"]["url"], record["page"].get("markdown"))
cursor = batch.cursor
if cursor is None:
break
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl "https://api.outcrawl.ai/v1/crawl//results?cursor=" \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
The pages a crawl has delivered, oldest first, paginated with a `cursor`. Pages land during the crawl, so this answers before the crawl has finished, and a cancelled crawl still delivers what it had already fetched.
## Cancel
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const job = await oc.crawl.cancel('');
console.log(job.status, job.completed);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
job = await oc.crawl.cancel("")
print(job["status"], job["completed"])
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl -X POST https://api.outcrawl.ai/v1/crawl//cancel \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Stops a crawl. It settles with the pages already reached. Pages already fetched stay billed, the `limit` you set on submit is the spending protection, not a refund argued afterward.
If the connection that started a streaming crawl drops, the crawl itself is cancelled. These three calls are how you read the pages you already paid for afterward.
# Overview
Source: https://outcrawl.ai/docs/crawl/overview
Walk a site from a seed URL, stream every page as it is fetched.
Crawl walks a site from a seed URL and streams every page as it is scraped. Use it when the set of URLs is not known in advance, not for one page (that is [scrape](/docs/scrape/overview)), and not for content behind a login flow.
A crawl always streams as newline-delimited JSON: a header frame, then one JSON object per page as it is fetched, and answers a job id immediately so you can follow along without holding the whole crawl in memory.
The rate is **1 credit per page**, the same rate as a direct scrape.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const crawl = await oc.crawl('https://example.com', { limit: 100 });
console.log(crawl.id); // store this to reach the crawl later
for await (const page of crawl) {
console.log(page.url, page.markdown);
}
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
crawl = await oc.crawl("https://example.com", limit=100)
async for page in crawl:
print(page["url"], page.get("markdown"))
print(crawl.job["id"]) # store this to reach the crawl later
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/crawl \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"limit": 100
}'
```
A crawl is read once, as it streams. To read it again, or after the connection drops, use its id with the [job lifecycle](/docs/crawl/jobs) calls.
The MCP tool `outcrawl_crawl` folds the whole stream into one tool response. A model gets the complete set of pages back from a single call rather than reading a stream itself.
Poll status, read results and cancel a crawl by its id.
# Book the first appointment slot that fits your week
Source: https://outcrawl.ai/docs/examples/appointment-slot-hunter
A monitor watches a booking page; when a slot opens, your rules pick it and an agent books it, or holds it for you to confirm in the live view.
The dermatologist is booked out for four months, the passport office for six weeks, and cancellations appear at random and go within the hour. Refreshing a booking page all day is the only way to get one. This program watches the page for you, reads every open slot the moment the page changes, picks the first one that fits your rules in your own code, and either books it or gets it to the last screen and hands you the live browser to confirm it yourself.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* A [profile](/docs/account/profiles) signed in to the booking site with your details saved on the account, and its id in `PROFILE`. The monitor and every run use it, so they see the page as you do.
* A server reachable from the internet (the monitor posts to it), and a long random string in `HOOK_TOKEN` for its paths.
* A way to reach your phone. The program posts to an [ntfy](https://ntfy.sh) topic; any push or chat webhook works the same way.
Book for yourself, one appointment at a time. The program deletes its monitor once a booking is made (in hand-off mode, when you tap the done link), so it never takes a second slot.
## How it works
[`monitors.create`](/docs/monitors/overview) checks the page every 10 minutes with your profile. It posts to your endpoint only when the page really changes, and a diff that only removes lines (a slot going) is ignored.
Your endpoint starts an [agent run](/docs/agents/overview) that books nothing: it answers a [schema](/docs/agents/overview#typed-results) listing each open slot's date, time, clinician and location.
`acceptable()` applies your rules (weekdays, before 10:00 or after 16:00, within 21 days) to that list. The agent never decides which slot is good enough; your code does, before anything irreversible happens.
It books the slot your code approved and no other, and returns the confirmation number. [Rules](/docs/account/rules) keep it on the clinic's site and stop it making a payment.
The run picks the slot and goes through the form. When its event stream shows the slot being picked, your code calls [`control('take')`](/docs/agents/human-in-the-loop), and your phone gets the live view link. You press take control there, confirm, and tap the done link, which ends the run and deletes the monitor.
## The program
Run it once with `watch` to create the monitor, then run it as a server. Set `SLOT_MODE=hand-off` to confirm bookings yourself; it books on its own otherwise.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import express from 'express';
import { setTimeout as sleep } from 'node:timers/promises';
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY
const BOOKING_PAGE = 'https://booking.riversideclinic.example/dermatology';
const PROFILE = 'prof_8c1d2e'; // the id of a profile already signed in to the booking site
const PUBLIC_URL = 'https://yourapp.example';
const TOKEN = process.env.HOOK_TOKEN ?? ''; // long and random: it is the only credential on these paths
const NTFY = 'https://ntfy.sh/your-private-topic'; // push to your phone
const MODE = process.env.SLOT_MODE === 'hand-off' ? 'hand-off' : 'book';
// Your rules, checked in code before anything is booked.
const WITHIN_DAYS = 21;
const WEEKDAYS = [1, 2, 3, 4, 5]; // Monday to Friday
const acceptable = (slot: Slot) => {
const at = new Date(`${slot.date}T${slot.time}:00`);
const hour = at.getHours();
return (
at.getTime() > Date.now() &&
at.getTime() < Date.now() + WITHIN_DAYS * 86_400_000 &&
WEEKDAYS.includes(at.getDay()) &&
(hour < 10 || hour >= 16) // before work or after it
);
};
interface Slot { date: string; time: string; clinician: string; location: string }
interface Booking { booked: boolean; confirmation: string; date: string; time: string; clinician: string }
const slotSchema = {
type: 'object',
properties: {
slots: {
type: 'array',
items: {
type: 'object',
properties: {
date: { type: 'string', description: 'YYYY-MM-DD' },
time: { type: 'string', description: '24-hour HH:MM' },
clinician: { type: 'string' },
location: { type: 'string' },
},
required: ['date', 'time', 'clinician', 'location'],
},
},
},
required: ['slots'],
};
const bookingSchema = {
type: 'object',
properties: {
booked: { type: 'boolean' },
confirmation: { type: 'string', description: "The confirmation number the site shows, or ''." },
date: { type: 'string' },
time: { type: 'string' },
clinician: { type: 'string' },
},
required: ['booked', 'confirmation', 'date', 'time', 'clinician'],
};
const rules = ['Only use riversideclinic.example.', 'Never make a payment.'];
async function push(title: string, message: string, click?: string): Promise {
await fetch(NTFY, {
method: 'POST',
headers: { Title: title, Priority: 'urgent', ...(click ? { Click: click } : {}) },
body: message,
});
}
let hunting = false;
const handOffs = new Map(); // run id to the monitor that found its slot
async function hunt(monitorId: string): Promise {
// 1. Read every open slot. Nothing is booked in this run.
const found = await oc.agent({
task: `Open ${BOOKING_PAGE} and list every appointment slot that can be booked now, across the next ${WITHIN_DAYS} days.`,
schema: slotSchema,
profile: PROFILE,
caps: { budget: '40.00', duration: '3m' },
rules,
});
const listed = found.data as { slots: Slot[] } | undefined; // shaped by slotSchema
const slots = (listed?.slots ?? []).sort((a, b) => `${a.date}T${a.time}`.localeCompare(`${b.date}T${b.time}`));
// 2. Your code decides, not the agent.
const slot = slots.find(acceptable);
if (slot === undefined) return console.log(`${slots.length} open slots, none within your rules`);
const when = `${slot.date} at ${slot.time} with ${slot.clinician}`;
// 3a. Book mode: a second run books exactly the slot your code approved.
if (MODE === 'book') {
const run = await oc.agent({
task: `On ${BOOKING_PAGE}, book the slot on ${slot.date} at ${slot.time} with ${slot.clinician} at ${slot.location}.
Use the details already saved on the account. Book that slot and no other; if it is gone, book nothing.`,
schema: bookingSchema,
profile: PROFILE,
caps: { budget: '65.00', duration: '5m' },
rules,
});
const booking = run.data as Booking | undefined;
if (booking?.booked) {
await oc.monitors.delete(monitorId); // done hunting
await push('Appointment booked', `${when}. Confirmation ${booking.confirmation}.`, run.replayUrl);
} else {
console.log(`could not book ${when}: ${run.stopReason ?? run.status}`, run.replayUrl);
}
return;
}
// 3b. Hand-off mode: the agent gets the slot in hand, and you confirm it yourself.
const job = oc.agent({
task: `On ${BOOKING_PAGE}, pick the slot on ${slot.date} at ${slot.time} with ${slot.clinician} at ${slot.location}
and go through the booking form with the details saved on the account. Do not press the final confirm button.`,
profile: PROFILE,
caps: { budget: '150.00', duration: '10m' }, // time the run waits for you is billed; these caps bound it
rules,
});
const runId = await job.id();
handOffs.set(runId, monitorId);
for await (const record of job) {
const step = record.value as { action?: { description?: string } } | null;
if (record.kind === 'step' && step?.action?.description?.includes(slot.time)) {
await job.control('take', 'me'); // holds at the agent's next step boundary
break;
}
}
let run = await job.status();
while (run.status === 'queued' || run.status === 'running') {
await sleep(2000);
run = await job.status();
}
if (run.pause?.viewerUrl === undefined) return console.log(`run ended before the hand-off: ${run.stopReason}`);
await push(
`Slot held: ${when}`,
`Tap to open the live view, press take control, and confirm. When you are done: ${PUBLIC_URL}/done/${TOKEN}/${runId}`,
run.pause.viewerUrl,
);
}
if (process.argv[2] === 'watch') {
const monitor = await oc.monitors.create({
url: BOOKING_PAGE,
label: 'slots-dermatology',
schedule: '*/10 * * * *', // every 10 minutes
profile: PROFILE, // see the page the way you see it when signed in
notify: { webhook: `${PUBLIC_URL}/hooks/slots/${TOKEN}` },
});
console.log('watching', monitor.id);
} else {
const app = express();
app.use(express.json({ limit: '2mb' }));
app.post(`/hooks/slots/${TOKEN}`, (req, res) => {
res.sendStatus(202); // answer at once: a monitor delivers each change once and never retries
const { monitorId, diff } = req.body as { monitorId: string; diff?: string };
const added = diff === undefined || /^\+(?!\+\+)/m.test(diff); // a diff that only removes lines is a slot going, not coming
if (!added || hunting) return;
hunting = true;
hunt(monitorId)
.catch((err) => console.error('hunt failed', err))
.finally(() => (hunting = false));
});
// The link in the push: you have confirmed, so end the run, stop paying for it, and stop hunting.
app.get(`/done/${TOKEN}/:runId`, async (req, res) => {
await oc.agent.cancel(req.params.runId);
const monitorId = handOffs.get(req.params.runId);
if (monitorId !== undefined) await oc.monitors.delete(monitorId);
res.send('Run ended.');
});
app.listen(3000, () => console.log(`listening on :3000 in ${MODE} mode`));
}
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import os
import re
from datetime import datetime, timedelta
import httpx
from fastapi import BackgroundTasks, FastAPI, Request
from outcrawl import Outcrawl
oc = Outcrawl() # one client for the life of the server; reads OUTCRAWL_API_KEY
app = FastAPI()
BOOKING_PAGE = "https://booking.riversideclinic.example/dermatology"
PROFILE = "prof_8c1d2e" # the id of a profile already signed in to the booking site
PUBLIC_URL = "https://yourapp.example"
TOKEN = os.environ.get("HOOK_TOKEN", "") # long and random: it is the only credential on these paths
NTFY = "https://ntfy.sh/your-private-topic" # push to your phone
MODE = "hand-off" if os.environ.get("SLOT_MODE") == "hand-off" else "book"
RULES = ["Only use riversideclinic.example.", "Never make a payment."]
# Your rules, checked in code before anything is booked.
WITHIN_DAYS = 21
WEEKDAYS = {0, 1, 2, 3, 4} # Monday to Friday
def acceptable(slot: dict) -> bool:
at = datetime.fromisoformat(f"{slot['date']}T{slot['time']}")
return (
datetime.now() < at < datetime.now() + timedelta(days=WITHIN_DAYS)
and at.weekday() in WEEKDAYS
and (at.hour < 10 or at.hour >= 16) # before work or after it
)
SLOT_SCHEMA = {
"type": "object",
"properties": {
"slots": {
"type": "array",
"items": {
"type": "object",
"properties": {
"date": {"type": "string", "description": "YYYY-MM-DD"},
"time": {"type": "string", "description": "24-hour HH:MM"},
"clinician": {"type": "string"},
"location": {"type": "string"},
},
"required": ["date", "time", "clinician", "location"],
},
}
},
"required": ["slots"],
}
BOOKING_SCHEMA = {
"type": "object",
"properties": {
"booked": {"type": "boolean"},
"confirmation": {"type": "string", "description": "The confirmation number the site shows, or ''."},
"date": {"type": "string"},
"time": {"type": "string"},
"clinician": {"type": "string"},
},
"required": ["booked", "confirmation", "date", "time", "clinician"],
}
async def push(title: str, message: str, click: str | None = None) -> None:
headers = {"Title": title, "Priority": "urgent", **({"Click": click} if click else {})}
async with httpx.AsyncClient() as http:
await http.post(NTFY, headers=headers, content=message)
hunting = False
hand_offs: dict[str, str] = {} # run id to the monitor that found its slot
async def hunt(monitor_id: str) -> None:
# 1. Read every open slot. Nothing is booked in this run.
found = await oc.agent(
task=f"Open {BOOKING_PAGE} and list every appointment slot that can be booked now, across the next {WITHIN_DAYS} days.",
schema=SLOT_SCHEMA,
profile=PROFILE,
caps={"budget": "40.00", "duration": "3m"},
rules=RULES,
)
slots = sorted((found.get("data") or {}).get("slots", []), key=lambda s: f"{s['date']}T{s['time']}")
# 2. Your code decides, not the agent.
slot = next((s for s in slots if acceptable(s)), None)
if slot is None:
print(f"{len(slots)} open slots, none within your rules")
return
when = f"{slot['date']} at {slot['time']} with {slot['clinician']}"
# 3a. Book mode: a second run books exactly the slot your code approved.
if MODE == "book":
run = await oc.agent(
task=f"""On {BOOKING_PAGE}, book the slot on {slot['date']} at {slot['time']} with {slot['clinician']} at {slot['location']}.
Use the details already saved on the account. Book that slot and no other; if it is gone, book nothing.""",
schema=BOOKING_SCHEMA,
profile=PROFILE,
caps={"budget": "65.00", "duration": "5m"},
rules=RULES,
)
booking = run.get("data") or {}
if booking.get("booked"):
await oc.monitors.delete(monitor_id) # done hunting
await push("Appointment booked", f"{when}. Confirmation {booking['confirmation']}.", run.get("replayUrl"))
else:
print(f"could not book {when}: {run.get('stopReason') or run['status']}", run.get("replayUrl"))
return
# 3b. Hand-off mode: the agent gets the slot in hand, and you confirm it yourself.
job = oc.agent(
task=f"""On {BOOKING_PAGE}, pick the slot on {slot['date']} at {slot['time']} with {slot['clinician']} at {slot['location']}
and go through the booking form with the details saved on the account. Do not press the final confirm button.""",
profile=PROFILE,
caps={"budget": "150.00", "duration": "10m"}, # time the run waits for you is billed; these caps bound it
rules=RULES,
)
run_id = await job.id()
hand_offs[run_id] = monitor_id
async for record in job:
if record["kind"] != "step":
continue
description = ((record["value"] or {}).get("action") or {}).get("description") or ""
if slot["time"] in description:
await job.control("take", "me") # holds at the agent's next step boundary
break
run = await job.status()
while run["status"] in ("queued", "running"):
await asyncio.sleep(2)
run = await job.status()
viewer = (run.get("pause") or {}).get("viewerUrl")
if viewer is None:
print(f"run ended before the hand-off: {run.get('stopReason')}")
return
await push(
f"Slot held: {when}",
f"Tap to open the live view, press take control, and confirm. When you are done: {PUBLIC_URL}/done/{TOKEN}/{run_id}",
viewer,
)
async def hunt_once(monitor_id: str) -> None:
global hunting
try:
await hunt(monitor_id)
finally:
hunting = False
@app.post(f"/hooks/slots/{TOKEN}", status_code=202)
async def slots_changed(request: Request, background: BackgroundTasks) -> dict:
# Answer at once: a monitor delivers each change once and never retries.
global hunting
change = await request.json()
diff = change.get("diff")
added = diff is None or re.search(r"^\+(?!\+\+)", diff, re.M) is not None # only removed lines: a slot going, not coming
if added and not hunting:
hunting = True
background.add_task(hunt_once, change["monitorId"])
return {"accepted": True}
# The link in the push: you have confirmed, so end the run, stop paying for it, and stop hunting.
@app.get(f"/done/{TOKEN}/{{run_id}}")
async def done(run_id: str) -> str:
await oc.agent.cancel(run_id)
if run_id in hand_offs:
await oc.monitors.delete(hand_offs.pop(run_id))
return "Run ended."
async def watch() -> None:
monitor = await oc.monitors.create(
url=BOOKING_PAGE,
label="slots-dermatology",
schedule="*/10 * * * *", # every 10 minutes
profile=PROFILE, # see the page the way you see it when signed in
notify={"webhook": f"{PUBLIC_URL}/hooks/slots/{TOKEN}"},
)
print("watching", monitor["id"])
await oc.aclose()
if __name__ == "__main__":
asyncio.run(watch()) # python slot_hunter.py, once; then serve with: SLOT_MODE=book uvicorn slot_hunter:app --port 3000
```
### The limits of each mode
* **Book mode** is two runs, so a slot can go in the seconds between them. The booking run is told to book that slot or nothing, and reports `booked: false` when it is gone; the next change starts a fresh hunt.
* **Hand-off mode** holds the run at the agent's next step boundary, not instantly. The step already under way finishes first. On a site where picking the slot and confirming are separate screens, that is one more field or a Continue button; on a site where one click books, it is the booking of a slot your rules already approved.
* If the site asks for a deposit, `Never make a payment.` blocks the agent at the card form in either mode. Use hand-off mode and pay it yourself.
## What you get back
In book mode, one push:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Appointment booked
2026-10-06 at 08:40 with Dr Osei. Confirmation RVS-7F3K2.
```
In hand-off mode, a push whose tap opens the live view:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Slot held: 2026-10-06 at 08:40 with Dr Osei
Tap to open the live view, press take control, and confirm. When you are done: https://yourapp.example/done//run_01JB2M6Q
```
The first run's answer, which your code filters:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"slots": [
{ "date": "2026-10-04", "time": "11:20", "clinician": "Dr Sun", "location": "Main building" },
{ "date": "2026-10-06", "time": "08:40", "clinician": "Dr Osei", "location": "Riverside" },
{ "date": "2026-10-09", "time": "13:00", "clinician": "Dr Osei", "location": "Riverside" }
]
}
```
The first slot falls on a Sunday and the third at lunchtime, so the program takes the 08:40 on Tuesday.
## What it costs
* **Watching** is 1 credit per check, whether or not the page changed, per [pricing](/docs/billing/pricing). A check every 10 minutes is 144 checks a day, so **144 credits a day**.
* **Listing slots** is one agent run, billed for the model's usage, browser time and proxy usage. A typical run costs about 15 credits a minute all in, and the real figure depends on the task and how heavy the pages are. About 40 s is **about 10 credits** each time new slots appear.
* **Booking** is about a minute: **about 15 credits**, once.
* **Hand-off** bills the agent's part the same way. While you hold the wheel, whether after `control('take')` or by driving in the live viewer, the run bills browser time (1 credit a minute) and proxy usage only, with no model usage, until you hand it back. Here the done link cancels the run, which stops the clock: five minutes from the push to your tap is about 5 credits, plus proxy usage for the pages you load. `caps.duration` ends the run after 10 minutes if you never come back.
A page where new slots appear five times a day costs about 144 + 5 × 10 = **194 credits a day** until the booking lands.
## Make it yours
* Watch several clinics at once: one monitor per booking page, all posting to the same endpoint, and delete every monitor once one books.
* Change `acceptable()` to match your week: a particular clinician, a location near you, or a date before you travel.
* Send the hand-off link to Slack or SMS instead of ntfy; it is one HTTP call.
* Check every 5 minutes instead of 10 on a page where slots go fast. That is 288 checks a day, so watching costs 288 credits a day instead of 144.
## Related
Schedules, notify targets and profiles.
Take and release control of a live run.
Stay signed in to the booking site.
Stop the agent paying for anything.
# Book a table the moment one opens
Source: https://outcrawl.ai/docs/examples/book-a-table-when-it-opens
A monitor watches a fully booked restaurant, and when a table appears your server sends an agent to book it for your party, inside your limits.
The restaurant you want is booked solid, and the tables that come back from cancellations go within minutes to whoever happens to be refreshing. This program watches the Resy page every minute. When it changes, your server sends an agent, signed in as you, to book the earliest time in your window for your party, with your card for the hold if Resy asks for one. You get a Slack message with the confirmation, and the watching stops.
Book tables you will sit at. The program books one table and deletes its monitor; it is not a way to hold tables for resale.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* A [profile](/docs/account/profiles) signed in to your Resy account, labelled `resy-personal`.
* Your card stored as a [secret](/docs/account/secrets) named `card_dining` with `kind: 'card'`. The agent types it by name when Resy asks for a card; the model never sees the number.
* A server reachable from the internet, with a long random string in `HOOK_TOKEN`. The monitor and the run both post to it.
* Optional: a Slack incoming webhook URL in `SLACK_WEBHOOK_URL`. Python also needs `fastapi` and `uvicorn`.
## How it works
[`monitors.create`](/docs/monitors/overview) checks the Resy page for your date and party size every minute. Each check is 1 credit, whether or not the page changed; a real change posts to your endpoint. A monitor cannot start an agent itself, so your endpoint does.
It submits one [agent run](/docs/agents/overview) on your profile, with your card secret, your window and party size in the task, and a [schema](/docs/agents/overview#typed-results) for the booking. The [rule](/docs/account/rules) `Only use resy.com.` is enforced. One run at a time: a change that arrives while a run is booking is ignored.
The run's [webhook](/docs/agents/monitoring#webhooks) posts to your second endpoint when it ends, and retries if your server is down. The endpoint reads the run by id rather than trusting the body, and checks the booked time against your window.
A booking goes to Slack and `monitors.delete` removes the monitor. A change with nothing that fits is logged, and the monitor keeps watching.
Your window, party size and deposit ceiling are instructions in the task. What is enforced is the site (`Only use resy.com.`), the card (the run can type only the one secret it is given), and your code's check of the booked time. A spend-limit rule is left out on purpose: it has to read a total on the page before a booking button and blocks the booking when it cannot, and most reservations that need no deposit show no total.
## The program
Run it as a server. It creates the monitor when it starts.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import express from 'express';
import { Outcrawl } from '@outcrawl/sdk';
import type { AgentJob } from '@outcrawl/sdk';
const oc = new Outcrawl();
const PUBLIC_URL = 'https://tables.yourapp.example'; // where Outcrawl can reach this server
const HOOKS = `/hooks/${process.env.HOOK_TOKEN ?? ''}`; // a long random token: the only credential on these paths
const SLACK_WEBHOOK_URL = process.env.SLACK_WEBHOOK_URL ?? ''; // a Slack incoming webhook of yours
if ((process.env.HOOK_TOKEN ?? '').length < 24) throw new Error('Set HOOK_TOKEN to a long random string.');
const want = {
restaurant: 'Lilia',
page: 'https://resy.com/cities/new-york-ny/venues/lilia?date=2026-10-17&seats=4',
date: '2026-10-17',
partySize: 4,
earliest: '19:00',
latest: '21:00',
maxDepositPerPerson: 25, // USD
};
interface Booking {
booked: boolean;
time?: string;
partySize?: number;
confirmation?: string;
depositPerPerson?: string;
cancellationPolicy?: string;
reason?: string;
}
const bookingSchema = {
type: 'object',
properties: {
booked: { type: 'boolean' },
time: { type: 'string', description: 'Reservation time, 24-hour HH:MM' },
partySize: { type: 'integer' },
confirmation: { type: 'string', description: 'Confirmation number or message' },
depositPerPerson: { type: 'string', description: 'Deposit or charge per person, with currency, or "none"' },
cancellationPolicy: { type: 'string' },
reason: { type: 'string', description: 'Why nothing was booked, when nothing was' },
},
required: ['booked'],
};
const [profile] = await oc.profiles.list({ label: 'resy-personal' });
if (profile === undefined) throw new Error('Create the resy-personal profile and sign in to Resy with it first.');
// Checked every minute. Each check is 1 credit, changed or not.
const monitor = await oc.monitors.create({
url: want.page,
schedule: '* * * * *',
label: 'table-lilia',
judge: true,
notify: { webhook: `${PUBLIC_URL}${HOOKS}/table-changed` },
});
let inFlight: AgentJob | undefined; // the run in flight, so two changes never book twice
let done = false;
async function tell(text: string): Promise {
console.log(text);
if (SLACK_WEBHOOK_URL !== '') {
await fetch(SLACK_WEBHOOK_URL, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ text }),
});
}
}
const app = express();
app.use(express.json());
// 1. The monitor saw the page change. Send an agent to book.
app.post(`${HOOKS}/table-changed`, async (req, res) => {
const change = req.body as { monitorId: string; status: string; summary?: string };
if (done || inFlight !== undefined || change.status !== 'changed') {
res.sendStatus(204);
return;
}
inFlight = oc.agent({
task:
`On Resy, book a table at ${want.restaurant} for ${want.partySize} people on ${want.date}, at the earliest ` +
`available time from ${want.earliest} to ${want.latest}. Start at ${want.page}; you are signed in. If Resy ` +
'asks for a card to hold the table, use {{secret:card_dining}}. Do not book if no time in that window is ' +
`free, or if the reservation needs a deposit or charge now above $${want.maxDepositPerPerson} per person: ` +
'report why instead.',
caps: { budget: '75.00', duration: '6m' },
profile: profile.id,
secrets: ['card_dining'],
rules: ['Only use resy.com.'],
schema: bookingSchema,
webhook: `${PUBLIC_URL}${HOOKS}/booking-finished`,
});
try {
res.status(202).json({ run: await inFlight.id() });
} catch (error) {
inFlight = undefined; // refused, for example every browser slot busy; the next change tries again
throw error;
}
});
// 2. The run finished. Check what it booked and tell you.
app.post(`${HOOKS}/booking-finished`, async (req, res) => {
const { runId } = req.body as { runId: string };
res.sendStatus(200);
if (inFlight === undefined || runId !== (await inFlight.id())) return; // a repeat delivery
inFlight = undefined;
const run = await oc.agent.get(runId); // read the run itself, not the webhook body
const booking = run.data as Booking | undefined;
if (booking?.booked !== true) {
console.log(`No table yet: ${booking?.reason ?? run.stopReason ?? run.status}`);
return; // keep watching
}
done = true;
await oc.monitors.delete(monitor.id);
const inWindow = booking.time !== undefined && booking.time >= want.earliest && booking.time <= want.latest;
await tell(
`Booked ${want.restaurant}, ${want.date} at ${booking.time ?? '?'} for ${booking.partySize ?? want.partySize}. ` +
`Confirmation ${booking.confirmation ?? 'on Resy'}. Deposit: ${booking.depositPerPerson ?? 'none'}. ` +
`${booking.cancellationPolicy ?? ''}${inWindow ? '' : ' This is OUTSIDE your window: check it on Resy.'}`,
);
});
app.listen(3000, () => console.log(`Watching ${want.page} as monitor ${monitor.id}`));
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import os
from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, Request, Response
from fastapi.responses import JSONResponse
from outcrawl import Outcrawl
PUBLIC_URL = "https://tables.yourapp.example" # where Outcrawl can reach this server
HOOK_TOKEN = os.environ.get("HOOK_TOKEN", "") # a long random token: the only credential on these paths
if len(HOOK_TOKEN) < 24:
raise SystemExit("Set HOOK_TOKEN to a long random string.")
HOOKS = f"/hooks/{HOOK_TOKEN}"
SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL", "") # a Slack incoming webhook of yours
WANT = {
"restaurant": "Lilia",
"page": "https://resy.com/cities/new-york-ny/venues/lilia?date=2026-10-17&seats=4",
"date": "2026-10-17",
"party_size": 4,
"earliest": "19:00",
"latest": "21:00",
"max_deposit_per_person": 25, # USD
}
BOOKING_SCHEMA = {
"type": "object",
"properties": {
"booked": {"type": "boolean"},
"time": {"type": "string", "description": "Reservation time, 24-hour HH:MM"},
"partySize": {"type": "integer"},
"confirmation": {"type": "string", "description": "Confirmation number or message"},
"depositPerPerson": {"type": "string", "description": 'Deposit or charge per person, with currency, or "none"'},
"cancellationPolicy": {"type": "string"},
"reason": {"type": "string", "description": "Why nothing was booked, when nothing was"},
},
"required": ["booked"],
}
oc = Outcrawl() # one client for the life of the server
state = {"profile": None, "monitor": None, "in_flight": None, "done": False}
@asynccontextmanager
async def lifespan(app: FastAPI):
profiles = await oc.profiles.list(label="resy-personal")
if not profiles:
raise SystemExit("Create the resy-personal profile and sign in to Resy with it first.")
state["profile"] = profiles[0]["id"]
# Checked every minute. Each check is 1 credit, changed or not.
monitor = await oc.monitors.create(
url=WANT["page"],
schedule="* * * * *",
label="table-lilia",
judge=True,
notify={"webhook": f"{PUBLIC_URL}{HOOKS}/table-changed"},
)
state["monitor"] = monitor["id"]
print(f"Watching {WANT['page']} as monitor {monitor['id']}")
yield
await oc.aclose()
app = FastAPI(lifespan=lifespan)
async def tell(text: str) -> None:
print(text)
if SLACK_WEBHOOK_URL:
async with httpx.AsyncClient() as http:
await http.post(SLACK_WEBHOOK_URL, json={"text": text})
# 1. The monitor saw the page change. Send an agent to book.
@app.post(f"{HOOKS}/table-changed")
async def table_changed(request: Request) -> Response:
change = await request.json()
if state["done"] or state["in_flight"] or change.get("status") != "changed":
return Response(status_code=204)
state["in_flight"] = "submitting" # so two changes never book twice
try:
job = oc.agent(
task=(
f"On Resy, book a table at {WANT['restaurant']} for {WANT['party_size']} people on {WANT['date']}, "
f"at the earliest available time from {WANT['earliest']} to {WANT['latest']}. Start at "
f"{WANT['page']}; you are signed in. If Resy asks for a card to hold the table, use "
"{{secret:card_dining}}. Do not book if no time in that window is free, or if the reservation needs "
f"a deposit or charge now above ${WANT['max_deposit_per_person']} per person: report why instead."
),
caps={"budget": "75.00", "duration": "6m"},
profile=state["profile"],
secrets=["card_dining"],
rules=["Only use resy.com."],
schema=BOOKING_SCHEMA,
webhook=f"{PUBLIC_URL}{HOOKS}/booking-finished",
)
state["in_flight"] = await job.id()
except Exception:
state["in_flight"] = None # refused, for example every browser slot busy; the next change tries again
raise
return JSONResponse({"run": state["in_flight"]}, status_code=202)
# 2. The run finished. Check what it booked and tell you.
@app.post(f"{HOOKS}/booking-finished")
async def booking_finished(request: Request) -> Response:
run_id = (await request.json())["runId"]
if run_id != state["in_flight"]:
return Response(status_code=200) # a repeat delivery of a run already handled
state["in_flight"] = None
run = await oc.agent.get(run_id) # read the run itself, not the webhook body
booking = run.get("data") or {}
if not booking.get("booked"):
print(f"No table yet: {booking.get('reason') or run.get('stopReason') or run['status']}")
return Response(status_code=200) # keep watching
state["done"] = True
await oc.monitors.delete(state["monitor"])
time = booking.get("time", "?")
in_window = WANT["earliest"] <= time <= WANT["latest"]
await tell(
f"Booked {WANT['restaurant']}, {WANT['date']} at {time} for {booking.get('partySize', WANT['party_size'])}. "
f"Confirmation {booking.get('confirmation', 'on Resy')}. Deposit: {booking.get('depositPerPerson', 'none')}. "
f"{booking.get('cancellationPolicy', '')}{'' if in_window else ' This is OUTSIDE your window: check it on Resy.'}"
)
return Response(status_code=200)
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, port=3000)
```
## What you get back
In Slack:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Booked Lilia, 2026-10-17 at 19:45 for 4. Confirmation RSY-88213. Deposit: none. $25 per person if cancelled within 24 hours of the reservation.
```
The run's answer, as your endpoint reads it with `oc.agent.get`:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"booked": true,
"time": "19:45",
"partySize": 4,
"confirmation": "RSY-88213",
"depositPerPerson": "none",
"cancellationPolicy": "$25 per person if cancelled within 24 hours of the reservation."
}
```
When the page changed but nothing fit, the answer is `{ "booked": false, "reason": "The only new time is 22:15, outside 19:00 to 21:00." }` and the monitor keeps going.
## What it costs
* **Watching** is 1 credit per check, whether or not the page changed, plus the model's usage when a changed page is judged (`judge: true`), per [pricing](/docs/billing/pricing). A check every minute is 1,440 checks a day, so **1,440 credits a day**.
* **A change with nothing that fits** starts a short run. An agent run is billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in; the real figure depends on the task and how heavy the pages are. About 30 s is **about 8 credits**.
* **The booking** is about 2 minutes: **about 30 credits**, once.
A popular page also changes when other people take times. A week of watching with 30 changes before a table fits is 10,080 × 1 + 29 × 8 + 30 = **about 10,340 credits**, plus model usage for the 30 judged changes. Each run's `caps.budget` is 75. Checking every 5 minutes instead cuts the watching to 2,016 credits a week.
## Make it yours
* Check the deposit in code before anything is booked: split the run into one that lists open times with their deposits and one that books the time your code picks, as [Book the first appointment slot that fits your week](/docs/examples/appointment-slot-hunter) does. It costs about a minute more, and tables go fast.
* Skip changes that only remove times: `judge: true` puts a model's `summary` of the change in the monitor's post, and your endpoint can start a run only when it mentions a new time.
* Watch several dates: one monitor per date's page, all posting to the same endpoint, and delete them all once one books.
* Text yourself instead of Slack: `tell` is one HTTP call.
## Related
Schedules and notify targets.
Run webhooks and their retries.
A card the agent types but never reads.
Stay signed in to Resy.
# Cancel every subscription you forgot about
Source: https://outcrawl.ai/docs/examples/cancel-forgotten-subscriptions
One agent per service walks the cancellation flow, declines every offer, and stops at the last button until you approve.
Cancelling a subscription is slow on purpose: a survey, a discount, a pause offer, a second discount, then the real button at the bottom of the page. Four of them take an evening. This program sends one agent per service, each signed in through its own profile, to walk every cancellation flow as far as the last button and stop there. You read what each one found and pick which to cancel, and a second run presses the button for those, and only those.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* One [profile](/docs/account/profiles) per service, signed in to that site and labelled `subs-` (`subs-nytimes`, `subs-audible`...). Sign each one in once: from TypeScript with `oc.browser({ profile })`, or with an agent run that types your password from a [text secret](/docs/account/secrets). If a site emails a sign-in code, that run reads it through a connected inbox [integration](/docs/account/integrations), such as AgentMail.
* A terminal. The program asks you which services to cancel between its two passes.
## How it works
Every run names its service's [profile](/docs/account/profiles), so it starts signed in. A profile opens one browser at a time, so a service's two runs never overlap, while different services run side by side, `MAX_AT_ONCE` at a time. Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot. A submit over the limit is refused with `ConcurrencyLimitError`, and the program waits 20 seconds and tries again.
An [agent run](/docs/agents/overview) answers the survey, declines every offer, and stops on the final screen without pressing its button. Its [schema](/docs/agents/overview#typed-results) brings back the plan, the price, the renewal date, every screen it passed, the offers it declined, any fee to cancel, and the exact label of the final button. The [rule](/docs/account/rules) `Only use nytimes.com.` is enforced: the agent cannot go to any other site.
Nothing has been cancelled yet. The program prints what each run found and asks which to cancel. No run is waiting on you: the approval sits between two runs.
A fresh run on the same profile follows the path pass 1 found, declines the offers again, and presses the final button, unless the fee on that screen differs from the one you saw. It returns the confirmation and the last day of access, or exactly what is left for you to do.
The stop at the last screen is in the task, not in a rule. Rules can block a payment, an order, a message or a sign-up, but Outcrawl's irreversible check reads a button that starts with "Cancel" as a way out of a page, so no rule can block a cancellation. That is why nothing rests on pass 1: the only run told to press the final button is the one you approved. Read `path` in pass 1's answer before you say yes.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { writeFile } from 'node:fs/promises';
import { stdin, stdout } from 'node:process';
import { createInterface } from 'node:readline/promises';
import { setTimeout as sleep } from 'node:timers/promises';
import { ConcurrencyLimitError, Outcrawl, QuotaExceededError } from '@outcrawl/sdk';
import type { AgentRequest } from '@outcrawl/sdk';
const oc = new Outcrawl();
// One row per subscription. Each profile is already signed in to that site.
const services = [
{ name: 'The New York Times', site: 'nytimes.com', start: 'https://myaccount.nytimes.com/seg/subscription', profile: 'subs-nytimes' },
{ name: 'Audible', site: 'audible.com', start: 'https://www.audible.com/account/overview', profile: 'subs-audible' },
{ name: 'Adobe Creative Cloud', site: 'adobe.com', start: 'https://account.adobe.com/plans', profile: 'subs-adobe' },
{ name: 'Headspace', site: 'headspace.com', start: 'https://www.headspace.com/subscriptions', profile: 'subs-headspace' },
];
// Runs at once. Keep it below your plan's browser count (Free 2, Builder 5, Pro 25, Scale 100).
const MAX_AT_ONCE = 2;
interface Walkthrough {
active: boolean;
plan?: string;
price?: string;
renewsOn?: string;
path: string[];
offersDeclined?: string[];
finalButton?: string;
feeToCancel?: string;
blocker?: string;
}
interface Cancellation {
cancelled: boolean;
accessEndsOn?: string;
confirmation?: string;
leftForYou?: string;
}
const walkthroughSchema = {
type: 'object',
properties: {
active: { type: 'boolean', description: 'Whether this account has an active paid subscription' },
plan: { type: 'string' },
price: { type: 'string', description: 'Price per billing period, with currency' },
renewsOn: { type: 'string', description: 'Next renewal date, YYYY-MM-DD' },
path: { type: 'array', items: { type: 'string' }, description: 'Each screen passed, in order' },
offersDeclined: { type: 'array', items: { type: 'string' } },
finalButton: { type: 'string', description: 'Exact label of the button that ends the subscription' },
feeToCancel: { type: 'string', description: 'Any fee for cancelling now, with currency, or "none"' },
blocker: { type: 'string', description: 'What stopped you before the final screen, if anything' },
},
required: ['active', 'path'],
};
const cancellationSchema = {
type: 'object',
properties: {
cancelled: { type: 'boolean' },
accessEndsOn: { type: 'string', description: 'Last day of access, YYYY-MM-DD' },
confirmation: { type: 'string', description: 'Confirmation message or number shown' },
leftForYou: { type: 'string', description: 'What a person still has to do, if not cancelled' },
},
required: ['cancelled'],
};
// Submit a run and answer its id, waiting for a free browser slot instead of failing.
async function start(request: AgentRequest): Promise {
for (;;) {
try {
return (await oc.agent(request).submitted).id;
} catch (error) {
if (error instanceof ConcurrencyLimitError) {
await sleep(20_000); // no Retry-After: a slot frees when one of your runs ends
} else if (error instanceof QuotaExceededError && error.ceiling === 'requests_per_minute') {
await sleep(error.retryAfterSeconds * 1000); // the error says how long to wait
} else {
throw error; // out of monthly credits, or a bad request: retrying cannot help
}
}
}
}
// Run `work` over every item, at most `size` at a time.
async function inPool(items: readonly T[], size: number, work: (item: T) => Promise): Promise {
const results: R[] = new Array(items.length);
let next = 0;
const worker = async () => {
while (next < items.length) {
const index = next++;
results[index] = await work(items[index] as T);
}
};
await Promise.all(Array.from({ length: Math.min(size, items.length) }, worker));
return results;
}
const profiles = await oc.profiles.list({ label: 'subs-*' });
const profileIds = new Map(profiles.map((profile) => [profile.label, profile.id]));
for (const service of services) {
if (!profileIds.has(service.profile)) throw new Error(`No profile labelled ${service.profile}`);
}
// Pass 1: walk each cancellation flow up to its last screen, and stop there.
const walkthroughs = await inPool(services, MAX_AT_ONCE, async (service) => {
const run = await oc.agent.job(await start({
task:
`You are signed in to ${service.name}. Starting at ${service.start}, find how to cancel the subscription ` +
'on this account and work through the cancellation flow. Answer any survey with "I no longer use it" and ' +
'decline every retention offer, discount, downgrade and pause. Stop on the final screen, the one whose ' +
'button would end the subscription, and do not press that button. If there is no active subscription, say so.',
caps: { budget: '150.00', duration: '10m' },
profile: profileIds.get(service.profile),
rules: [`Only use ${service.site}.`],
schema: walkthroughSchema,
})).settled();
return { service, walkthrough: run.data as Walkthrough | undefined, credits: run.credits };
});
walkthroughs.forEach(({ service, walkthrough }, index) => {
if (walkthrough === undefined) {
console.log(`${index + 1}. ${service.name}: no answer, cancel this one yourself`);
} else if (!walkthrough.active) {
console.log(`${index + 1}. ${service.name}: no active subscription`);
} else {
console.log(
`${index + 1}. ${service.name}: ${walkthrough.plan ?? '?'} at ${walkthrough.price ?? '?'}, renews ${walkthrough.renewsOn ?? '?'}. ` +
(walkthrough.finalButton !== undefined
? `Fee to cancel: ${walkthrough.feeToCancel ?? 'none'}. Ready to press "${walkthrough.finalButton}".`
: `Stopped early: ${walkthrough.blocker ?? 'no reason given'}`),
);
}
});
// You decide. Nothing has been cancelled yet.
const prompt = createInterface({ input: stdin, output: stdout });
const answer = await prompt.question('Cancel which? Numbers separated by commas, "all", or Enter for none: ');
prompt.close();
const ready = walkthroughs.filter(({ walkthrough }) => walkthrough?.active === true && walkthrough.finalButton !== undefined);
const approved =
answer.trim() === 'all'
? ready
: answer
.split(',')
.map((part) => walkthroughs[Number(part.trim()) - 1])
.filter((row): row is (typeof walkthroughs)[number] => row !== undefined && ready.includes(row));
// Pass 2: only the ones you approved, following the path pass 1 found.
const cancellations = await inPool(approved, MAX_AT_ONCE, async ({ service, walkthrough }) => {
const steps = walkthrough as Walkthrough;
const run = await oc.agent.job(await start({
task:
`You are signed in to ${service.name}. I have approved cancelling this subscription. Last time you reached ` +
`the final screen this way: ${steps.path.join(' > ')}. Decline every retention offer again, then press ` +
`"${steps.finalButton}" on the final screen. The fee to cancel I approved is: ${steps.feeToCancel ?? 'none'}. ` +
'If the final screen shows a different fee, do not press it. Report the confirmation. If anything stops you (a sign-in, a ' +
'cancellation that must be done by phone or chat, a page that changed), stop and say exactly what is left for me to do.',
caps: { budget: '100.00', duration: '10m' },
profile: profileIds.get(service.profile),
rules: [`Only use ${service.site}.`],
schema: cancellationSchema,
})).settled();
const result = run.data as Cancellation | undefined;
return { service: service.name, ...(result ?? { cancelled: false, leftForYou: `Run ended ${run.stopReason ?? run.status}; see ${run.replayUrl ?? 'the console'}` }) };
});
console.table(cancellations);
await writeFile('cancellations.json', JSON.stringify({ walkthroughs, cancellations }, null, 2));
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import json
from outcrawl import ConcurrencyLimitError, Outcrawl, QuotaExceededError
# One row per subscription. Each profile is already signed in to that site.
SERVICES = [
{"name": "The New York Times", "site": "nytimes.com", "start": "https://myaccount.nytimes.com/seg/subscription", "profile": "subs-nytimes"},
{"name": "Audible", "site": "audible.com", "start": "https://www.audible.com/account/overview", "profile": "subs-audible"},
{"name": "Adobe Creative Cloud", "site": "adobe.com", "start": "https://account.adobe.com/plans", "profile": "subs-adobe"},
{"name": "Headspace", "site": "headspace.com", "start": "https://www.headspace.com/subscriptions", "profile": "subs-headspace"},
]
# Runs at once. Keep it below your plan's browser count (Free 2, Builder 5, Pro 25, Scale 100).
MAX_AT_ONCE = 2
WALKTHROUGH_SCHEMA = {
"type": "object",
"properties": {
"active": {"type": "boolean", "description": "Whether this account has an active paid subscription"},
"plan": {"type": "string"},
"price": {"type": "string", "description": "Price per billing period, with currency"},
"renewsOn": {"type": "string", "description": "Next renewal date, YYYY-MM-DD"},
"path": {"type": "array", "items": {"type": "string"}, "description": "Each screen passed, in order"},
"offersDeclined": {"type": "array", "items": {"type": "string"}},
"finalButton": {"type": "string", "description": "Exact label of the button that ends the subscription"},
"feeToCancel": {"type": "string", "description": 'Any fee for cancelling now, with currency, or "none"'},
"blocker": {"type": "string", "description": "What stopped you before the final screen, if anything"},
},
"required": ["active", "path"],
}
CANCELLATION_SCHEMA = {
"type": "object",
"properties": {
"cancelled": {"type": "boolean"},
"accessEndsOn": {"type": "string", "description": "Last day of access, YYYY-MM-DD"},
"confirmation": {"type": "string", "description": "Confirmation message or number shown"},
"leftForYou": {"type": "string", "description": "What a person still has to do, if not cancelled"},
},
"required": ["cancelled"],
}
async def start(oc: Outcrawl, **request) -> str:
"""Submit a run and answer its id, waiting for a free browser slot instead of failing."""
while True:
try:
return (await oc.agent(**request).submitted)["id"]
except ConcurrencyLimitError:
await asyncio.sleep(20) # no Retry-After: a slot frees when one of your runs ends
except QuotaExceededError as error:
if error.ceiling != "requests_per_minute":
raise # out of monthly credits: retrying cannot help
await asyncio.sleep(error.retry_after_seconds) # the error says how long to wait
async def main() -> None:
async with Outcrawl() as oc:
profiles = await oc.profiles.list(label="subs-*")
profile_ids = {profile["label"]: profile["id"] for profile in profiles}
for service in SERVICES:
if service["profile"] not in profile_ids:
raise SystemExit(f"No profile labelled {service['profile']}")
slots = asyncio.Semaphore(MAX_AT_ONCE)
# Pass 1: walk each cancellation flow up to its last screen, and stop there.
async def walk(service: dict) -> dict:
async with slots:
run_id = await start(
oc,
task=(
f"You are signed in to {service['name']}. Starting at {service['start']}, find how to cancel "
"the subscription on this account and work through the cancellation flow. Answer any survey "
'with "I no longer use it" and decline every retention offer, discount, downgrade and pause. '
"Stop on the final screen, the one whose button would end the subscription, and do not press "
"that button. If there is no active subscription, say so."
),
caps={"budget": "150.00", "duration": "10m"},
profile=profile_ids[service["profile"]],
rules=[f"Only use {service['site']}."],
schema=WALKTHROUGH_SCHEMA,
)
run = await oc.agent.job(run_id).settled()
return {"service": service, "walkthrough": run.get("data"), "credits": run["credits"]}
walkthroughs = await asyncio.gather(*(walk(service) for service in SERVICES))
for number, row in enumerate(walkthroughs, start=1):
name, steps = row["service"]["name"], row["walkthrough"]
if steps is None:
print(f"{number}. {name}: no answer, cancel this one yourself")
elif not steps["active"]:
print(f"{number}. {name}: no active subscription")
elif steps.get("finalButton"):
print(f"{number}. {name}: {steps.get('plan')} at {steps.get('price')}, renews {steps.get('renewsOn')}. "
f"Fee to cancel: {steps.get('feeToCancel', 'none')}. Ready to press \"{steps['finalButton']}\".")
else:
print(f"{number}. {name}: stopped early: {steps.get('blocker', 'no reason given')}")
# You decide. Nothing has been cancelled yet.
answer = await asyncio.to_thread(input, 'Cancel which? Numbers separated by commas, "all", or Enter for none: ')
ready = [row for row in walkthroughs if row["walkthrough"] and row["walkthrough"]["active"] and row["walkthrough"].get("finalButton")]
if answer.strip() == "all":
approved = ready
else:
picked = {int(part) - 1 for part in answer.split(",") if part.strip().isdigit()}
approved = [row for index, row in enumerate(walkthroughs) if index in picked and row in ready]
# Pass 2: only the ones you approved, following the path pass 1 found.
async def cancel(row: dict) -> dict:
service, steps = row["service"], row["walkthrough"]
async with slots:
run_id = await start(
oc,
task=(
f"You are signed in to {service['name']}. I have approved cancelling this subscription. Last "
f"time you reached the final screen this way: {' > '.join(steps['path'])}. Decline every "
f'retention offer again, then press "{steps["finalButton"]}" on the final screen. The fee to '
f"cancel I approved is: {steps.get('feeToCancel', 'none')}. If the final screen shows a different "
"fee, do not press it. Report the confirmation. If anything stops you (a sign-in, a cancellation "
"that must be done by phone or chat, a page that changed), stop and say exactly what is left "
"for me to do."
),
caps={"budget": "100.00", "duration": "10m"},
profile=profile_ids[service["profile"]],
rules=[f"Only use {service['site']}."],
schema=CANCELLATION_SCHEMA,
)
run = await oc.agent.job(run_id).settled()
result = run.get("data") or {
"cancelled": False,
"leftForYou": f"Run ended {run.get('stopReason') or run['status']}; see {run.get('replayUrl') or 'the console'}",
}
return {"service": service["name"], **result}
cancellations = await asyncio.gather(*(cancel(row) for row in approved))
for row in cancellations:
print(row)
with open("cancellations.json", "w") as out:
json.dump({"walkthroughs": walkthroughs, "cancellations": cancellations}, out, indent=2)
asyncio.run(main())
```
## What you get back
Pass 1, and your answer:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
1. The New York Times: All Access at $25.00 every 4 weeks, renews 2026-10-08. Fee to cancel: none. Ready to press "Complete cancellation".
2. Audible: Premium Plus at $14.95 a month, renews 2026-10-02. Fee to cancel: none. Ready to press "Cancel membership".
3. Adobe Creative Cloud: Photography (20GB) at US$11.99 a month, renews 2026-10-14. Fee to cancel: none. Ready to press "Confirm".
4. Headspace: no active subscription
Cancel which? Numbers separated by commas, "all", or Enter for none: 1,2,3
```
Pass 2, also written to `cancellations.json` with pass 1's walkthroughs:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
[
{
"service": "The New York Times",
"cancelled": true,
"accessEndsOn": "2026-10-07",
"confirmation": "Your subscription has been canceled. You will have access until October 7, 2026."
},
{
"service": "Audible",
"cancelled": false,
"leftForYou": "The final screen asked for a verification code sent to your email. Sign in on the subs-audible profile again, or give the run an inbox integration so it can read the code."
},
{
"service": "Adobe Creative Cloud",
"cancelled": true,
"accessEndsOn": "2026-10-13",
"confirmation": "Your plan has been cancelled. Order number 7710482213."
}
]
```
## What it costs
An agent run is billed for the model's usage, browser time and proxy usage, per [pricing](/docs/billing/pricing). A typical run costs about 15 credits a minute all in, proxy usage included. That is an estimate: the real figure depends on the task, and a page-heavy site costs more.
* **Pass 1** takes about 2.5 minutes per service, so **about 38 credits** a service.
* **Pass 2** takes about 1.5 minutes, **about 23 credits** for each service you approve.
Four services with three approved: 4 × 38 + 3 × 23 = **about 221 credits**, once. Each run's `caps.budget` is 150 in pass 1 and 100 in pass 2.
## Make it yours
* Watch pass 2 as it happens: log the `watchUrl` from `job.submitted` and open it; press take control in the live viewer to step in at any point, then hand it back. While you hold the wheel, the run bills browser time (1 credit a minute) and proxy usage only, with no model usage.
* Let a run sign back in when a session has lapsed: give it your password as a [secret](/docs/account/secrets) and an inbox [integration](/docs/account/integrations) for the emailed code.
* Run pass 1 alone each month as an audit: every plan, price and renewal date you pay, with nothing cancelled.
* Post the pass 2 results to Slack with one HTTP call to an incoming webhook.
## Related
Submit a run, caps and typed results.
Stay signed in to each service.
Which rules are enforced, and which are guidance.
Take and release control of a live run.
# Get a brief in Slack when a competitor changes its pricing
Source: https://outcrawl.ai/docs/examples/competitor-pricing-brief
Monitors watch competitors' pricing pages; on a real change an agent reads the page, compares it with the old plans and posts what it means.
A competitor moves a seat limit from the Team plan to Business on a Tuesday afternoon, and your sales team finds out from a prospect three weeks later. Checking pricing pages by hand is the job everyone agrees matters and nobody does. This program watches each pricing page, and when one really changes, an agent reads the new page and its fine print, lines it up against the plans as they were, and posts a brief to Slack: what changed, plan by plan, and what it means for your own pricing.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* A Slack [incoming webhook](https://api.slack.com/messaging/webhooks) URL in `SLACK_WEBHOOK_URL`. Posting to it is plain HTTP from your code, so no Outcrawl integration is involved.
* A server reachable from the internet (the monitors post to it), and a long random string in `HOOK_TOKEN` for its path.
* Node with `express`, or Python with `fastapi`, `uvicorn` and `httpx`.
## How it works
[`monitors.create`](/docs/monitors/overview) checks each page every 30 minutes. Timestamps, nonces and ads are stripped before comparing, `ignoreSelectors` drops the rotating testimonials, and each check is 1 credit whether or not anything changed.
The monitor sends `monitorId`, `url`, `label`, `checkedAt`, the `diff` and, with `judge: true`, a one-line `summary`. It sends once and never retries, so the endpoint answers `202` at once and does the work afterwards.
A monitor cannot start an agent itself; your code calls [`oc.agent`](/docs/agents/overview) with the diff and the plans recorded after the last change. [Rules](/docs/account/rules) keep the run on that competitor's site and stop it signing up for anything.
It reads the live page and the comparison table, FAQ and fine print it links to, and answers in a [schema](/docs/agents/overview#typed-results): every plan with prices and limits, each change with before and after, what it means, and one suggested response.
The brief goes to Slack as Block Kit, with the run's [replay](/docs/browsers/sessions) so anyone on your team can check what the agent read. The plan table is saved as the baseline for the next change.
## The program
Run it once with `watch` to create the monitors, then run it as a server.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import express from 'express';
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY
const PUBLIC_URL = 'https://yourapp.example'; // where this server is reachable
const HOOK = `/hooks/pricing/${process.env.HOOK_TOKEN}`; // the path is the credential: long and random
const SLACK_WEBHOOK_URL = process.env.SLACK_WEBHOOK_URL ?? ''; // a Slack incoming webhook
const OUR_PRICING = 'Team $29 per seat per month, Business $59 per seat per month, annual billing saves 20%.';
const COMPETITORS = [
{ label: 'pricing-northwind', url: 'https://northwind.example/pricing' },
{ label: 'pricing-fabrikam', url: 'https://fabrikam.example/pricing' },
{ label: 'pricing-tailspin', url: 'https://www.tailspin.example/plans' },
];
interface Plan { name: string; monthly: string; annual: string; limits: string[] }
interface Brief {
headline: string;
changes: { plan: string; what: string; before: string; after: string }[];
plans: Plan[];
whatItMeans: string;
suggestedResponse: string;
}
// What a monitor POSTs to your endpoint.
interface Change {
monitorId: string;
url: string;
label?: string;
status: 'changed' | 'removed';
checkedAt: string;
previousCheckedAt?: string;
diff?: string;
summary?: string;
}
const briefSchema = {
type: 'object',
properties: {
headline: { type: 'string', description: 'One sentence a salesperson can repeat.' },
changes: {
type: 'array',
items: {
type: 'object',
properties: { plan: { type: 'string' }, what: { type: 'string' }, before: { type: 'string' }, after: { type: 'string' } },
required: ['plan', 'what', 'before', 'after'],
},
},
plans: {
type: 'array',
items: {
type: 'object',
properties: {
name: { type: 'string' },
monthly: { type: 'string' },
annual: { type: 'string' },
limits: { type: 'array', items: { type: 'string' } },
},
required: ['name', 'monthly', 'annual', 'limits'],
},
},
whatItMeans: { type: 'string', description: 'Two or three sentences on what this means for us.' },
suggestedResponse: { type: 'string', description: 'One concrete thing we could do this week.' },
},
required: ['headline', 'changes', 'plans', 'whatItMeans', 'suggestedResponse'],
};
async function slack(text: string, blocks?: unknown[]): Promise {
await fetch(SLACK_WEBHOOK_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text, blocks }),
});
}
async function brief(change: Change): Promise {
const key = change.label ?? change.monitorId;
const file = `./pricing-state/${key}.json`;
const before = await readFile(file, 'utf8').catch(() => 'none on record yet');
const run = await oc.agent({
task: `The pricing page ${change.url} changed at ${change.checkedAt}.
The plans as we last recorded them: ${before}
The change the monitor saw, as a diff of the page (lines starting "-" were removed, "+" were added):
${(change.diff ?? 'no diff available').slice(0, 8000)}
Read the page as it is now, including any plan comparison table, FAQ and fine print it links to on the same site.
List every plan with its monthly and annual price and its limits. List what changed against the old plans.
Then explain what the change means for a company that sells: ${OUR_PRICING}`,
schema: briefSchema,
caps: { budget: '100.00', duration: '5m' },
rules: [`Only use ${new URL(change.url).hostname.replace(/^www\./, '')}.`, 'Never sign up for an account.'],
});
const report = run.data as Brief | undefined;
if (run.status !== 'completed' || report === undefined) {
await slack(`${key} changed, but the brief did not finish (${run.stopReason ?? run.status}). Replay: ${run.replayUrl ?? 'none'}`);
return;
}
await mkdir('./pricing-state', { recursive: true });
await writeFile(file, JSON.stringify(report.plans));
const changes = report.changes.map((c) => `• *${c.plan}*: ${c.what} (was ${c.before}, now ${c.after})`).join('\n');
await slack(report.headline, [
{ type: 'header', text: { type: 'plain_text', text: report.headline.slice(0, 150) } },
{ type: 'section', text: { type: 'mrkdwn', text: changes || 'No plan-level change found.' } },
{ type: 'section', text: { type: 'mrkdwn', text: `*What it means*\n${report.whatItMeans}` } },
{ type: 'section', text: { type: 'mrkdwn', text: `*Suggested response*\n${report.suggestedResponse}` } },
{
type: 'context',
elements: [{ type: 'mrkdwn', text: `<${change.url}|${key}> · ${run.replayUrl ? `<${run.replayUrl}|replay> · ` : ''}${run.credits} credits` }],
},
]);
}
if (process.argv[2] === 'watch') {
// One-off setup: a monitor per pricing page, skipping any that already exist.
const existing = await oc.monitors.list({ label: 'pricing-*' });
for (const { label, url } of COMPETITORS) {
if (existing.some((m) => m.url === url)) continue;
const monitor = await oc.monitors.create({
url,
label,
schedule: '*/30 * * * *', // every 30 minutes
judge: true, // a model summarises the diff into `summary`
normalise: { ignoreSelectors: ['[class*="testimonial"]', '[class*="carousel"]'] },
notify: { webhook: `${PUBLIC_URL}${HOOK}` },
});
console.log('watching', url, monitor.id);
}
} else {
const app = express();
app.use(express.json({ limit: '2mb' }));
app.post(HOOK, (req, res) => {
res.sendStatus(202); // answer at once: a monitor delivers each change once and never retries
brief(req.body as Change).catch((err) => console.error('brief failed', err));
});
app.listen(3000, () => console.log('listening on :3000'));
}
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import json
import os
from pathlib import Path
from urllib.parse import urlparse
import httpx
from fastapi import BackgroundTasks, FastAPI, Request
from outcrawl import Outcrawl
oc = Outcrawl() # one client for the life of the server; reads OUTCRAWL_API_KEY
app = FastAPI()
PUBLIC_URL = "https://yourapp.example" # where this server is reachable
HOOK = f"/hooks/pricing/{os.environ.get('HOOK_TOKEN', '')}" # the path is the credential: long and random
SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL", "") # a Slack incoming webhook
OUR_PRICING = "Team $29 per seat per month, Business $59 per seat per month, annual billing saves 20%."
STATE = Path("pricing-state")
COMPETITORS = [
{"label": "pricing-northwind", "url": "https://northwind.example/pricing"},
{"label": "pricing-fabrikam", "url": "https://fabrikam.example/pricing"},
{"label": "pricing-tailspin", "url": "https://www.tailspin.example/plans"},
]
BRIEF_SCHEMA = {
"type": "object",
"properties": {
"headline": {"type": "string", "description": "One sentence a salesperson can repeat."},
"changes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"plan": {"type": "string"},
"what": {"type": "string"},
"before": {"type": "string"},
"after": {"type": "string"},
},
"required": ["plan", "what", "before", "after"],
},
},
"plans": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"monthly": {"type": "string"},
"annual": {"type": "string"},
"limits": {"type": "array", "items": {"type": "string"}},
},
"required": ["name", "monthly", "annual", "limits"],
},
},
"whatItMeans": {"type": "string", "description": "Two or three sentences on what this means for us."},
"suggestedResponse": {"type": "string", "description": "One concrete thing we could do this week."},
},
"required": ["headline", "changes", "plans", "whatItMeans", "suggestedResponse"],
}
async def slack(text: str, blocks: list | None = None) -> None:
async with httpx.AsyncClient() as http:
await http.post(SLACK_WEBHOOK_URL, json={"text": text, "blocks": blocks})
async def brief(change: dict) -> None:
key = change.get("label") or change["monitorId"]
file = STATE / f"{key}.json"
before = file.read_text() if file.exists() else "none on record yet"
diff = (change.get("diff") or "no diff available")[:8000]
run = await oc.agent(
task=f"""The pricing page {change['url']} changed at {change['checkedAt']}.
The plans as we last recorded them: {before}
The change the monitor saw, as a diff of the page (lines starting "-" were removed, "+" were added):
{diff}
Read the page as it is now, including any plan comparison table, FAQ and fine print it links to on the same site.
List every plan with its monthly and annual price and its limits. List what changed against the old plans.
Then explain what the change means for a company that sells: {OUR_PRICING}""",
schema=BRIEF_SCHEMA,
caps={"budget": "100.00", "duration": "5m"},
rules=[f"Only use {urlparse(change['url']).hostname.removeprefix('www.')}.", "Never sign up for an account."],
)
report = run.get("data")
if run["status"] != "completed" or not report:
reason = run.get("stopReason") or run["status"]
await slack(f"{key} changed, but the brief did not finish ({reason}). Replay: {run.get('replayUrl') or 'none'}")
return
STATE.mkdir(exist_ok=True)
file.write_text(json.dumps(report["plans"]))
changes = "\n".join(f"• *{c['plan']}*: {c['what']} (was {c['before']}, now {c['after']})" for c in report["changes"])
replay = f"<{run['replayUrl']}|replay> · " if run.get("replayUrl") else ""
await slack(report["headline"], [
{"type": "header", "text": {"type": "plain_text", "text": report["headline"][:150]}},
{"type": "section", "text": {"type": "mrkdwn", "text": changes or "No plan-level change found."}},
{"type": "section", "text": {"type": "mrkdwn", "text": f"*What it means*\n{report['whatItMeans']}"}},
{"type": "section", "text": {"type": "mrkdwn", "text": f"*Suggested response*\n{report['suggestedResponse']}"}},
{"type": "context", "elements": [
{"type": "mrkdwn", "text": f"<{change['url']}|{key}> · {replay}{run['credits']} credits"},
]},
])
@app.post(HOOK, status_code=202)
async def pricing_changed(request: Request, background: BackgroundTasks) -> dict:
# Answer at once: a monitor delivers each change once and never retries.
background.add_task(brief, await request.json())
return {"accepted": True}
async def watch() -> None:
# One-off setup: a monitor per pricing page, skipping any that already exist.
existing = {m["url"] for m in await oc.monitors.list(label="pricing-*")}
for competitor in COMPETITORS:
if competitor["url"] in existing:
continue
monitor = await oc.monitors.create(
url=competitor["url"],
label=competitor["label"],
schedule="*/30 * * * *", # every 30 minutes
judge=True, # a model summarises the diff into `summary`
normalise={"ignoreSelectors": ['[class*="testimonial"]', '[class*="carousel"]']},
notify={"webhook": f"{PUBLIC_URL}{HOOK}"},
)
print("watching", competitor["url"], monitor["id"])
await oc.aclose()
if __name__ == "__main__":
asyncio.run(watch()) # python pricing_brief.py, once; then serve with: uvicorn pricing_brief:app --port 3000
```
Outcrawl keeps no history of a monitor's checks: `monitors.list` answers each monitor's `lastCheckedAt` and `lastStatus`, not old diffs. That is why the program saves the plan table itself in `pricing-state/` and hands the last one to the next run as "before". The first change for each competitor has only the diff to compare against; every one after that has the full plan table.
A monitor delivers each change once. If your server is down when it posts, that change is not sent again, and the next brief compares against the plans saved before it. Keep the endpoint up, and have it answer before doing any work, as both versions above do.
## What you get back
The agent's answer, as the schema shapes it:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"headline": "Northwind moved SSO from Team to Business and raised Business to $49 a seat.",
"changes": [
{ "plan": "Team", "what": "SSO removed", "before": "SAML SSO included", "after": "Not included" },
{ "plan": "Business", "what": "Monthly price", "before": "$39 per seat", "after": "$49 per seat" },
{ "plan": "Business", "what": "Minimum seats", "before": "None", "after": "10 seats" }
],
"plans": [
{ "name": "Free", "monthly": "$0", "annual": "$0", "limits": ["3 seats", "1 project"] },
{ "name": "Team", "monthly": "$19 per seat", "annual": "$15 per seat", "limits": ["Unlimited projects", "No SSO"] },
{ "name": "Business", "monthly": "$49 per seat", "annual": "$41 per seat", "limits": ["10 seat minimum", "SAML SSO", "Audit log"] }
],
"whatItMeans": "A 10-person team that needs SSO now pays Northwind $490 a month against $590 for our Business plan, down from a $200 gap. A 4-person team that needs SSO now pays Northwind for 10 seats.",
"suggestedResponse": "Brief sales on small teams that need SSO this week: our Business plan has no seat minimum."
}
```
In Slack it arrives as a header with the headline, one line per change (`• *Business*: Monthly price (was $39 per seat, now $49 per seat)`), the two paragraphs, and a footer linking the page, the replay and the credits the brief cost.
## What it costs
* **Watching** is 1 credit per check, whether or not the page changed, plus the model's usage when a changed page is judged (`judge: true`), per [pricing](/docs/billing/pricing). Three pages checked every 30 minutes is 4,320 checks a month, so **4,320 credits a month**.
* **Each brief** is one agent run, billed for the model's usage, browser time and proxy usage. A typical run costs about 15 credits a minute all in; the real figure depends on the task and how heavy the pages are. About 90 s is **about 23 credits per brief**. Each run's `caps.budget` is 100.
* Six real changes a month across the three competitors: 4,320 + 6 × 23, **about 4,458 credits a month**, plus model usage for the six judged changes. That fits the Builder plan's 10,000.
A change that is only a new banner still starts a brief. Add its selector to `ignoreSelectors` and the monitor stops seeing it.
## Make it yours
* Post straight to a channel with no server: `notify: { slack: '' }` sends the monitor's own diff and summary to Slack, without the agent's brief.
* Watch the changelog and the docs' limits page as well as the pricing page, with one monitor per URL on the same endpoint.
* Put the brief where your team decides: open a Linear issue, update a Notion table or a HubSpot property with one more HTTP call after the Slack post.
* Watch a pricing page behind a login by creating the monitor and the run with the same `profile`.
## Related
Schedules, notify targets and normalisation.
Tasks, caps and typed results.
Keep a run on one site.
See exactly what the agent read.
# Write a cited research report
Source: https://outcrawl.ai/docs/examples/deep-research-report
Search, read every source in parallel, send an agent where a page needs using, and get a report where every claim has a URL.
Research by hand is a day of tabs: search a few angles, open twenty results, find that half the answers hide behind a country picker, and paste quotes into a document with their links. This program does the whole pass for one question. It searches each angle, reads every result in parallel with the claims extracted as it reads, sends an agent to the pages whose answer sits behind a control, and writes a markdown report in which every claim is numbered to its source.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* `npm install @outcrawl/sdk` or `pip install outcrawl`. No other service: extraction happens on the page read.
## How it works
One [search](/docs/search/overview) per topic, five results each. A URL two topics both found is read once.
Each page is [scraped](/docs/scrape/overview) with the `json` format and a schema that asks for claims, each with the sentence it came from, and for two verdicts: is the page relevant, and is the answer behind a control on the page.
A page whose answer sits behind a destination picker or a plan selector goes to an [agent run](/docs/agents/overview) that uses the control and answers in the same schema. A per-run [rule](/docs/account/rules) tells it not to sign in, buy anything or enter personal details, and the buying part is enforced.
Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot. A small semaphore keeps the program below your plan's limit, waits 20 seconds and retries on `ConcurrencyLimitError`, and waits out a per-minute `QuotaExceededError` for the seconds it names.
Claims go under their topic with a numbered citation, and the sources list says which ones an agent read. `claims.json` keeps every quote so a reviewer can check each claim against its page.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { writeFile } from 'node:fs/promises';
import { setTimeout as sleep } from 'node:timers/promises';
import { ConcurrencyLimitError, Outcrawl, QuotaExceededError } from '@outcrawl/sdk';
const oc = new Outcrawl();
const QUESTION = 'What does it cost a US traveller to use their phone in Japan for one week?';
const TOPICS = [
{ heading: 'T-Mobile', query: 'T-Mobile international roaming Japan data price' },
{ heading: 'Verizon', query: 'Verizon TravelPass Japan daily price' },
{ heading: 'AT&T', query: 'AT&T International Day Pass Japan price' },
{ heading: 'Travel eSIMs', query: 'Japan eSIM 7 days data plan price' },
];
interface Claim {
claim: string;
quote: string;
}
interface Extracted {
relevant: boolean;
needsInteraction: boolean;
claims: Claim[];
}
const claimsSchema = {
type: 'object',
properties: {
relevant: { type: 'boolean', description: `whether this page helps answer: ${QUESTION}` },
needsInteraction: {
type: 'boolean',
description:
'true when the answer sits behind a control on this page (a destination picker, a plan selector, a tab) ' +
'and is not in the text as loaded',
},
claims: {
type: 'array',
description: `facts on this page that answer: ${QUESTION}`,
items: {
type: 'object',
properties: {
claim: { type: 'string', description: 'one fact, with its price, allowance and conditions' },
quote: { type: 'string', description: 'the sentence on the page it comes from, verbatim' },
},
required: ['claim', 'quote'],
},
},
},
required: ['relevant', 'needsInteraction', 'claims'],
};
// Stay below your plan's concurrent browsers (Builder: 5). Every search, uncached scrape and run holds one.
const SLOTS = 4;
let free = SLOTS;
const waiting: (() => void)[] = [];
async function withSlot(work: () => PromiseLike): Promise {
if (free > 0) free -= 1;
else await new Promise((wake) => waiting.push(wake));
try {
for (let attempt = 1; ; attempt += 1) {
try {
return await work();
} catch (err) {
// Another job on the account holds a slot: wait for one of yours to end.
if (err instanceof ConcurrencyLimitError && attempt < 6) {
await sleep(20_000);
continue;
}
// Over the requests-per-minute ceiling: the error says how long to wait. Out of monthly credits: stop.
if (err instanceof QuotaExceededError && err.ceiling === 'requests_per_minute' && attempt < 6) {
await sleep(err.retryAfterSeconds * 1000);
continue;
}
throw err;
}
}
} finally {
const next = waiting.shift();
if (next !== undefined) next();
else free += 1;
}
}
// 1. Search each topic. One URL is read once, under the first topic that found it.
const sources: { topic: string; url: string; title: string }[] = [];
const hitsPerTopic = await Promise.all(TOPICS.map((t) => withSlot(() => oc.search(t.query, { limit: 5 }))));
TOPICS.forEach((topic, i) => {
for (const hit of hitsPerTopic[i]!) {
if (!sources.some((s) => s.url === hit.url)) sources.push({ topic: topic.heading, url: hit.url, title: hit.title });
}
});
// 2. Read every page in parallel, extracting claims with the quote each one rests on.
const reads = await Promise.allSettled(
sources.map((s) => withSlot(() => oc.scrape(s.url, { formats: ['json'], schema: claimsSchema }))),
);
// 3. Pages whose answer is behind a picker or a tab go to an agent that uses the page.
const found: { topic: string; url: string; title: string; via: string; claims: Claim[] }[] = [];
await Promise.all(
sources.map(async (source, i) => {
const read = reads[i]!;
const page = read.status === 'fulfilled' ? (read.value.json as Extracted | undefined) : undefined;
if (page === undefined || !page.relevant) return;
if (!page.needsInteraction) {
found.push({ ...source, via: 'read', claims: page.claims });
return;
}
const run = await withSlot(() =>
oc.agent({
task:
`Open ${source.url}. The answer to "${QUESTION}" for ${source.topic} is behind a control on this page, ` +
'such as a destination picker or a plan selector. Choose Japan and a 7-day trip, then report every fact ' +
'that answers the question, each with the sentence on the page it comes from.',
schema: claimsSchema,
caps: { budget: '40.00' },
rules: ['Do not sign in, buy anything, or enter personal details.'],
}),
);
const data = run.data as Extracted | undefined;
if (data !== undefined) found.push({ ...source, via: 'agent', claims: data.claims });
}),
);
// 4. Write the report: claims under each topic, numbered sources at the end.
const cited = sources.filter((s) => found.some((f) => f.url === s.url && f.claims.length > 0));
let report = `# ${QUESTION}\n\nResearched ${new Date().toISOString().slice(0, 10)}.\n`;
for (const topic of TOPICS) {
report += `\n## ${topic.heading}\n\n`;
for (const f of found.filter((x) => x.topic === topic.heading)) {
const n = cited.findIndex((s) => s.url === f.url) + 1;
for (const c of f.claims) report += `- ${c.claim} [${n}]\n`;
}
}
report += '\n## Sources\n\n';
cited.forEach((s, i) => {
const via = found.find((f) => f.url === s.url)?.via === 'agent' ? ', read by an agent' : '';
report += `${i + 1}. [${s.title}](${s.url})${via}\n`;
});
await writeFile('report.md', report);
await writeFile('claims.json', JSON.stringify(found, null, 2));
console.log(`${cited.length} sources, ${found.reduce((n, f) => n + f.claims.length, 0)} claims: report.md`);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import datetime
import json
from pathlib import Path
from outcrawl import ConcurrencyLimitError, Outcrawl, QuotaExceededError
QUESTION = "What does it cost a US traveller to use their phone in Japan for one week?"
TOPICS = [
{"heading": "T-Mobile", "query": "T-Mobile international roaming Japan data price"},
{"heading": "Verizon", "query": "Verizon TravelPass Japan daily price"},
{"heading": "AT&T", "query": "AT&T International Day Pass Japan price"},
{"heading": "Travel eSIMs", "query": "Japan eSIM 7 days data plan price"},
]
CLAIMS_SCHEMA = {
"type": "object",
"properties": {
"relevant": {"type": "boolean", "description": f"whether this page helps answer: {QUESTION}"},
"needsInteraction": {
"type": "boolean",
"description": "true when the answer sits behind a control on this page (a destination picker, "
"a plan selector, a tab) and is not in the text as loaded",
},
"claims": {
"type": "array",
"description": f"facts on this page that answer: {QUESTION}",
"items": {
"type": "object",
"properties": {
"claim": {"type": "string", "description": "one fact, with its price, allowance and conditions"},
"quote": {"type": "string", "description": "the sentence on the page it comes from, verbatim"},
},
"required": ["claim", "quote"],
},
},
},
"required": ["relevant", "needsInteraction", "claims"],
}
# Stay below your plan's concurrent browsers (Builder: 5). Every search, uncached scrape and run holds one.
slots = asyncio.Semaphore(4)
async def with_slot(work):
async with slots:
for attempt in range(1, 7):
try:
return await work()
except ConcurrencyLimitError:
# Another job on the account holds a slot: wait for one of yours to end.
if attempt == 6:
raise
await asyncio.sleep(20)
except QuotaExceededError as err:
# Over the requests-per-minute ceiling: the error says how long to wait. Out of monthly credits: stop.
if attempt == 6 or err.ceiling != "requests_per_minute":
raise
await asyncio.sleep(err.retry_after_seconds)
async def main() -> None:
async with Outcrawl() as oc:
# 1. Search each topic. One URL is read once, under the first topic that found it.
hits_per_topic = await asyncio.gather(
*(with_slot(lambda t=t: oc.search(t["query"], limit=5)) for t in TOPICS)
)
sources: list[dict] = []
for topic, hits in zip(TOPICS, hits_per_topic):
for hit in hits:
if all(s["url"] != hit["url"] for s in sources):
sources.append({"topic": topic["heading"], "url": hit["url"], "title": hit["title"]})
# 2. Read every page in parallel, extracting claims with the quote each one rests on.
reads = await asyncio.gather(
*(with_slot(lambda s=s: oc.scrape(s["url"], formats=["json"], schema=CLAIMS_SCHEMA)) for s in sources),
return_exceptions=True,
)
# 3. Pages whose answer is behind a picker or a tab go to an agent that uses the page.
found: list[dict] = []
async def finish(source: dict, read) -> None:
page = None if isinstance(read, BaseException) else read.get("json")
if not page or not page["relevant"]:
return
if not page["needsInteraction"]:
found.append({**source, "via": "read", "claims": page["claims"]})
return
run = await with_slot(
lambda: oc.agent(
task=f'Open {source["url"]}. The answer to "{QUESTION}" for {source["topic"]} is behind a control '
"on this page, such as a destination picker or a plan selector. Choose Japan and a 7-day trip, "
"then report every fact that answers the question, each with the sentence on the page it comes from.",
schema=CLAIMS_SCHEMA,
caps={"budget": "40.00"},
rules=["Do not sign in, buy anything, or enter personal details."],
)
)
data = run.get("data")
if data:
found.append({**source, "via": "agent", "claims": data["claims"]})
await asyncio.gather(*(finish(s, r) for s, r in zip(sources, reads)))
# 4. Write the report: claims under each topic, numbered sources at the end.
cited = [s for s in sources if any(f["url"] == s["url"] and f["claims"] for f in found)]
number = {s["url"]: i + 1 for i, s in enumerate(cited)}
report = f"# {QUESTION}\n\nResearched {datetime.date.today().isoformat()}.\n"
for topic in TOPICS:
report += f"\n## {topic['heading']}\n\n"
for f in (f for f in found if f["topic"] == topic["heading"]):
for c in f["claims"]:
report += f"- {c['claim']} [{number[f['url']]}]\n"
report += "\n## Sources\n\n"
via = {f["url"]: f["via"] for f in found}
for s in cited:
by_agent = ", read by an agent" if via[s["url"]] == "agent" else ""
report += f"{number[s['url']]}. [{s['title']}]({s['url']}){by_agent}\n"
Path("report.md").write_text(report)
Path("claims.json").write_text(json.dumps(found, indent=2))
print(f"{len(cited)} sources, {sum(len(f['claims']) for f in found)} claims: report.md")
asyncio.run(main())
```
## What you get back
`report.md` (an illustrative sample; your run reports what the pages say on the day):
```markdown theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
# What does it cost a US traveller to use their phone in Japan for one week?
Researched 2026-09-25.
## T-Mobile
- Go5G Plus and Go5G Next include unlimited data and texting in Japan at no extra cost, with 15 GB of high-speed data, then up to 256 kbps. [1]
- Calls from Japan cost $0.25 a minute on those plans. [1]
## Verizon
- TravelPass for Japan is $12 a day, charged only on days the phone is used abroad. [2]
- A monthly international plan covers 30 days for $100. [3]
## AT&T
- International Day Pass is $12 a day for the first device, with a monthly cap on charges per billing cycle. [4]
## Travel eSIMs
- 7-day Japan eSIM data plans range from $9 for 3 GB to $27 for unlimited data. [5] [6]
## Sources
1. [International Roaming with T-Mobile](https://www.t-mobile.com/travel-abroad-with-simple-global)
2. [TravelPass](https://www.verizon.com/plans/international/travelpass/), read by an agent
3. [International monthly plans](https://www.verizon.com/plans/international/)
4. [International Day Pass](https://www.att.com/international/international-day-pass/), read by an agent
5. [Japan eSIM plans](https://www.airalo.com/japan-esim)
6. [Japan eSIM 7 days](https://www.holafly.com/esim-japan/)
```
`claims.json` holds the same claims with their verbatim quotes:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
[
{
"topic": "AT&T",
"url": "https://www.att.com/international/international-day-pass/",
"title": "International Day Pass",
"via": "agent",
"claims": [
{
"claim": "International Day Pass is $12 a day for the first device, with a monthly cap on charges per billing cycle.",
"quote": "Japan: $12/day for the first device on your account"
}
]
}
]
```
## What it costs
* Search: 5 credits per 10 results. Four searches of 5 results is 20 results: **10 credits**.
* Reads: 1 credit per page. About 16 distinct pages: **16 credits**.
* Agents: billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in. About 3 pages need an agent, each about a minute: 3 × 15 = **about 45 credits**, with a `caps.budget` of 40 for each run.
That is **about 71 credits per report**. The agent figure is an estimate that includes proxy usage; the real cost depends on the task, and a heavy page costs more. A Builder plan's 10,000 monthly credits covers about 140 reports of this size.
## Make it yours
* Let your own model write the topics and queries from the question, and write prose from `claims.json`, keeping the `[n]` markers.
* Add a second pass: search again for any topic that ended with no claims.
* Raise `limit` on each search to 10 for a wider report; the search line doubles and the reads grow with it.
* Set the semaphore to your plan's browser count less one, so other jobs on the account still have a slot.
## Related
Queries, limits and hits.
The `json` format and schemas.
Schemas, caps and statuses.
`ConcurrencyLimitError` and `QuotaExceededError`.
Every rate on this page.
# Keep a docs knowledge base in sync
Source: https://outcrawl.ai/docs/examples/docs-knowledge-base
Crawl a docs site into embedded chunks, then re-embed only the pages that changed.
A support assistant is only as good as the docs it retrieves, and docs change every week. Re-embedding a whole site every night wastes money, and never re-embedding it means the assistant quotes last quarter's settings. This program crawls a docs section into heading-aware chunks with metadata, stores them in Postgres with pgvector, and on every later run re-embeds only the pages whose content changed and removes the ones that are gone.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* Postgres with the `pgvector` extension, in `DATABASE_URL`. The program creates its two tables.
* An embeddings key. The program uses OpenAI's `text-embedding-3-small` (`OPENAI_API_KEY`); any embedding model works if you change the vector size.
* `npm install @outcrawl/sdk openai pg` or `pip install outcrawl openai "psycopg[binary]"`.
## How it works
A [crawl](/docs/crawl/overview) starts at the section root, stays inside `includePaths`, and streams each page as clean markdown with its title and fetch time. Nothing waits for the whole site.
Each page's markdown is hashed and compared with the hash stored last time. Only new or changed pages are chunked and embedded. Pages that came back empty or behind an interstitial are skipped, so a bad fetch never replaces a good copy.
Chunks follow the page's own headings, up to about 1,500 characters, and keep the heading path (`Workflows > Triggers > schedule`) as metadata. Headings inside code blocks are ignored.
The page row and its chunks are replaced together, so a search never sees half of an update.
Pages missing from a crawl that finished below its `limit` are deleted with their chunks. A crawl that failed or hit the limit deletes nothing.
## The program
````typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { createHash } from 'node:crypto';
import OpenAI from 'openai';
import pg from 'pg';
import { Outcrawl } from '@outcrawl/sdk';
const SITE = 'https://docs.github.com/en/actions';
const LIMIT = 1000; // a crawl that stops here may have missed pages, so nothing is deleted after it
const oc = new Outcrawl();
const openai = new OpenAI();
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });
await db.query(`
create extension if not exists vector;
create table if not exists kb_pages (
url text primary key, hash text not null, title text, crawled_at timestamptz not null);
create table if not exists kb_chunks (
id text primary key, url text not null references kb_pages (url) on delete cascade,
heading text not null, content text not null, embedding vector(1536) not null);
`);
// Split on headings, keeping the heading path as metadata. Headings inside code blocks are not headings.
function chunk(markdown: string, max = 1500): { heading: string; text: string }[] {
const chunks: { heading: string; text: string }[] = [];
let path: string[] = [];
let lines: string[] = [];
let inCode = false;
const flush = () => {
let piece = '';
for (const paragraph of lines.join('\n').split(/\n{2,}/)) {
if (piece !== '' && piece.length + paragraph.length > max) {
chunks.push({ heading: path.join(' > '), text: piece.trim() });
piece = '';
}
piece += `${paragraph}\n\n`;
}
if (piece.trim() !== '') chunks.push({ heading: path.join(' > '), text: piece.trim() });
lines = [];
};
for (const line of markdown.split('\n')) {
if (line.startsWith('```')) inCode = !inCode;
const heading = inCode ? null : /^(#{1,3})\s+(.+)/.exec(line);
if (heading !== null) {
flush();
path = [...path.slice(0, heading[1]!.length - 1), heading[2]!.trim()];
}
lines.push(line);
}
flush();
return chunks;
}
const known = new Map(
(await db.query<{ url: string; hash: string }>('select url, hash from kb_pages')).rows.map((r) => [r.url, r.hash]),
);
const seen = new Set();
let updated = 0;
const crawl = await oc.crawl(SITE, {
limit: LIMIT,
includePaths: ['/en/actions/**'], // `**` spans path segments; `*` would stop at one
formats: ['markdown'],
});
for await (const page of crawl) {
// A page that failed or came back empty must not replace a good copy.
if (page.statusCode !== 200 || page.markdown === undefined) continue;
if (page.warnings?.some((w) => w.code === 'empty_capture' || w.code === 'blocked_interstitial')) continue;
seen.add(page.url);
const hash = createHash('sha256').update(page.markdown).digest('hex');
if (known.get(page.url) === hash) continue; // unchanged since the last crawl
const title = page.metadata.title ?? page.url;
const chunks = chunk(page.markdown);
if (chunks.length === 0) continue;
const embedded = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: chunks.map((c) => `${title}\n${c.heading}\n\n${c.text}`),
});
const tx = await db.connect();
try {
await tx.query('begin');
await tx.query(
`insert into kb_pages (url, hash, title, crawled_at) values ($1, $2, $3, $4)
on conflict (url) do update set hash = $2, title = $3, crawled_at = $4`,
[page.url, hash, title, page.metadata.scrapedAt],
);
await tx.query('delete from kb_chunks where url = $1', [page.url]);
for (const [i, c] of chunks.entries()) {
await tx.query(
'insert into kb_chunks (id, url, heading, content, embedding) values ($1, $2, $3, $4, $5)',
[`${page.url}#${i}`, page.url, c.heading, c.text, JSON.stringify(embedded.data[i]!.embedding)],
);
}
await tx.query('commit');
updated += 1;
} catch (err) {
await tx.query('rollback');
throw err;
} finally {
tx.release();
}
}
// Pages the site no longer serves, removed only after a crawl that finished below its limit.
let removed = 0;
if (crawl.status === 'completed' && crawl.completed < LIMIT) {
const gone = [...known.keys()].filter((url) => !seen.has(url));
removed = (await db.query('delete from kb_pages where url = any($1)', [gone])).rowCount ?? 0;
}
console.log(`crawl ${crawl.id}: ${seen.size} pages read, ${updated} re-embedded, ${removed} removed`);
await db.end();
````
````python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import hashlib
import json
import os
import re
import psycopg
from openai import AsyncOpenAI
from outcrawl import Outcrawl
SITE = "https://docs.github.com/en/actions"
LIMIT = 1000 # a crawl that stops here may have missed pages, so nothing is deleted after it
SCHEMA = """
create extension if not exists vector;
create table if not exists kb_pages (
url text primary key, hash text not null, title text, crawled_at timestamptz not null);
create table if not exists kb_chunks (
id text primary key, url text not null references kb_pages (url) on delete cascade,
heading text not null, content text not null, embedding vector(1536) not null);
"""
def chunk(markdown: str, limit: int = 1500) -> list[dict]:
"""Split on headings, keeping the heading path as metadata. Headings inside code blocks are not headings."""
chunks: list[dict] = []
path: list[str] = []
lines: list[str] = []
in_code = False
def flush() -> None:
piece = ""
for paragraph in re.split(r"\n{2,}", "\n".join(lines)):
if piece and len(piece) + len(paragraph) > limit:
chunks.append({"heading": " > ".join(path), "text": piece.strip()})
piece = ""
piece += paragraph + "\n\n"
if piece.strip():
chunks.append({"heading": " > ".join(path), "text": piece.strip()})
lines.clear()
for line in markdown.split("\n"):
if line.startswith("```"):
in_code = not in_code
heading = None if in_code else re.match(r"^(#{1,3})\s+(.+)", line)
if heading:
flush()
path[:] = [*path[: len(heading[1]) - 1], heading[2].strip()]
lines.append(line)
flush()
return chunks
async def main() -> None:
llm = AsyncOpenAI()
async with Outcrawl() as oc, await psycopg.AsyncConnection.connect(os.environ["DATABASE_URL"], autocommit=True) as db:
await db.execute(SCHEMA)
known = dict(await (await db.execute("select url, hash from kb_pages")).fetchall())
seen: set[str] = set()
updated = 0
crawl = await oc.crawl(SITE, limit=LIMIT, includePaths=["/en/actions/**"], formats=["markdown"])
async for page in crawl:
# A page that failed or came back empty must not replace a good copy.
markdown = page.get("markdown")
if page["statusCode"] != 200 or not markdown:
continue
if any(w["code"] in ("empty_capture", "blocked_interstitial") for w in page.get("warnings", [])):
continue
seen.add(page["url"])
digest = hashlib.sha256(markdown.encode()).hexdigest()
if known.get(page["url"]) == digest:
continue # unchanged since the last crawl
title = page["metadata"].get("title") or page["url"]
chunks = chunk(markdown)
if not chunks:
continue
embedded = await llm.embeddings.create(
model="text-embedding-3-small",
input=[f"{title}\n{c['heading']}\n\n{c['text']}" for c in chunks],
)
async with db.transaction():
await db.execute(
"""insert into kb_pages (url, hash, title, crawled_at) values (%s, %s, %s, %s)
on conflict (url) do update set hash = excluded.hash, title = excluded.title,
crawled_at = excluded.crawled_at""",
(page["url"], digest, title, page["metadata"]["scrapedAt"]),
)
await db.execute("delete from kb_chunks where url = %s", (page["url"],))
for i, c in enumerate(chunks):
await db.execute(
"insert into kb_chunks (id, url, heading, content, embedding) values (%s, %s, %s, %s, %s)",
(f"{page['url']}#{i}", page["url"], c["heading"], c["text"], json.dumps(embedded.data[i].embedding)),
)
updated += 1
# Pages the site no longer serves, removed only after a crawl that finished below its limit.
removed = 0
job = crawl.job or {}
if job.get("status") == "completed" and job.get("completed", LIMIT) < LIMIT:
gone = [url for url in known if url not in seen]
removed = (await db.execute("delete from kb_pages where url = any(%s)", (gone,))).rowcount
print(f"crawl {job.get('id')}: {len(seen)} pages read, {updated} re-embedded, {removed} removed")
asyncio.run(main())
````
Run it nightly from any scheduler, for example `0 3 * * * cd /srv/kb && npx tsx kb.ts` (or `python kb.py`) in cron. The first run embeds everything; later runs embed only the difference.
## What you get back
The console, on a night when the docs team shipped two edits and retired a page:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
crawl crw_01K5ZT4N8Q: 412 pages read, 2 re-embedded, 1 removed
```
One row of `kb_chunks`:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"id": "https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#14",
"url": "https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows",
"heading": "Events that trigger workflows > schedule",
"content": "## schedule\n\nThe schedule event allows you to trigger a workflow at a scheduled time. You can schedule a workflow to run at specific UTC times using POSIX cron syntax...",
"embedding": "[0.0132, -0.0417, 0.0088, ...]"
}
```
Retrieval is one query: `select url, heading, content from kb_chunks order by embedding <=> $1 limit 8`, and every chunk carries the URL your assistant cites.
## What it costs
A crawl is 1 credit per page. A 400-page docs section is **400 credits per crawl**, whether or not the pages changed, because a page has to be read to know.
| Schedule | Crawls a month | Credits a month | Fits |
| - | - | - | - |
| Nightly | 30 | about 12,000 | Pro (50,000) |
| Twice a week | 9 | about 3,600 | Builder (10,000) |
| Weekly | 4 to 5 | about 1,800 | Builder (10,000) |
The saving is on your side: only changed pages are embedded and written, so a quiet night costs your embedding provider almost nothing.
## Make it yours
* Put a [monitor](/docs/monitors/overview) on the docs changelog. It calls a webhook endpoint of yours when the page changes, and your endpoint starts this crawl the moment something ships instead of waiting for night.
* Crawl several sections or sites in one run by looping over `[root, includePaths]` pairs; the stored hashes already work for any URL.
* Store `page.metadata.language` too and filter retrieval by the user's language.
* Replace pgvector with your own vector store: the upsert is the only part that changes.
## Related
Limits, paths and streaming.
Recover a crawl's pages if the stream drops.
A webhook when a page changes.
Every rate on this page.
# Collect every vendor invoice into one sheet
Source: https://outcrawl.ai/docs/examples/download-and-file-vendor-invoices
Agents sign in to each billing portal at once, bring back every invoice's number, date and amount, download its PDF, and file it all as a CSV.
Month end means signing in to six billing portals, finding the invoices page in each one, and copying numbers into a spreadsheet. This program sends one agent per portal, each signed in through its own profile, a few at a time. Each returns every invoice from the month with its number, date, total and status, and downloads each invoice's PDF. The program then saves the PDFs beside the numbers and writes a CSV your bookkeeper can import.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* One [profile](/docs/account/profiles) per portal, signed in and labelled `ap-` (`ap-aws`, `ap-zoom`...). Sign each one in once: from TypeScript with `oc.browser({ profile })`, or with an agent run that types your password from a [text secret](/docs/account/secrets), reading any emailed code through an inbox [integration](/docs/account/integrations) such as AgentMail.
* Optional: `SHEET_WEBHOOK_URL`, an endpoint of yours that appends rows to your sheet or accounting system, for example a Google Apps Script web app.
## How it works
Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot. The program keeps `MAX_AT_ONCE` portals going, below your plan's count. A submit over the limit is refused with `ConcurrencyLimitError`, and the program waits 20 seconds and tries again; a `QuotaExceededError` for requests per minute says how long to wait in `retryAfterSeconds`.
Each [agent run](/docs/agents/overview) starts signed in through its [profile](/docs/account/profiles), finds the month's invoices, and answers a [schema](/docs/agents/overview#typed-results): number, date, total, currency, status, and the PDF link when the page shows one. It then downloads each invoice's PDF. The task is read-only, and the [rule](/docs/account/rules) `Only use aws.amazon.com.` is enforced for each portal's own site.
Every file a run downloads is kept for 7 days. Once a run ends, `job.downloads()` lists its PDFs and `saveDownload` writes each one to `invoices/2026-08/`. Each PDF is matched to its invoice by its link or by the invoice number in its name.
Every row goes into `invoices/2026-08/invoices.csv`, and, if you set `SHEET_WEBHOOK_URL`, to your own endpoint as JSON.
A portal that builds the PDF in the page and shows no link still works: the agent presses its download button, and the file is kept like any other. See [downloads](/docs/agents/downloads). A PDF the program could not match to an invoice is still saved in the folder; its row leaves `stored` empty.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { mkdir, writeFile } from 'node:fs/promises';
import { setTimeout as sleep } from 'node:timers/promises';
import { ConcurrencyLimitError, Outcrawl, QuotaExceededError, saveDownload } from '@outcrawl/sdk';
import type { AgentRequest } from '@outcrawl/sdk';
const oc = new Outcrawl();
const MONTH = { label: 'August 2026', key: '2026-08' };
const SHEET_WEBHOOK_URL = process.env['SHEET_WEBHOOK_URL'] ?? ''; // optional: your own endpoint that appends rows
// Each profile is signed in to that vendor's billing portal.
const vendors = [
{ name: 'AWS', site: 'aws.amazon.com', billing: 'https://us-east-1.console.aws.amazon.com/billing/home#/bills', profile: 'ap-aws' },
{ name: 'Google Workspace', site: 'google.com', billing: 'https://admin.google.com/ac/billing/accounts', profile: 'ap-google' },
{ name: 'Slack', site: 'slack.com', billing: 'https://acme-team.slack.com/admin/billing/history', profile: 'ap-slack' },
{ name: 'Atlassian', site: 'atlassian.com', billing: 'https://admin.atlassian.com', profile: 'ap-atlassian' },
{ name: 'Zoom', site: 'zoom.us', billing: 'https://zoom.us/billing', profile: 'ap-zoom' },
{ name: 'Twilio', site: 'twilio.com', billing: 'https://console.twilio.com/us1/billing/manage-billing/invoices', profile: 'ap-twilio' },
];
// Runs at once. Keep it below your plan's browser count (Free 2, Builder 5, Pro 25, Scale 100).
const MAX_AT_ONCE = 3;
interface Invoice {
number: string;
date: string;
total: string;
currency: string;
status: string;
documentUrl?: string;
}
const invoiceSchema = {
type: 'object',
properties: {
invoices: {
type: 'array',
items: {
type: 'object',
properties: {
number: { type: 'string' },
date: { type: 'string', description: 'Invoice date, YYYY-MM-DD' },
total: { type: 'string', description: 'Total due as a plain decimal, e.g. 412.87' },
currency: { type: 'string', description: 'ISO code, e.g. USD' },
status: { type: 'string', description: 'paid, due or overdue' },
documentUrl: { type: 'string', description: 'Direct link to the invoice PDF, if the page shows one' },
},
required: ['number', 'date', 'total', 'currency', 'status'],
},
},
},
required: ['invoices'],
};
// Submit a run and answer its id, waiting for a free browser slot instead of failing.
async function start(request: AgentRequest): Promise {
for (;;) {
try {
return (await oc.agent(request).submitted).id;
} catch (error) {
if (error instanceof ConcurrencyLimitError) {
await sleep(20_000); // no Retry-After: a slot frees when one of your runs ends
} else if (error instanceof QuotaExceededError && error.ceiling === 'requests_per_minute') {
await sleep(error.retryAfterSeconds * 1000); // the error says how long to wait
} else {
throw error; // out of monthly credits, or a bad request: retrying cannot help
}
}
}
}
async function inPool(items: readonly T[], size: number, work: (item: T) => Promise): Promise {
const results: R[] = new Array(items.length);
let next = 0;
const worker = async () => {
while (next < items.length) {
const index = next++;
results[index] = await work(items[index] as T);
}
};
await Promise.all(Array.from({ length: Math.min(size, items.length) }, worker));
return results;
}
const profiles = await oc.profiles.list({ label: 'ap-*' });
const profileIds = new Map(profiles.map((profile) => [profile.label, profile.id]));
const folder = `invoices/${MONTH.key}`;
await mkdir(folder, { recursive: true });
const perVendor = await inPool(vendors, MAX_AT_ONCE, async (vendor) => {
const profile = profileIds.get(vendor.profile);
if (profile === undefined) return { vendor: vendor.name, invoices: [], problem: `no profile ${vendor.profile}` };
const job = oc.agent.job(await start({
task:
`You are signed in to ${vendor.name}. Open ${vendor.billing} and find every invoice dated in ${MONTH.label}. ` +
'For each one, report its number, date, total, currency, whether it is paid, and the direct link to its PDF ' +
'if the page shows one, then download its PDF. Read only: do not pay, dispute or change anything.',
caps: { budget: '150.00', duration: '10m' },
profile,
rules: [`Only use ${vendor.site}.`],
schema: invoiceSchema,
}));
const run = await job.settled();
const invoices = (run.data as { invoices: Invoice[] } | undefined)?.invoices ?? [];
const problem = run.data === undefined ? `run ended ${run.stopReason ?? run.status}, replay ${run.replayUrl ?? 'none'}` : undefined;
// Every PDF the run downloaded is kept for 7 days: save each one beside the CSV.
const saved = [];
for (const download of await job.downloads()) {
saved.push({ download, path: await saveDownload(download, folder) });
}
const rows = invoices.map((invoice) => {
const file = saved.find(({ download }) =>
download.sourceUrl === invoice.documentUrl || download.name.includes(invoice.number));
return { vendor: vendor.name, ...invoice, stored: file?.path ?? '' };
});
return { vendor: vendor.name, invoices: rows, problem };
});
const rows = perVendor.flatMap((entry) => entry.invoices);
const columns = ['vendor', 'number', 'date', 'total', 'currency', 'status', 'documentUrl', 'stored'] as const;
const csv = [
columns.join(','),
...rows.map((row) => columns.map((column) => `"${String(row[column] ?? '').replaceAll('"', '""')}"`).join(',')),
].join('\n');
await writeFile(`${folder}/invoices.csv`, `${csv}\n`);
if (SHEET_WEBHOOK_URL !== '') {
await fetch(SHEET_WEBHOOK_URL, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ month: MONTH.key, rows }),
});
}
console.table(rows.map(({ vendor, number, date, total, currency, status }) => ({ vendor, number, date, total, currency, status })));
for (const entry of perVendor) if (entry.problem !== undefined) console.log(`${entry.vendor}: ${entry.problem}`);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import csv
import os
from pathlib import Path
import httpx
from outcrawl import ConcurrencyLimitError, Outcrawl, QuotaExceededError, save_download
MONTH = {"label": "August 2026", "key": "2026-08"}
SHEET_WEBHOOK_URL = os.environ.get("SHEET_WEBHOOK_URL", "") # optional: your own endpoint that appends rows
# Each profile is signed in to that vendor's billing portal.
VENDORS = [
{"name": "AWS", "site": "aws.amazon.com", "billing": "https://us-east-1.console.aws.amazon.com/billing/home#/bills", "profile": "ap-aws"},
{"name": "Google Workspace", "site": "google.com", "billing": "https://admin.google.com/ac/billing/accounts", "profile": "ap-google"},
{"name": "Slack", "site": "slack.com", "billing": "https://acme-team.slack.com/admin/billing/history", "profile": "ap-slack"},
{"name": "Atlassian", "site": "atlassian.com", "billing": "https://admin.atlassian.com", "profile": "ap-atlassian"},
{"name": "Zoom", "site": "zoom.us", "billing": "https://zoom.us/billing", "profile": "ap-zoom"},
{"name": "Twilio", "site": "twilio.com", "billing": "https://console.twilio.com/us1/billing/manage-billing/invoices", "profile": "ap-twilio"},
]
# Runs at once. Keep it below your plan's browser count (Free 2, Builder 5, Pro 25, Scale 100).
MAX_AT_ONCE = 3
INVOICE_SCHEMA = {
"type": "object",
"properties": {
"invoices": {
"type": "array",
"items": {
"type": "object",
"properties": {
"number": {"type": "string"},
"date": {"type": "string", "description": "Invoice date, YYYY-MM-DD"},
"total": {"type": "string", "description": "Total due as a plain decimal, e.g. 412.87"},
"currency": {"type": "string", "description": "ISO code, e.g. USD"},
"status": {"type": "string", "description": "paid, due or overdue"},
"documentUrl": {"type": "string", "description": "Direct link to the invoice PDF, if the page shows one"},
},
"required": ["number", "date", "total", "currency", "status"],
},
}
},
"required": ["invoices"],
}
COLUMNS = ["vendor", "number", "date", "total", "currency", "status", "documentUrl", "stored"]
async def start(oc: Outcrawl, **request) -> str:
"""Submit a run and answer its id, waiting for a free browser slot instead of failing."""
while True:
try:
return (await oc.agent(**request).submitted)["id"]
except ConcurrencyLimitError:
await asyncio.sleep(20) # no Retry-After: a slot frees when one of your runs ends
except QuotaExceededError as error:
if error.ceiling != "requests_per_minute":
raise # out of monthly credits: retrying cannot help
await asyncio.sleep(error.retry_after_seconds) # the error says how long to wait
async def main() -> None:
folder = Path("invoices") / MONTH["key"]
folder.mkdir(parents=True, exist_ok=True)
async with Outcrawl() as oc:
profiles = await oc.profiles.list(label="ap-*")
profile_ids = {profile["label"]: profile["id"] for profile in profiles}
slots = asyncio.Semaphore(MAX_AT_ONCE)
async def collect(vendor: dict) -> dict:
profile = profile_ids.get(vendor["profile"])
if profile is None:
return {"vendor": vendor["name"], "rows": [], "problem": f"no profile {vendor['profile']}"}
async with slots:
run_id = await start(
oc,
task=(
f"You are signed in to {vendor['name']}. Open {vendor['billing']} and find every invoice "
f"dated in {MONTH['label']}. For each one, report its number, date, total, currency, whether "
"it is paid, and the direct link to its PDF if the page shows one, then download its PDF. "
"Read only: do not pay, dispute or change anything."
),
caps={"budget": "150.00", "duration": "10m"},
profile=profile,
rules=[f"Only use {vendor['site']}."],
schema=INVOICE_SCHEMA,
)
job = oc.agent.job(run_id)
run = await job.settled()
data = run.get("data")
problem = None if data else f"run ended {run.get('stopReason') or run['status']}, replay {run.get('replayUrl')}"
# Every PDF the run downloaded is kept for 7 days: save each one beside the CSV.
saved = []
for download in await job.downloads():
saved.append((download, await save_download(download, folder)))
rows = []
for invoice in (data or {}).get("invoices", []):
path = next(
(
path
for download, path in saved
if download["sourceUrl"] == invoice.get("documentUrl") or invoice["number"] in download["name"]
),
None,
)
rows.append({"vendor": vendor["name"], **invoice, "stored": str(path) if path else ""})
return {"vendor": vendor["name"], "rows": rows, "problem": problem}
per_vendor = await asyncio.gather(*(collect(vendor) for vendor in VENDORS))
rows = [row for entry in per_vendor for row in entry["rows"]]
with open(folder / "invoices.csv", "w", newline="") as out:
writer = csv.DictWriter(out, fieldnames=COLUMNS, extrasaction="ignore")
writer.writeheader()
writer.writerows(rows)
if SHEET_WEBHOOK_URL:
async with httpx.AsyncClient() as http:
await http.post(SHEET_WEBHOOK_URL, json={"month": MONTH["key"], "rows": rows})
for row in rows:
print(f"{row['vendor']:<18} {row['number']:<22} {row['date']} {row['total']:>10} {row['currency']} {row['status']}")
for entry in per_vendor:
if entry["problem"]:
print(f"{entry['vendor']}: {entry['problem']}")
asyncio.run(main())
```
## What you get back
`invoices/2026-08/invoices.csv`, with each invoice's PDF saved beside it:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
vendor,number,date,total,currency,status,documentUrl,stored
"AWS","EUINGB26-1182443","2026-08-02","1843.27","USD","paid","https://us-east-1.console.aws.amazon.com/billing/home#/bills?invoice=EUINGB26-1182443","invoices/2026-08/EUINGB26-1182443.pdf"
"Google Workspace","5261834903","2026-08-01","432.00","USD","paid","https://payments.google.com/payments/apis-secure/doc/5261834903.pdf","invoices/2026-08/5261834903.pdf"
"Slack","SLK-0823-77120","2026-08-15","1275.00","USD","paid","https://acme-team.slack.com/admin/billing/invoice/SLK-0823-77120.pdf","invoices/2026-08/SLK-0823-77120.pdf"
"Atlassian","IN-004-918-223","2026-08-09","1512.50","USD","due","","invoices/2026-08/IN-004-918-223.pdf"
"Zoom","INV214773905","2026-08-12","299.80","USD","paid","https://zoom.us/billing/invoice/INV214773905.pdf","invoices/2026-08/INV214773905.pdf"
"Twilio","TW-AUG26-00419","2026-08-31","86.42","USD","paid","https://console.twilio.com/us1/billing/invoices/TW-AUG26-00419.pdf","invoices/2026-08/TW-AUG26-00419.pdf"
```
Here Atlassian showed no PDF link, so its agent pressed the download button instead, and the PDF was matched by its invoice number. A portal whose run did not finish is printed at the end with its replay link, so you can watch where it stopped.
## What it costs
An agent run is billed for the model's usage, browser time and proxy usage, per [pricing](/docs/billing/pricing). A typical agent run costs about 15 credits a minute all in, proxy usage included. That is an estimate: the real figure depends on the task, and a page-heavy portal costs more.
* **Each portal** takes about 2 minutes: **about 30 credits**.
* **Keeping and fetching the PDFs** costs nothing extra.
Six portals: 6 × 30 = **about 180 credits a month**, under a fifth of the Free plan's 1,000. Each portal run's `caps.budget` is 150.
## Make it yours
* File straight into your accounting: replace the sheet POST with one call per row to your accounting system's API to create a bill.
* Run it on the 3rd of every month from your own scheduler (cron, a CI schedule) and set `MONTH` from the date.
* Catch what is unpaid: post every row whose `status` is `due` or `overdue` to Slack.
* Let a run sign back in when a session has lapsed: give it your password as a [secret](/docs/account/secrets) and an inbox [integration](/docs/account/integrations) for the code.
## Related
One signed-in browser per portal.
Runs, caps and typed results.
Every file a run downloaded, kept 7 days.
Hand a file to a run so it can upload it.
# Track every open role across 300 startups
Source: https://outcrawl.ai/docs/examples/every-open-role-across-300-startups
Read 300 careers pages every night, send agents to the applicant-tracking sites that need clicking, and get a clean sheet of what is new.
Recruiters, job seekers and investors all want the same list: every open platform engineering role at the 300 companies they follow, with location, remote policy and pay, and what appeared since yesterday. Half of those careers pages are a plain list, and the other half sit on applicant-tracking systems where you have to search, filter and page through results. This program reads the plain ones as structured data, sends an agent to the rest, normalises and dedupes every role, and runs nightly to report what is new.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* `startups.csv` with a header row and `name,careersUrl` per line.
* A plan with enough browsers for your lanes: Free 2, Builder 5, Pro 25, Scale 100. Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot. A nightly job shares that count with anything else running on your account at the same hour.
* Somewhere to run it nightly: cron, a CI schedule, or any scheduler.
## How it works
A [scrape](/docs/scrape/overview) with `formats: ['json']` and a schema turns a careers page into a list of roles: title, location, remote policy, posted pay, link. It is 1 credit a page.
Workday, iCIMS, Taleo, SuccessFactors and Oracle boards, and any page whose roles did not come back from the scrape, go to an [agent](/docs/agents/overview). It uses the site's own search and filters, pages through every result, opens each posting and returns the same schema. A [rule](/docs/account/rules) tells it not to apply or sign in, and blocks it from creating an account.
Roles are keyed by their posting URL, without tracking parameters. A role seen on a previous night keeps its details; only a new one gets its posting scraped for pay and remote policy.
Four lanes. A refused call (`ConcurrencyLimitError`, or `QuotaExceededError` on requests per minute) waits and retries: 20 seconds for a slot, or the `retryAfterSeconds` the error names. Running out of monthly credits stops. Every step reserves its maximum cost against the night's ceiling before it starts.
`roles.csv` holds every open role, `new-roles.json` holds tonight's new ones, and `roles.json` is the state for tomorrow. A company that could not be read tonight keeps yesterday's roles, so a failure never reads as a closed role.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { existsSync } from 'node:fs';
import { readFile, writeFile } from 'node:fs/promises';
import { ConcurrencyLimitError, NoCapacityError, Outcrawl, QuotaExceededError } from '@outcrawl/sdk';
const oc = new Outcrawl();
// Every key on your account shares one browser slot count (Free 2, Builder 5,
// Pro 25, Scale 100). Keep LANES below yours so other work still gets a slot.
const LANES = 4;
const NIGHT_CEILING = 2_000; // credits, per night
const AGENT_BUDGET = '50.00'; // about 3 minutes of a typical run
const WANTED = /\b(platform|infrastructure|site reliability|devops|backend)\b.*\bengineer/i;
const WANTED_WORDS = 'platform, infrastructure, site reliability, DevOps or backend engineer';
const NEEDS_AGENT = /myworkdayjobs\.com|icims\.com|taleo\.net|successfactors\.(com|eu)|oraclecloud\.com/;
interface Found {
title: string;
url: string;
location?: string;
workplace?: 'remote' | 'hybrid' | 'onsite' | 'unknown';
salaryMin?: number;
salaryMax?: number;
currency?: string;
period?: 'year' | 'month' | 'hour';
}
interface Role {
company: string;
title: string;
location: string;
workplace: string;
salaryMin: number | null;
salaryMax: number | null;
currency: string | null;
period: string | null;
url: string;
firstSeen: string;
}
const ROLE = {
type: 'object',
properties: {
title: { type: 'string' },
location: { type: 'string' },
workplace: { type: 'string', enum: ['remote', 'hybrid', 'onsite', 'unknown'] },
salaryMin: { type: 'number', description: 'Lowest posted pay. Omit when no pay is posted.' },
salaryMax: { type: 'number', description: 'Highest posted pay. Omit when no pay is posted.' },
currency: { type: 'string', description: 'ISO 4217 code of the posted pay' },
period: { type: 'string', enum: ['year', 'month', 'hour'] },
url: { type: 'string', description: 'Link to the posting itself' },
},
required: ['title', 'url'],
};
const ROLES = { type: 'object', properties: { roles: { type: 'array', items: ROLE } }, required: ['roles'] };
const sleep = (ms: number) => new Promise((wake) => setTimeout(wake, ms));
// A refused call is retried; anything else is the caller's to see.
async function withSlot(call: () => Promise): Promise {
for (let attempt = 1; ; attempt++) {
try {
return await call();
} catch (error) {
if (attempt === 8) throw error;
if (error instanceof ConcurrencyLimitError || error instanceof NoCapacityError) {
// Refused, not queued, and no Retry-After: a slot frees when one of your calls ends.
await sleep(20_000);
} else if (error instanceof QuotaExceededError && error.ceiling === 'requests_per_minute') {
await sleep(error.retryAfterSeconds * 1_000);
} else {
throw error; // out of monthly credits, or a bad request: retrying cannot help
}
}
}
}
// Reserve a step's maximum before it starts, settle to what it really cost.
let spent = 0;
let held = 0;
async function withinBudget(maximum: number, step: () => Promise, cost: (value: T) => number): Promise {
if (spent + held + maximum > NIGHT_CEILING) return undefined;
held += maximum;
let actual = maximum;
try {
const value = await step();
actual = cost(value);
return value;
} finally {
held -= maximum;
spent += actual;
}
}
// At most LANES startups in flight: each lane takes the next one when it is free.
async function inLanes(items: readonly T[], work: (item: T) => Promise): Promise {
const results: R[] = [];
let next = 0;
const lane = async () => {
while (next < items.length) {
const index = next++;
results[index] = await work(items[index] as T);
}
};
await Promise.all(Array.from({ length: Math.min(LANES, items.length) }, lane));
return results;
}
// One posting, one key: no fragment, no tracking parameters, no trailing slash.
function keyOf(url: string): string {
const parsed = new URL(url);
for (const name of [...parsed.searchParams.keys()]) {
if (/^(utm_|gh_src$|lever-source|source$|ref$)/.test(name)) parsed.searchParams.delete(name);
}
return `${parsed.host.toLowerCase()}${parsed.pathname.replace(/\/$/, '')}${parsed.search}`;
}
function absolute(url: string, base: string): string | null {
try {
return new URL(url, base).href;
} catch {
return null;
}
}
const today = new Date().toISOString().slice(0, 10);
const known = new Map(
existsSync('roles.json') ? Object.entries(JSON.parse(await readFile('roles.json', 'utf8')) as Record) : [],
);
const startups: [string, string][] = (await readFile('startups.csv', 'utf8'))
.split('\n')
.slice(1)
.filter((line) => line.includes(','))
.map((line) => [line.slice(0, line.lastIndexOf(',')).trim(), line.slice(line.lastIndexOf(',') + 1).trim()]);
async function askAgent(name: string, careersUrl: string): Promise {
const run = await withinBudget(
Number(AGENT_BUDGET),
async () => {
// Only the submit is retried: a refusal arrives before any run exists.
const { job } = await withSlot(async () => {
const job = oc.agent({
task:
`Open ${careersUrl}, the careers site of ${name}. Use its search box or filters to find every ` +
`open ${WANTED_WORDS} role, and page through all results. Open each posting and return its ` +
`title, location, whether it is remote, hybrid or onsite, the pay range if the posting shows one, ` +
`and the link to the posting.`,
caps: { budget: AGENT_BUDGET, duration: '5m' },
schema: ROLES,
rules: ['Never apply to a role, create an account or sign in.'],
});
await job.submitted; // a refused submit throws here
return { job }; // wrapped, because a job is awaitable and returning it bare waits for the run
});
return await job;
},
(settled) => Number(settled.credits),
);
// Not run, or did not finish: tonight says nothing about this company.
if (run === undefined || run.status !== 'completed') return null;
return (run.data as { roles?: Found[] } | undefined)?.roles ?? [];
}
// null means "could not read tonight": the company keeps yesterday's roles.
async function collect([name, careersUrl]: [string, string]): Promise<{ name: string; roles: Found[] | null }> {
try {
let listed: Found[] = [];
if (!NEEDS_AGENT.test(careersUrl)) {
const page = await withinBudget(1, () => withSlot(() => oc.scrape(careersUrl, { formats: ['json'], schema: ROLES })), () => 1);
listed = (page?.json as { roles?: Found[] } | undefined)?.roles ?? [];
}
if (listed.length === 0) {
const found = await askAgent(name, careersUrl);
if (found === null) return { name, roles: null };
return {
name,
roles: found.flatMap((role) => {
const url = absolute(role.url, careersUrl);
return url !== null && WANTED.test(role.title) ? [{ ...role, url }] : [];
}),
};
}
const roles: Found[] = [];
for (const role of listed.filter((r) => WANTED.test(r.title))) {
const url = absolute(role.url, careersUrl);
if (url === null) continue;
if (!known.has(keyOf(url))) {
// New tonight: read the posting itself for pay and remote policy.
const posting = await withinBudget(1, () => withSlot(() => oc.scrape(url, { formats: ['json'], schema: ROLE })), () => 1);
Object.assign(role, (posting?.json ?? {}) as Partial);
}
roles.push({ ...role, url });
}
return { name, roles };
} catch (error) {
console.error(`${name}: ${String(error)}`);
return { name, roles: null };
}
}
const results = await inLanes(startups, collect);
// Normalise, dedupe, and compare with last night.
const open = new Map();
const fresh: Role[] = [];
for (const { name, roles } of results) {
if (roles === null) {
for (const [key, role] of known) if (role.company === name) open.set(key, role);
continue;
}
const seen = new Set(); // the same role listed twice, e.g. under two teams
for (const found of roles) {
const key = keyOf(found.url); // every url is absolute by now
const sameRole = `${found.title.trim().toLowerCase()}|${(found.location ?? '').trim().toLowerCase()}`;
if (open.has(key) || seen.has(sameRole)) continue;
seen.add(sameRole);
const role: Role = known.get(key) ?? {
company: name,
title: found.title.trim(),
location: found.location?.trim() ?? '',
workplace: found.workplace ?? 'unknown',
salaryMin: found.salaryMin ?? null,
salaryMax: found.salaryMax ?? null,
currency: found.currency ?? null,
period: found.period ?? null,
url: found.url,
firstSeen: today,
};
if (!known.has(key)) fresh.push(role);
open.set(key, role);
}
}
const closed = [...known.keys()].filter((key) => !open.has(key)).length;
const COLUMNS = ['company', 'title', 'location', 'workplace', 'salaryMin', 'salaryMax', 'currency', 'period', 'url', 'firstSeen'] as const;
const cell = (value: unknown) => `"${String(value ?? '').replaceAll('"', '""')}"`;
const csv = [COLUMNS.map(cell).join(','), ...[...open.values()].map((role) => COLUMNS.map((column) => cell(role[column])).join(','))];
await writeFile('roles.json', JSON.stringify(Object.fromEntries(open), null, 2));
await writeFile('roles.csv', `${csv.join('\n')}\n`);
await writeFile('new-roles.json', JSON.stringify(fresh, null, 2));
console.log(`${open.size} open roles at ${startups.length} startups: ${fresh.length} new tonight, ${closed} closed, ${spent.toFixed(2)} credits`);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import csv
import json
import os
import re
from datetime import date
from urllib.parse import parse_qsl, urlencode, urljoin, urlsplit
from outcrawl import ConcurrencyLimitError, NoCapacityError, Outcrawl, QuotaExceededError
# Every key on your account shares one browser slot count (Free 2, Builder 5,
# Pro 25, Scale 100). Keep LANES below yours so other work still gets a slot.
LANES = 4
NIGHT_CEILING = 2_000 # credits, per night
AGENT_BUDGET = "50.00" # about 3 minutes of a typical run
WANTED = re.compile(r"\b(platform|infrastructure|site reliability|devops|backend)\b.*\bengineer", re.I)
WANTED_WORDS = "platform, infrastructure, site reliability, DevOps or backend engineer"
NEEDS_AGENT = re.compile(r"myworkdayjobs\.com|icims\.com|taleo\.net|successfactors\.(com|eu)|oraclecloud\.com")
ROLE = {
"type": "object",
"properties": {
"title": {"type": "string"},
"location": {"type": "string"},
"workplace": {"type": "string", "enum": ["remote", "hybrid", "onsite", "unknown"]},
"salaryMin": {"type": "number", "description": "Lowest posted pay. Omit when no pay is posted."},
"salaryMax": {"type": "number", "description": "Highest posted pay. Omit when no pay is posted."},
"currency": {"type": "string", "description": "ISO 4217 code of the posted pay"},
"period": {"type": "string", "enum": ["year", "month", "hour"]},
"url": {"type": "string", "description": "Link to the posting itself"},
},
"required": ["title", "url"],
}
ROLES = {"type": "object", "properties": {"roles": {"type": "array", "items": ROLE}}, "required": ["roles"]}
COLUMNS = ["company", "title", "location", "workplace", "salaryMin", "salaryMax", "currency", "period", "url", "firstSeen"]
tally = {"spent": 0.0, "held": 0.0}
async def with_slot(call):
"""A refused call is retried; anything else is the caller's to see."""
for attempt in range(1, 9):
try:
return await call()
except (ConcurrencyLimitError, NoCapacityError):
# Refused, not queued, and no Retry-After: a slot frees when one of your calls ends.
if attempt == 8:
raise
await asyncio.sleep(20)
except QuotaExceededError as error:
# Out of monthly credits: retrying cannot help. Per-minute: wait as told.
if error.ceiling != "requests_per_minute" or attempt == 8:
raise
await asyncio.sleep(error.retry_after_seconds)
async def within_budget(maximum, step, cost):
"""Reserve a step's maximum before it starts, settle to what it really cost."""
if tally["spent"] + tally["held"] + maximum > NIGHT_CEILING:
return None
tally["held"] += maximum
actual = maximum
try:
value = await step()
actual = cost(value)
return value
finally:
tally["held"] -= maximum
tally["spent"] += actual
def key_of(url: str) -> str:
"""One posting, one key: no fragment, no tracking parameters, no trailing slash."""
parts = urlsplit(url)
query = [(k, v) for k, v in parse_qsl(parts.query) if not re.match(r"^(utm_|gh_src$|lever-source|source$|ref$)", k)]
return f"{parts.netloc.lower()}{parts.path.rstrip('/')}{'?' + urlencode(query) if query else ''}"
async def main() -> None:
today = date.today().isoformat()
known: dict[str, dict] = {}
if os.path.exists("roles.json"):
with open("roles.json") as state:
known = json.load(state)
with open("startups.csv") as source:
startups = [tuple(part.strip() for part in line.rsplit(",", 1))
for line in source.read().splitlines()[1:] if "," in line]
async with Outcrawl() as oc:
lanes = asyncio.Semaphore(LANES)
async def ask_agent(name: str, careers_url: str) -> list | None:
async def submit():
job = oc.agent(
task=(
f"Open {careers_url}, the careers site of {name}. Use its search box or filters to find "
f"every open {WANTED_WORDS} role, and page through all results. Open each posting and "
f"return its title, location, whether it is remote, hybrid or onsite, the pay range if "
f"the posting shows one, and the link to the posting."
),
caps={"budget": AGENT_BUDGET, "duration": "5m"},
schema=ROLES,
rules=["Never apply to a role, create an account or sign in."],
)
await job.submitted # a refused submit raises here
return job
async def run_agent():
# Only the submit is retried: a refusal arrives before any run exists.
job = await with_slot(submit)
return await job
run = await within_budget(float(AGENT_BUDGET), run_agent, lambda r: float(r["credits"]))
# Not run, or did not finish: tonight says nothing about this company.
if run is None or run["status"] != "completed":
return None
return (run.get("data") or {}).get("roles") or []
async def collect(name: str, careers_url: str) -> tuple[str, list | None]:
"""None means "could not read tonight": the company keeps yesterday's roles."""
async with lanes:
try:
listed: list = []
if not NEEDS_AGENT.search(careers_url):
page = await within_budget(
1,
lambda: with_slot(lambda: oc.scrape(careers_url, formats=["json"], schema=ROLES)),
lambda _: 1,
)
listed = ((page or {}).get("json") or {}).get("roles") or []
if not listed:
found = await ask_agent(name, careers_url)
if found is None:
return name, None
return name, [{**r, "url": urljoin(careers_url, r["url"])}
for r in found if WANTED.search(r["title"])]
roles = []
for role in (r for r in listed if WANTED.search(r["title"])):
url = urljoin(careers_url, role["url"])
if key_of(url) not in known:
# New tonight: read the posting itself for pay and remote policy.
posting = await within_budget(
1,
lambda: with_slot(lambda: oc.scrape(url, formats=["json"], schema=ROLE)),
lambda _: 1,
)
role.update((posting or {}).get("json") or {})
roles.append({**role, "url": url})
return name, roles
except Exception as error:
print(f"{name}: {error}")
return name, None
results = await asyncio.gather(*(collect(name, url) for name, url in startups))
# Normalise, dedupe, and compare with last night.
open_roles: dict[str, dict] = {}
fresh: list[dict] = []
for name, roles in results:
if roles is None:
open_roles.update({k: r for k, r in known.items() if r["company"] == name})
continue
seen: set[str] = set() # the same role listed twice, e.g. under two teams
for found in roles:
key = key_of(found["url"])
same_role = f"{found['title'].strip().lower()}|{(found.get('location') or '').strip().lower()}"
if key in open_roles or same_role in seen:
continue
seen.add(same_role)
role = known.get(key) or {
"company": name,
"title": found["title"].strip(),
"location": (found.get("location") or "").strip(),
"workplace": found.get("workplace") or "unknown",
"salaryMin": found.get("salaryMin"),
"salaryMax": found.get("salaryMax"),
"currency": found.get("currency"),
"period": found.get("period"),
"url": found["url"],
"firstSeen": today,
}
if key not in known:
fresh.append(role)
open_roles[key] = role
closed = sum(1 for key in known if key not in open_roles)
with open("roles.json", "w") as state:
json.dump(open_roles, state, indent=2, ensure_ascii=False)
with open("roles.csv", "w", newline="") as sheet:
writer = csv.DictWriter(sheet, fieldnames=COLUMNS, quoting=csv.QUOTE_ALL)
writer.writeheader()
writer.writerows(open_roles.values())
with open("new-roles.json", "w") as news:
json.dump(fresh, news, indent=2, ensure_ascii=False)
print(f"{len(open_roles)} open roles at {len(startups)} startups: {len(fresh)} new tonight, "
f"{closed} closed, {tally['spent']:.2f} credits")
asyncio.run(main())
```
Run it nightly. With cron, at 06:00:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
0 6 * * * cd /srv/roles && npx tsx roles.ts >> roles.log 2>&1
# or
0 6 * * * cd /srv/roles && python roles.py >> roles.log 2>&1
```
## What you get back
`roles.csv`, one row per open role, opened in any spreadsheet:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
"company","title","location","workplace","salaryMin","salaryMax","currency","period","url","firstSeen"
"Brightloom Data","Senior Platform Engineer","Berlin, Germany","hybrid","85000","105000","EUR","year","https://brightloom.example/careers/senior-platform-engineer","2026-09-18"
"Kelpie Systems","Site Reliability Engineer","Remote (US)","remote","165000","210000","USD","year","https://jobs.kelpie.example/roles/412?team=infra","2026-09-25"
"Northwind Freight","Backend Engineer, Payments","Chicago, IL","onsite","","","","","https://northwind.wd5.myworkdayjobs.com/careers/job/Chicago/Backend-Engineer_R-10442","2026-09-25"
```
`new-roles.json`, only what appeared tonight, in the same shape:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
[
{
"company": "Kelpie Systems",
"title": "Site Reliability Engineer",
"location": "Remote (US)",
"workplace": "remote",
"salaryMin": 165000,
"salaryMax": 210000,
"currency": "USD",
"period": "year",
"url": "https://jobs.kelpie.example/roles/412?team=infra",
"firstSeen": "2026-09-25"
}
]
```
And the line for your log:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
412 open roles at 300 startups: 17 new tonight, 9 closed, 1309.40 credits
```
## What it costs
Rates from [pricing](/docs/billing/pricing): a scrape is 1 credit a page. An agent run is billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in, proxy usage included. That is an estimate: the real figure depends on the task, and a page-heavy board costs more.
* Careers pages: about 255 of the 300 read as a plain scrape, **255 credits** a night.
* Postings: only new roles are read. The first night that is every matching role, about 400 credits; after that, a few dozen a night.
* Agents: about 45 startups on applicant-tracking sites, at about 90 s each, so about 23 credits a run, or **about 1,035 credits** a night.
* A typical night: **about 1,310 credits**, and the first night about 1,690, both under the 2,000 ceiling. Thirty nights is about 39,300 credits, which fits Pro's 50,000.
* Wall clock: about 255 scrapes at a few seconds and 45 agent runs at 90 s, over 4 lanes, is about 20 minutes.
## Make it yours
* Send the agent startups weekly instead of nightly: they are about 80% of the cost.
* Post `new-roles.json` to a Slack incoming webhook, one message per role.
* Swap `WANTED` for any title, and add a location filter such as `remote` or `London` in the same place.
* Keep closed roles in a second file with the date they disappeared, to see how long each role stayed open.
## Related
One URL in, structured content out.
Tasks, caps, schemas and what a run returns.
Plain-language limits the run is held to.
What `concurrency_limit` and `quota_exceeded` mean, and which are retryable.
# Give your AI app a browser
Source: https://outcrawl.ai/docs/examples/give-your-ai-app-a-browser
One tool, browse(task), that lets your own assistant use live websites and cite them.
Your assistant can reason about anything except what is on a website right now: a room rate for next weekend, a return policy, a price behind a date picker. This program gives your own tool-calling model one tool, `browse(task)`. Each call runs an Outcrawl agent in a real browser, returns a schema-shaped answer with its sources, and your model writes the reply.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* Your model provider's key. The program uses OpenAI tool calling (`OPENAI_API_KEY`); Anthropic tool use works the same way.
* `npm install @outcrawl/sdk openai` or `pip install outcrawl openai`.
## How it works
Your model sees a single function, `browse`, that takes a plain-language task. Nothing about Outcrawl leaks into your prompt.
Each call submits an [agent run](/docs/agents/overview) with a JSON schema, so every answer comes back as `{ answer, sources }` rather than free text.
Each run's `caps.budget` is 40 credits. A per-run [rule](/docs/account/rules) tells it not to sign in, buy anything or submit personal details, and the buying part is enforced.
The submit answers with a `watchUrl` you can open to watch the browser live. The finished run's `data` goes back to your model as the tool result, with its status and what it cost.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import OpenAI from 'openai';
import type { ChatCompletionMessageParam, ChatCompletionTool } from 'openai/resources/chat/completions';
import { ConcurrencyLimitError, Outcrawl } from '@outcrawl/sdk';
const openai = new OpenAI(); // reads OPENAI_API_KEY
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY
// The one tool your assistant gets.
const browseTool: ChatCompletionTool = {
type: 'function',
function: {
name: 'browse',
description:
'Do a task in a real web browser on live websites: search, open pages, pick dates, read what the page shows. ' +
'Use it when the answer depends on current information from a specific site. ' +
'Returns an answer and the URLs it came from.',
parameters: {
type: 'object',
properties: {
task: {
type: 'string',
description: 'What to do and what to report back, in one or two sentences. Name the site when you know it.',
},
},
required: ['task'],
additionalProperties: false,
},
},
};
// Every browse call answers in this shape, so the model reads the same thing each time.
const answerSchema = {
type: 'object',
properties: {
answer: { type: 'string', description: 'The answer to the task, with the specific numbers, dates and names found.' },
sources: {
type: 'array',
items: {
type: 'object',
properties: { url: { type: 'string' }, title: { type: 'string' } },
required: ['url'],
},
},
},
required: ['answer', 'sources'],
};
async function browse(task: string): Promise {
try {
const job = oc.agent({
task,
schema: answerSchema,
caps: { budget: '40.00' }, // about 2.5 minutes of a typical run
rules: ['Do not sign in, buy anything, or submit any form with personal details.'],
});
const { watchUrl } = await job.submitted;
console.log(`browse: ${task}\n watch it live: ${watchUrl ?? 'recording declined'}`);
const run = await job;
if (run.data === undefined) {
return JSON.stringify({ error: `The browser run ended ${run.status} (${run.stopReason ?? 'no reason given'}).` });
}
// `partial` and `capped` runs still carry what they found. Tell the model which it got.
return JSON.stringify({ ...(run.data as object), status: run.status, credits: run.credits });
} catch (err) {
if (err instanceof ConcurrencyLimitError) {
return JSON.stringify({ error: 'Every browser on this account is busy right now. Try again shortly.' });
}
throw err;
}
}
const messages: ChatCompletionMessageParam[] = [
{
role: 'system',
content:
'You are a travel assistant. Use the browse tool for anything that depends on live prices or availability, ' +
'and cite the URLs it returns.',
},
{
role: 'user',
content:
'Two adults at The Hoxton, Williamsburg, Friday 16 to Sunday 18 October. ' +
'What is the cheapest room I can still cancel for free, and until when?',
},
];
for (;;) {
const completion = await openai.chat.completions.create({ model: 'gpt-5', messages, tools: [browseTool] });
const message = completion.choices[0]!.message;
messages.push(message);
const calls = message.tool_calls ?? [];
if (calls.length === 0) {
console.log(message.content);
break;
}
// One call at a time: each browse holds one of your account's browser slots while it runs.
for (const call of calls) {
if (call.type !== 'function') continue;
const { task } = JSON.parse(call.function.arguments) as { task: string };
messages.push({ role: 'tool', tool_call_id: call.id, content: await browse(task) });
}
}
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import json
from openai import AsyncOpenAI
from outcrawl import ConcurrencyLimitError, Outcrawl
# The one tool your assistant gets.
BROWSE_TOOL = {
"type": "function",
"function": {
"name": "browse",
"description": (
"Do a task in a real web browser on live websites: search, open pages, pick dates, read what the page shows. "
"Use it when the answer depends on current information from a specific site. "
"Returns an answer and the URLs it came from."
),
"parameters": {
"type": "object",
"properties": {
"task": {
"type": "string",
"description": "What to do and what to report back, in one or two sentences. Name the site when you know it.",
}
},
"required": ["task"],
"additionalProperties": False,
},
},
}
# Every browse call answers in this shape, so the model reads the same thing each time.
ANSWER_SCHEMA = {
"type": "object",
"properties": {
"answer": {"type": "string", "description": "The answer to the task, with the specific numbers, dates and names found."},
"sources": {
"type": "array",
"items": {
"type": "object",
"properties": {"url": {"type": "string"}, "title": {"type": "string"}},
"required": ["url"],
},
},
},
"required": ["answer", "sources"],
}
async def browse(oc: Outcrawl, task: str) -> str:
try:
job = oc.agent(
task=task,
schema=ANSWER_SCHEMA,
caps={"budget": "40.00"}, # about 2.5 minutes of a typical run
rules=["Do not sign in, buy anything, or submit any form with personal details."],
)
submitted = await job.submitted
print(f"browse: {task}\n watch it live: {submitted.get('watchUrl', 'recording declined')}")
run = await job
except ConcurrencyLimitError:
return json.dumps({"error": "Every browser on this account is busy right now. Try again shortly."})
data = run.get("data")
if data is None:
return json.dumps({"error": f"The browser run ended {run['status']} ({run.get('stopReason', 'no reason given')})."})
# `partial` and `capped` runs still carry what they found. Tell the model which it got.
return json.dumps({**data, "status": run["status"], "credits": run["credits"]})
async def main() -> None:
llm = AsyncOpenAI() # reads OPENAI_API_KEY
async with Outcrawl() as oc: # reads OUTCRAWL_API_KEY
messages: list = [
{
"role": "system",
"content": "You are a travel assistant. Use the browse tool for anything that depends on live prices "
"or availability, and cite the URLs it returns.",
},
{
"role": "user",
"content": "Two adults at The Hoxton, Williamsburg, Friday 16 to Sunday 18 October. "
"What is the cheapest room I can still cancel for free, and until when?",
},
]
while True:
completion = await llm.chat.completions.create(model="gpt-5", messages=messages, tools=[BROWSE_TOOL])
message = completion.choices[0].message
messages.append(message)
if not message.tool_calls:
print(message.content)
break
# One call at a time: each browse holds one of your account's browser slots while it runs.
for call in message.tool_calls:
if call.type != "function":
continue
task = json.loads(call.function.arguments)["task"]
messages.append({"role": "tool", "tool_call_id": call.id, "content": await browse(oc, task)})
asyncio.run(main())
```
A tool call that fails still answers the model. A run that ends without data returns its status and stop reason; a full account returns "busy" instead of throwing. `ConcurrencyLimitError` means every browser slot on your account is in use. Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot. This is why the program runs tool calls one at a time.
## What you get back
The console, while it runs:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
browse: On thehoxton.com, find rooms at The Hoxton, Williamsburg for 2 adults, 16 to 18 October, and report the cheapest rate with free cancellation and its cancellation deadline.
watch it live: https://outcrawl.ai/play/ses_01K5ZQ7M3W
```
The tool result your model receives (a sample; the figures are illustrative):
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"answer": "Cheapest rate with free cancellation: Cosy King, Flexible rate, $289 a night, $578 for both nights before tax. Free cancellation until 3 pm on Wednesday 14 October, after which the first night is charged. The same room is $259 a night on the non-refundable rate.",
"sources": [
{ "url": "https://thehoxton.com/williamsburg/rooms/", "title": "Rooms | The Hoxton, Williamsburg" }
],
"status": "completed",
"credits": "15.50"
}
```
And the reply your model writes from it:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
The cheapest room you can cancel for free is the Cosy King on the Flexible rate: $289 a night, $578 for the two nights before tax. You can cancel without charge until 3 pm on Wednesday 14 October. (Source: thehoxton.com/williamsburg/rooms)
```
## What it costs
An agent run is billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in, proxy usage included. A lookup like this one takes about 60 seconds: **about 15 credits per browse call**, with a `caps.budget` of 40 for each call. That is an estimate: the real figure depends on the task, and how much a page weighs varies.
A Builder plan's 10,000 monthly credits covers about 650 browse calls at that size. The model in your own app is billed by your model provider, not by Outcrawl.
## Make it yours
* Run several tool calls at once with `Promise.all` or `asyncio.gather`, keeping the number below your plan's concurrent browsers.
* Add a `profile` to the agent call so `browse` works on sites where your user is signed in.
* Stream the run to your UI: `for await (const record of job)` yields each step as it happens.
* Give the model a second tool, `read(url)`, backed by `oc.scrape` at 1 credit a page, for pages that need no interaction.
## Related
Schemas, caps and statuses.
The watch link and the event stream.
What a run is allowed to do.
Every rate on this page.
# Run a long agent task from your editor
Source: https://outcrawl.ai/docs/examples/mcp-agent-from-editor
Submit and check on an agent run without leaving the conversation.
Hand a longer, multi-step browser task to an agent, then keep working while it runs, checking in when you want rather than waiting on it.
## Setup
Connect the MCP server first, see [setup](/docs/mcp/setup).
## What you'll do
* Submit a task with `outcrawl_agent`
* Check its status separately, it does not block
* Read what it has produced once it has something
## Example exchange
**"Find three vendors that ship the part I described and get me their pricing pages."**
The assistant calls `outcrawl_agent` with the task and a budget. It gets an id back in milliseconds, the run itself can take minutes, so the assistant should check status rather than treat the id as a failure. A little later, **"how is that going?"** calls `outcrawl_agent_status`, and once it is done, `outcrawl_agent_results` returns what it found.
## What it can't do
Once submitted, no tool call can redirect the task mid-run. `outcrawl_agent_control` steps in by taking the browser yourself, and when the run stops to ask something only you know, `outcrawl_agent_answer` answers it and the run carries on.
# Connect an integration, then let an agent use it
Source: https://outcrawl.ai/docs/examples/mcp-connect-integration
Two steps: a human connects it, a run is granted access to it.
Give an agent reach beyond the browser, sending a Slack message, sending an email, by connecting an integration first and then naming it when you submit a run.
## Setup
Connecting an integration is a human-only step, it has no MCP tool. Do it from the console or the CLI first:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
outcrawl integrations connect slack --url https://mcp.slack.example --secret
```
Then connect the MCP server as usual for the agent side, see [setup](/docs/mcp/setup).
## What you'll do
* Connect a Slack or Gmail-style integration from the console or CLI
* Submit an agent run that names it explicitly
* Let the agent call that integration's tools mid-task
## Example exchange
**"Research this topic and post a summary to #research on Slack when you're done."**
Because the Slack integration was already connected and this run named it at submit time, the agent's task can end with a message posted through it. If the run had not been given that integration explicitly, it would have no reach to Slack at all, connecting a server to the workspace is not the same as granting one run access to it.
## What it can't do
An assistant cannot connect the integration itself, no MCP tool exists for `integrations.connect`, `list` or `delete`. That consent step is a person's alone, every time.
# Collect structured info from a directory site
Source: https://outcrawl.ai/docs/examples/mcp-lead-extraction
Crawl a listing site and read its pages as structured content.
Point your assistant at a directory-style site, a listings page, a member roster, a catalogue, and have it walk every entry.
## Setup
Connect the MCP server first, see [setup](/docs/mcp/setup).
## What you'll do
* Crawl a site from a seed URL
* Read the returned pages as clean markdown rather than raw HTML
* Have the assistant pull out the fields you care about from what it read
## Example exchange
**"Go through this directory site and pull out every listing's name, location and contact page."**
The assistant calls `outcrawl_crawl` with the directory's seed URL. `outcrawl_crawl` folds the whole stream into one response, so the assistant gets every page's content back from that single call rather than polling a separate results tool, there is no `outcrawl_crawl_results` tool, only the REST API and CLI expose the id-based lifecycle. It then reads the fields you asked for out of the returned pages.
## What it can't do
Each page crawled is 1 credit, the same as a direct scrape. There is no separate "extraction" capability with its own price, this is plain crawling plus reading the structured content already returned.
# Monitor a competitor from your editor
Source: https://outcrawl.ai/docs/examples/mcp-monitor-competitor
Set up a page watch without leaving your assistant.
Ask your assistant to watch a competitor's pricing page and tell you when it changes, without opening the console.
## Setup
Connect the MCP server first, see [setup](/docs/mcp/setup) for the full `.mcp.json` block.
## What you'll do
* Create a monitor on a URL with a schedule
* Choose Discord, Slack or a webhook for notification
* List existing monitors to check what is already being watched
## Example exchange
**"Watch [https://competitor.example/pricing](https://competitor.example/pricing) every hour and post to our #competitors Slack channel if it changes."**
The assistant calls `outcrawl_monitor_create` with the URL, a schedule, and the Slack target. It answers with the monitor's id, for example something like *"Monitor created, checking hourly, will post to #competitors on a real change."*
**"What am I currently watching?"**
The assistant calls `outcrawl_monitor_list` and summarizes each monitor's URL, schedule and last check status.
## What it can't do
Notifications are not retried. If the Slack endpoint is briefly unreachable when a change fires, that one notification is lost, though the change itself stays readable afterward from the monitor's own list entry.
# A research assistant in your editor
Source: https://outcrawl.ai/docs/examples/mcp-research-assistant
Search and scrape together to build a brief.
Ask your assistant to research a topic and it can search, then read the pages worth reading, all inside the conversation.
## Setup
Connect the MCP server first, see [setup](/docs/mcp/setup).
## What you'll do
* Search for a topic
* Scrape the pages worth reading in full
* Have the assistant synthesise a short brief from what it read
## Example exchange
**"Look into what people are saying about the new release and summarise it."**
The assistant calls `outcrawl_search` for the topic, picks a handful of promising results, then calls `outcrawl_scrape` on each to read the full content rather than just a snippet. It comes back with something like *"Three sources cover this: two are broadly positive about performance, one raises a compatibility concern. Here is a short summary of each."*
## What it can't do
Search results are pro rata on what is actually returned, five credits per ten results, and each page scraped afterward is billed separately at one credit per page. A broad research pass across many sources adds up the same way it would if you called both routes yourself.
# Pull a page into your coding assistant's context
Source: https://outcrawl.ai/docs/examples/mcp-scrape-into-context
Reference documentation without leaving your editor.
Bring a page's clean content directly into a coding assistant's context window while you build against it, instead of switching to a browser tab and copying by hand.
## Setup
Connect the MCP server first, see [setup](/docs/mcp/setup).
## What you'll do
* Ask the assistant to fetch one or a few reference pages
* Keep building with that content already in context
## Example exchange
**"Pull in the authentication section of this API's docs before we wire up the client."**
The assistant calls `outcrawl_scrape` on the page and reads the markdown back directly, no separate download step, no copy and paste, no HTML to strip by hand.
## What it can't do
This is one page at a time. For an entire docs site, [keeping a docs knowledge base in sync](/docs/examples/docs-knowledge-base) or the [lead extraction](/docs/examples/mcp-lead-extraction) pattern below is the better fit than repeated single scrapes.
# Price the same basket in 20 countries
Source: https://outcrawl.ai/docs/examples/price-a-basket-in-20-countries
Twenty agents, each browsing from a different country, return what a local customer pays, and your program builds the comparison.
A streaming plan, a phone and a software subscription cost different amounts in Tokyo, Istanbul and Oslo, and the only way to see the local price is to visit the site from that country. By hand that is twenty VPN hops, four sites each, and a spreadsheet of currencies. This program sends one agent per country, each browsing from a profile pinned to that country, and turns their structured answers into one table in US dollars.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* A plan with enough browsers for the lanes you choose: Free 2, Builder 5, Pro 25, Scale 100. Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot, and an agent run holds its slot until it ends. The program runs 4 at a time, which fits Builder.
* Node 18 or later for TypeScript, or Python 3.10 or later. Exchange rates come from the free, keyless Frankfurter API (ECB reference rates).
## How it works
[Profiles](/docs/account/profiles) are created once with an `exit` country and reused on every run. The browser's IP, timezone and language follow that exit, so each site serves its local store. Cookie banners dismissed on the first run stay dismissed.
Each [agent run](/docs/agents/overview) gets the same four items and a JSON Schema. The run is judged against the schema, and `run.data` comes back in that shape: item, price, currency, billing period, source URL.
Over your account's browser count, a submit is refused with `ConcurrencyLimitError`, never queued. The program runs 4 lanes, waits 20 seconds and retries a refused submit, and waits the stated `retryAfterSeconds` on a per-minute `QuotaExceededError`. See [errors](/docs/api/errors).
Each run has a hard `caps.budget`. Before a run starts, the program reserves that budget against a batch ceiling and releases it to the real cost when the run ends, so the batch cannot start a run it could not afford.
The program converts every local price to US dollars at today's reference rate, prints a table and writes `basket-prices.json`.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { writeFile } from 'node:fs/promises';
import {
ConcurrencyLimitError,
NoCapacityError,
Outcrawl,
QuotaExceededError,
} from '@outcrawl/sdk';
const oc = new Outcrawl();
// Every key on your account shares one browser slot count (Free 2, Builder 5,
// Pro 25, Scale 100). Keep LANES below yours so other work still gets a slot.
const LANES = 4;
const RUN_BUDGET = '100.00'; // per country: about 6.5 minutes of a typical run
const BATCH_CEILING = 1_500; // credits, across the whole batch
const BASKET = [
{ id: 'netflix-standard', ask: 'Netflix Standard plan, billed monthly, on netflix.com' },
{ id: 'spotify-premium', ask: 'Spotify Premium Individual, billed monthly, on spotify.com' },
{ id: 'iphone-16-128gb', ask: 'iPhone 16 with 128 GB, bought outright, on apple.com' },
{ id: 'm365-personal', ask: 'Microsoft 365 Personal, billed yearly, on microsoft.com' },
];
const COUNTRIES: [code: string, name: string][] = [
['us', 'United States'], ['ca', 'Canada'], ['mx', 'Mexico'], ['br', 'Brazil'],
['gb', 'United Kingdom'], ['de', 'Germany'], ['fr', 'France'], ['es', 'Spain'],
['se', 'Sweden'], ['no', 'Norway'], ['pl', 'Poland'], ['ch', 'Switzerland'],
['tr', 'Türkiye'], ['za', 'South Africa'], ['in', 'India'], ['jp', 'Japan'],
['kr', 'South Korea'], ['sg', 'Singapore'], ['au', 'Australia'], ['nz', 'New Zealand'],
];
const SCHEMA = {
type: 'object',
properties: {
items: {
type: 'array',
items: {
type: 'object',
properties: {
item: { type: 'string', enum: BASKET.map((b) => b.id) },
available: { type: 'boolean' },
price: { type: 'number', description: 'The number shown, tax as the page shows it' },
currency: { type: 'string', description: 'ISO 4217 code, e.g. JPY' },
period: { type: 'string', enum: ['month', 'year', 'one-off'] },
url: { type: 'string' },
},
required: ['item', 'available'],
},
},
},
required: ['items'],
};
interface Price {
item: string;
available: boolean;
price?: number;
currency?: string;
period?: string;
url?: string;
}
interface Row {
country: string;
name: string;
prices: Price[];
credits: number;
problem?: string;
}
const sleep = (ms: number) => new Promise((wake) => setTimeout(wake, ms));
// A refused call is retried; anything else is the caller's to see.
async function withSlot(call: () => Promise): Promise {
for (let attempt = 1; ; attempt++) {
try {
return await call();
} catch (error) {
if (attempt === 8) throw error;
if (error instanceof ConcurrencyLimitError || error instanceof NoCapacityError) {
// Refused, not queued, and no Retry-After: a slot frees when one of your runs ends.
await sleep(20_000);
} else if (error instanceof QuotaExceededError && error.ceiling === 'requests_per_minute') {
await sleep(error.retryAfterSeconds * 1_000);
} else {
throw error; // out of monthly credits, or a bad request: retrying cannot help
}
}
}
}
// At most LANES items in flight: each lane takes the next item when it is free.
async function inLanes(items: readonly T[], work: (item: T) => Promise): Promise {
const results: R[] = [];
let next = 0;
const lane = async () => {
while (next < items.length) {
const index = next++;
results[index] = await work(items[index] as T);
}
};
await Promise.all(Array.from({ length: Math.min(LANES, items.length) }, lane));
return results;
}
// Reserve a run's cap before it starts; settle to what it really cost.
let spent = 0;
let reserved = 0;
// 1. One profile per country, created once and reused on every run.
const profiles = new Map((await oc.profiles.list({ label: 'basket-*', limit: 100 })).map((p) => [p.label, p.id]));
for (const [code] of COUNTRIES) {
const label = `basket-${code}`;
if (!profiles.has(label)) profiles.set(label, (await oc.profiles.create({ label, exit: { country: code } })).id);
}
// 2. One agent per country.
async function priceCountry([code, name]: [string, string]): Promise {
const cap = Number(RUN_BUDGET);
if (spent + reserved + cap > BATCH_CEILING) {
return { country: code, name, prices: [], credits: 0, problem: 'skipped: batch ceiling reached' };
}
reserved += cap;
let credits = 0;
try {
// Only the submit is retried: a refusal arrives before any run exists.
const { job } = await withSlot(async () => {
const job = oc.agent({
task:
`You are a new customer in ${name}. For each item below, open the official site as it ` +
`is served in ${name} and find today's price for this country, in the local currency, ` +
`as the page shows it. If an item is not sold in ${name}, set available to false.\n` +
BASKET.map((b) => `- ${b.id}: ${b.ask}`).join('\n'),
caps: { budget: RUN_BUDGET, duration: '8m' },
schema: SCHEMA,
profile: profiles.get(`basket-${code}`),
rules: ['Never sign in, create an account, or start a checkout.'],
});
await job.submitted; // a refused submit throws here
return { job }; // wrapped, because a job is awaitable and returning it bare waits for the run
});
const run = await job; // the lane holds its slot until the run ends
credits = Number(run.credits);
const data = run.data as { items?: Price[] } | undefined;
return {
country: code,
name,
prices: data?.items ?? [],
credits,
problem: run.status === 'completed' ? undefined : `${run.status} (${run.stopReason ?? run.error ?? 'no reason'})`,
};
} catch (error) {
return { country: code, name, prices: [], credits, problem: String(error) };
} finally {
reserved -= cap;
spent += credits;
}
}
const rows = await inLanes(COUNTRIES, priceCountry);
// 3. One table, in US dollars at today's ECB reference rate.
const fx = (await (await fetch('https://api.frankfurter.dev/v1/latest?base=USD')).json()) as {
date: string;
rates: Record;
};
const toUsd = (p: Price): number | undefined => {
if (!p.available || p.price === undefined || p.currency === undefined) return undefined;
const rate = p.currency === 'USD' ? 1 : fx.rates[p.currency];
return rate === undefined ? undefined : p.price / rate;
};
console.log(`| Country | ${BASKET.map((b) => b.id).join(' | ')} |`);
console.log(`| --- |${BASKET.map(() => ' --- |').join('')}`);
for (const row of rows) {
const cells = BASKET.map((b) => {
const p = row.prices.find((price) => price.item === b.id);
const usd = p === undefined ? undefined : toUsd(p);
if (p === undefined || !p.available) return 'not sold';
return usd === undefined ? `${p.price} ${p.currency}` : `$${usd.toFixed(2)} (${p.price} ${p.currency})`;
});
console.log(`| ${row.name} | ${row.problem ?? cells.join(' | ')} |`);
}
console.log(`\n${rows.length} countries, ${spent.toFixed(2)} credits, rates of ${fx.date}`);
await writeFile('basket-prices.json', JSON.stringify({ rates: fx.date, rows }, null, 2));
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import json
import httpx
from outcrawl import ConcurrencyLimitError, NoCapacityError, Outcrawl, QuotaExceededError
# Every key on your account shares one browser slot count (Free 2, Builder 5,
# Pro 25, Scale 100). Keep LANES below yours so other work still gets a slot.
LANES = 4
RUN_BUDGET = "100.00" # per country: about 6.5 minutes of a typical run
BATCH_CEILING = 1_500 # credits, across the whole batch
BASKET = [
("netflix-standard", "Netflix Standard plan, billed monthly, on netflix.com"),
("spotify-premium", "Spotify Premium Individual, billed monthly, on spotify.com"),
("iphone-16-128gb", "iPhone 16 with 128 GB, bought outright, on apple.com"),
("m365-personal", "Microsoft 365 Personal, billed yearly, on microsoft.com"),
]
COUNTRIES = [
("us", "United States"), ("ca", "Canada"), ("mx", "Mexico"), ("br", "Brazil"),
("gb", "United Kingdom"), ("de", "Germany"), ("fr", "France"), ("es", "Spain"),
("se", "Sweden"), ("no", "Norway"), ("pl", "Poland"), ("ch", "Switzerland"),
("tr", "Türkiye"), ("za", "South Africa"), ("in", "India"), ("jp", "Japan"),
("kr", "South Korea"), ("sg", "Singapore"), ("au", "Australia"), ("nz", "New Zealand"),
]
SCHEMA = {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"item": {"type": "string", "enum": [item for item, _ in BASKET]},
"available": {"type": "boolean"},
"price": {"type": "number", "description": "The number shown, tax as the page shows it"},
"currency": {"type": "string", "description": "ISO 4217 code, e.g. JPY"},
"period": {"type": "string", "enum": ["month", "year", "one-off"]},
"url": {"type": "string"},
},
"required": ["item", "available"],
},
}
},
"required": ["items"],
}
async def with_slot(call):
"""A refused call is retried; anything else is the caller's to see."""
for attempt in range(1, 9):
try:
return await call()
except (ConcurrencyLimitError, NoCapacityError):
# Refused, not queued, and no Retry-After: a slot frees when one of your runs ends.
if attempt == 8:
raise
await asyncio.sleep(20)
except QuotaExceededError as error:
# Out of monthly credits: retrying cannot help. Per-minute: wait as told.
if error.ceiling != "requests_per_minute" or attempt == 8:
raise
await asyncio.sleep(error.retry_after_seconds)
async def main() -> None:
async with Outcrawl() as oc:
# 1. One profile per country, created once and reused on every run.
profiles = {p["label"]: p["id"] for p in await oc.profiles.list(label="basket-*", limit=100)}
for code, _ in COUNTRIES:
label = f"basket-{code}"
if label not in profiles:
profiles[label] = (await oc.profiles.create(label=label, exit={"country": code}))["id"]
lanes = asyncio.Semaphore(LANES)
tally = {"spent": 0.0, "reserved": 0.0} # reserve a run's cap; settle to its real cost
# 2. One agent per country.
async def price_country(code: str, name: str) -> dict:
async with lanes:
cap = float(RUN_BUDGET)
if tally["spent"] + tally["reserved"] + cap > BATCH_CEILING:
return {"country": code, "name": name, "prices": [], "credits": 0,
"problem": "skipped: batch ceiling reached"}
tally["reserved"] += cap
credits = 0.0
try:
async def submit():
job = oc.agent(
task=(
f"You are a new customer in {name}. For each item below, open the official "
f"site as it is served in {name} and find today's price for this country, "
f"in the local currency, as the page shows it. If an item is not sold in "
f"{name}, set available to false.\n"
+ "\n".join(f"- {item}: {ask}" for item, ask in BASKET)
),
caps={"budget": RUN_BUDGET, "duration": "8m"},
schema=SCHEMA,
profile=profiles[f"basket-{code}"],
rules=["Never sign in, create an account, or start a checkout."],
)
await job.submitted # a refused submit raises here
return job
# Only the submit is retried: a refusal arrives before any run exists.
job = await with_slot(submit)
run = await job # the lane holds its slot until the run ends
credits = float(run["credits"])
data = run.get("data") or {}
problem = None
if run["status"] != "completed":
problem = f"{run['status']} ({run.get('stopReason') or run.get('error') or 'no reason'})"
return {"country": code, "name": name, "prices": data.get("items", []),
"credits": credits, "problem": problem}
except Exception as error:
return {"country": code, "name": name, "prices": [], "credits": credits, "problem": str(error)}
finally:
tally["reserved"] -= cap
tally["spent"] += credits
rows = await asyncio.gather(*(price_country(code, name) for code, name in COUNTRIES))
# 3. One table, in US dollars at today's ECB reference rate.
async with httpx.AsyncClient() as http:
fx = (await http.get("https://api.frankfurter.dev/v1/latest", params={"base": "USD"})).json()
def cell(price: dict | None) -> str:
if price is None or not price.get("available"):
return "not sold"
amount, currency = price.get("price"), price.get("currency")
rate = 1 if currency == "USD" else fx["rates"].get(currency)
if amount is None or rate is None:
return f"{amount} {currency}"
return f"${amount / rate:.2f} ({amount} {currency})"
print("| Country | " + " | ".join(item for item, _ in BASKET) + " |")
print("| --- |" + " --- |" * len(BASKET))
for row in rows:
by_item = {p["item"]: p for p in row["prices"]}
cells = " | ".join(cell(by_item.get(item)) for item, _ in BASKET)
print(f"| {row['name']} | {row['problem'] or cells} |")
print(f"\n{len(rows)} countries, {tally['spent']:.2f} credits, rates of {fx['date']}")
with open("basket-prices.json", "w") as out:
json.dump({"rates": fx["date"], "rows": rows}, out, indent=2, ensure_ascii=False)
asyncio.run(main())
```
A refusal arrives on the submit, before any run exists, so the retry wraps the submit and nothing else. A retry never starts a second run for the same country. Once a run exists it holds its slot until it ends, and the lane waits for it.
## What you get back
The printed table (rows shortened, prices illustrative):
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
| Country | netflix-standard | spotify-premium | iphone-16-128gb | m365-personal |
| --- | --- | --- | --- | --- |
| United States | $17.99 (17.99 USD) | $11.99 (11.99 USD) | $799.00 (799 USD) | $99.99 (99.99 USD) |
| United Kingdom | $17.17 (12.99 GBP) | $15.85 (11.99 GBP) | $1056.25 (799 GBP) | $112.35 (84.99 GBP) |
| Türkiye | $3.89 (189.99 TRY) | $2.05 (99.99 TRY) | $1330.45 (64999 TRY) | not sold |
| Japan | $10.01 (1590 JPY) | $6.80 (1080 JPY) | $785.65 (124800 JPY) | $135.35 (21500 JPY) |
| India | $5.20 (499 INR) | $1.24 (119 INR) | $832.64 (79900 INR) | $60.43 (5799 INR) |
| Norway | capped (max-budget) |
20 countries, 629.00 credits, rates of 2026-09-24
```
And `basket-prices.json`, one row per country in the schema's shape:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"rates": "2026-09-24",
"rows": [
{
"country": "jp",
"name": "Japan",
"prices": [
{ "item": "netflix-standard", "available": true, "price": 1590, "currency": "JPY", "period": "month", "url": "https://www.netflix.com/jp-en/" },
{ "item": "spotify-premium", "available": true, "price": 1080, "currency": "JPY", "period": "month", "url": "https://www.spotify.com/jp/premium/" },
{ "item": "iphone-16-128gb", "available": true, "price": 124800, "currency": "JPY", "period": "one-off", "url": "https://www.apple.com/jp/shop/buy-iphone/iphone-16" },
{ "item": "m365-personal", "available": true, "price": 21500, "currency": "JPY", "period": "year", "url": "https://www.microsoft.com/ja-jp/microsoft-365/microsoft-365-personal" }
],
"credits": 31.0
}
]
}
```
A country the fleet has no exit for, or a run that hit its cap, comes back with `problem` set and no prices. The rest of the table is unaffected.
## What it costs
An agent run is billed for the model's usage, browser time and proxy usage ([pricing](/docs/billing/pricing)). A typical run costs about 15 credits a minute all in, proxy usage included; the real figure depends on the task, and a page-heavy site costs more.
* One country: about 2 minutes across the four sites, so **about 30 credits**.
* Twenty countries: **about 600 credits**, which the batch ceiling of 1,500 bounds.
* Wall clock: 20 runs over 4 lanes is 5 rounds of about 2 minutes, so about 10 minutes. With 20 lanes on Pro it is one round.
## Make it yours
* Swap the basket for flights, hotel nights or car rentals, and add a `date` field to the schema.
* Pin a city, not only a country: `exit: { country: 'us', region: 'texas', city: 'austin' }`.
* Run it weekly and diff `basket-prices.json` against last week's to catch a price change in one market.
* Set `LANES` from your plan: 4 on Builder, 20 on Pro, 90 on Scale, always below the plan's count.
## Related
Tasks, caps, schemas and what a run returns.
Saved logins and a pinned geography, reused across runs.
Plain-language limits the run is held to.
What `concurrency_limit` and `quota_exceeded` mean, and which are retryable.
# Sign up to your own product every hour, from three countries
Source: https://outcrawl.ai/docs/examples/qa-your-signup-flow-hourly
Agents run your real signup flow, email verification included, and file a GitHub issue with the replay when a step breaks.
Signup breaks quietly: a verification email that stops arriving, a form that rejects a Brazilian phone number, a consent banner in Germany that covers the submit button. Unit tests do not catch it and nobody on the team signs up to their own product every hour. This program does, from three countries at once, with a fresh inbox each time, and files one GitHub issue per broken step with the replay of the run attached.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* An inbox connected as the `agentmail` [integration](/docs/account/integrations), so the agent can create a throwaway address and read the verification email. Any inbox service with an MCP server works; change the name in the code.
* A GitHub token that can write issues, in `GITHUB_TOKEN`, and your repository as `owner/name` in `GITHUB_REPO`.
* Something that runs a command every hour: cron, a GitHub Actions schedule (shown below), or your own job runner. Outcrawl schedules [monitors](/docs/monitors/overview), which watch a URL; a task like this one is started by your code.
## How it works
Once an hour the program submits one [agent run](/docs/agents/overview) per country. Two run at a time, below your plan's browser count, and a run refused with `ConcurrencyLimitError` waits 20 seconds for a slot and tries again.
`exit: { country }` sends the run through a residential exit in that country, so it sees the language, currency, consent banners and phone formats a local customer sees.
The run may call the `agentmail` [integration](/docs/account/integrations) because it is named in `integrations`. It creates a new inbox, signs up with it, waits for the verification email and opens the link. [Rules](/docs/account/rules) keep it on your domain and stop it paying for anything.
A [schema](/docs/agents/overview#typed-results) makes every run answer the same shape: each step, pass or fail, how long it took, and the exact error text the page showed.
A failed step files a GitHub issue with the run's [replay](/docs/browsers/sessions) link. The same step failing again from the same country adds a comment to that issue instead of opening another.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { setTimeout as sleep } from 'node:timers/promises';
import { ConcurrencyLimitError, Outcrawl, QuotaExceededError } from '@outcrawl/sdk';
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY
const SIGNUP_URL = 'https://app.yourproduct.com/signup';
const COUNTRIES = ['us', 'de', 'br'];
const PARALLEL = 2; // below your plan's browser count, so other work still gets a slot
const REPO = process.env.GITHUB_REPO ?? 'your-org/your-app';
const GITHUB_TOKEN = process.env.GITHUB_TOKEN ?? '';
const STEPS = ['open-signup', 'submit-form', 'verification-email', 'verify-link', 'onboarding', 'dashboard'];
interface SignupCheck {
reachedDashboard: boolean;
email: string;
failedStep: string; // one of STEPS, or 'none'
pageError: string; // the error text the page showed, '' when none
steps: { name: string; ok: boolean; seconds: number; note: string }[];
}
const schema = {
type: 'object',
properties: {
reachedDashboard: { type: 'boolean' },
email: { type: 'string', description: 'The address you signed up with.' },
failedStep: { type: 'string', enum: [...STEPS, 'none'] },
pageError: { type: 'string', description: "The exact error text on the page, or '' when none." },
steps: {
type: 'array',
items: {
type: 'object',
properties: {
name: { type: 'string', enum: STEPS },
ok: { type: 'boolean' },
seconds: { type: 'number' },
note: { type: 'string' },
},
required: ['name', 'ok', 'seconds', 'note'],
},
},
},
required: ['reachedDashboard', 'email', 'failedStep', 'pageError', 'steps'],
};
const task = `Test the signup flow of our product as a brand new customer would.
open-signup: create a new inbox with the agentmail tools, then open ${SIGNUP_URL}.
submit-form: sign up with that inbox address, the name "Signup Check" and a strong password you make up.
verification-email: wait up to 3 minutes for the verification email in that inbox.
verify-link: open the link in that email.
onboarding: finish onboarding with the first option offered at every choice.
dashboard: stop when the product dashboard is showing.
Record each step with how long it took. At the first step that fails, stop, record the exact error text on the page, and set failedStep to that step.`;
async function checkFrom(country: string) {
for (;;) {
const job = oc.agent({
task,
schema,
caps: { budget: '100.00', duration: '6m' },
exit: { country }, // a residential exit in that country
integrations: ['agentmail'],
rules: ['Only use yourproduct.com.', 'Never make a payment.'],
});
try {
const { watchUrl } = await job.submitted;
const run = await job;
return { country, run, replay: run.replayUrl ?? watchUrl ?? '(no replay)' };
} catch (err) {
// Over your browser count: refused, not queued. A slot frees when one of your runs ends.
if (err instanceof ConcurrencyLimitError) await sleep(20_000);
// Over requests per minute: wait as long as the error says. Out of monthly credits: stop.
else if (err instanceof QuotaExceededError && err.ceiling === 'requests_per_minute') await sleep(err.retryAfterSeconds * 1000);
else throw err;
}
}
}
// Run `work` over every item, never more than `limit` at once.
async function inLanes(items: T[], limit: number, work: (item: T) => Promise): Promise {
const out: R[] = [];
let next = 0;
const lane = async () => {
while (next < items.length) {
const i = next++;
out[i] = await work(items[i]!);
}
};
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, lane));
return out;
}
async function github(path: string, method = 'GET', body?: unknown): Promise {
const res = await fetch(`https://api.github.com/repos/${REPO}${path}`, {
method,
headers: {
Authorization: `Bearer ${GITHUB_TOKEN}`,
Accept: 'application/vnd.github+json',
'Content-Type': 'application/json',
},
body: body === undefined ? undefined : JSON.stringify(body),
});
if (!res.ok) throw new Error(`GitHub ${res.status}: ${await res.text()}`);
return res.json();
}
const results = await inLanes(COUNTRIES, PARALLEL, checkFrom);
let failures = 0;
for (const { country, run, replay } of results) {
const check = run.data as SignupCheck | undefined;
if (run.status === 'completed' && check?.reachedDashboard === true) {
console.log(`${country}: ok in ${Math.round(run.durationMs / 1000)} s, ${run.credits} credits`);
continue; // success stays quiet
}
failures++;
const step = check?.failedStep && check.failedStep !== 'none' ? check.failedStep : `run ${run.stopReason ?? run.status}`;
const title = `Signup fails from ${country.toUpperCase()} at ${step}`;
const body = [
`**Failed step:** \`${step}\``,
`**Country:** ${country.toUpperCase()}`,
`**Page said:** ${check?.pageError ? `"${check.pageError}"` : 'nothing'}`,
`**Replay:** ${replay}`,
`**Run:** \`${run.id}\`, ${run.status}, ${run.credits} credits, ${new Date().toISOString()}`,
'',
'| Step | Result | Seconds | Note |',
'| --- | --- | --- | --- |',
...(check?.steps ?? []).map((s) => `| ${s.name} | ${s.ok ? 'pass' : 'FAIL'} | ${s.seconds} | ${s.note} |`),
].join('\n');
// One open issue per country and step: a repeat failure adds a comment, not a new issue.
const open = (await github('/issues?labels=signup-qa&state=open&per_page=100')) as { number: number; title: string }[];
const known = open.find((issue) => issue.title === title);
if (known) await github(`/issues/${known.number}/comments`, 'POST', { body });
else await github('/issues', 'POST', { title, body, labels: ['signup-qa'] });
console.log(`${country}: FAILED at ${step}, ${known ? `commented on #${known.number}` : 'filed an issue'}`);
}
process.exitCode = failures > 0 ? 1 : 0;
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import os
from datetime import datetime, timezone
import httpx
from outcrawl import ConcurrencyLimitError, Outcrawl, QuotaExceededError
SIGNUP_URL = "https://app.yourproduct.com/signup"
COUNTRIES = ["us", "de", "br"]
PARALLEL = 2 # below your plan's browser count, so other work still gets a slot
REPO = os.environ.get("GITHUB_REPO", "your-org/your-app")
GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "")
STEPS = ["open-signup", "submit-form", "verification-email", "verify-link", "onboarding", "dashboard"]
SCHEMA = {
"type": "object",
"properties": {
"reachedDashboard": {"type": "boolean"},
"email": {"type": "string", "description": "The address you signed up with."},
"failedStep": {"type": "string", "enum": [*STEPS, "none"]},
"pageError": {"type": "string", "description": "The exact error text on the page, or '' when none."},
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string", "enum": STEPS},
"ok": {"type": "boolean"},
"seconds": {"type": "number"},
"note": {"type": "string"},
},
"required": ["name", "ok", "seconds", "note"],
},
},
},
"required": ["reachedDashboard", "email", "failedStep", "pageError", "steps"],
}
TASK = f"""Test the signup flow of our product as a brand new customer would.
open-signup: create a new inbox with the agentmail tools, then open {SIGNUP_URL}.
submit-form: sign up with that inbox address, the name "Signup Check" and a strong password you make up.
verification-email: wait up to 3 minutes for the verification email in that inbox.
verify-link: open the link in that email.
onboarding: finish onboarding with the first option offered at every choice.
dashboard: stop when the product dashboard is showing.
Record each step with how long it took. At the first step that fails, stop, record the exact error text on the page, and set failedStep to that step."""
async def check_from(oc: Outcrawl, lanes: asyncio.Semaphore, country: str) -> tuple[str, dict, str]:
async with lanes: # never more than PARALLEL runs at once
while True:
job = oc.agent(
task=TASK,
schema=SCHEMA,
caps={"budget": "100.00", "duration": "6m"},
exit={"country": country}, # a residential exit in that country
integrations=["agentmail"],
rules=["Only use yourproduct.com.", "Never make a payment."],
)
try:
submitted = await job.submitted
run = await job
return country, run, run.get("replayUrl") or submitted.get("watchUrl") or "(no replay)"
except ConcurrencyLimitError:
# Over your browser count: refused, not queued. A slot frees when one of your runs ends.
await asyncio.sleep(20)
except QuotaExceededError as err:
# Over requests per minute: wait as long as the error says. Out of monthly credits: stop.
if err.ceiling != "requests_per_minute":
raise
await asyncio.sleep(err.retry_after_seconds)
async def report(gh: httpx.AsyncClient, country: str, run: dict, replay: str) -> str:
check = run.get("data") or {}
failed = check.get("failedStep")
step = failed if failed and failed != "none" else f"run {run.get('stopReason') or run['status']}"
title = f"Signup fails from {country.upper()} at {step}"
page_said = f'"{check["pageError"]}"' if check.get("pageError") else "nothing"
rows = [f"| {s['name']} | {'pass' if s['ok'] else 'FAIL'} | {s['seconds']} | {s['note']} |" for s in check.get("steps", [])]
body = "\n".join([
f"**Failed step:** `{step}`",
f"**Country:** {country.upper()}",
f"**Page said:** {page_said}",
f"**Replay:** {replay}",
f"**Run:** `{run['id']}`, {run['status']}, {run['credits']} credits, {datetime.now(timezone.utc).isoformat()}",
"",
"| Step | Result | Seconds | Note |",
"| --- | --- | --- | --- |",
*rows,
])
# One open issue per country and step: a repeat failure adds a comment, not a new issue.
listed = await gh.get("/issues", params={"labels": "signup-qa", "state": "open", "per_page": 100})
known = next((i for i in listed.raise_for_status().json() if i["title"] == title), None)
if known:
(await gh.post(f"/issues/{known['number']}/comments", json={"body": body})).raise_for_status()
return f"FAILED at {step}, commented on #{known['number']}"
(await gh.post("/issues", json={"title": title, "body": body, "labels": ["signup-qa"]})).raise_for_status()
return f"FAILED at {step}, filed an issue"
async def main() -> int:
github = httpx.AsyncClient(
base_url=f"https://api.github.com/repos/{REPO}",
headers={"Authorization": f"Bearer {GITHUB_TOKEN}", "Accept": "application/vnd.github+json"},
)
async with Outcrawl() as oc, github as gh: # Outcrawl() reads OUTCRAWL_API_KEY
lanes = asyncio.Semaphore(PARALLEL)
results = await asyncio.gather(*(check_from(oc, lanes, c) for c in COUNTRIES))
failures = 0
for country, run, replay in results:
check = run.get("data") or {}
if run["status"] == "completed" and check.get("reachedDashboard") is True:
print(f"{country}: ok in {round(run['durationMs'] / 1000)} s, {run['credits']} credits")
continue # success stays quiet
failures += 1
print(f"{country}: {await report(gh, country, run, replay)}")
return 1 if failures else 0
raise SystemExit(asyncio.run(main()))
```
The step names are an `enum` in the schema, so an issue title like `Signup fails from DE at verification-email` is identical from one hour to the next and the dedupe by title holds.
`Only use yourproduct.com.` is an enforced rule: it blocks navigation to any other host, subdomains of yourproduct.com excepted. If your email provider wraps links in a click-tracking domain, name it in the same rule, for example `Only use yourproduct.com and links.mailprovider.example.`, or the run fails at `verify-link`.
### Run it every hour
```yaml .github/workflows/signup-qa.yml theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
name: signup-qa
on:
schedule:
- cron: '0 * * * *'
permissions:
contents: read
issues: write
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm install @outcrawl/sdk tsx && npx tsx signup-qa.mts
env:
OUTCRAWL_API_KEY: ${{ secrets.OUTCRAWL_API_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPO: ${{ github.repository }}
```
Save the TypeScript program as `signup-qa.mts` so it runs as a module with top-level `await`. It exits `1` when any country failed, so the workflow run turns red as well.
## What you get back
A quiet hour prints three lines and files nothing:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
us: ok in 64 s, 16.50 credits
de: ok in 71 s, 18.25 credits
br: ok in 69 s, 17.50 credits
```
An hour where German signups stopped receiving the verification email prints `de: FAILED at verification-email, filed an issue` and opens this issue:
```markdown theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Signup fails from DE at verification-email
**Failed step:** `verification-email`
**Country:** DE
**Page said:** nothing
**Replay:** https://outcrawl.ai/play/ses_7Hq2cW
**Run:** `run_01JAX4T9`, partial, 17.75 credits, 2026-09-25T14:00:41.512Z
| Step | Result | Seconds | Note |
| --- | --- | --- | --- |
| open-signup | pass | 4 | Signup page loaded in German, cookie banner accepted |
| submit-form | pass | 11 | Account created, page asked to confirm the address |
| verification-email | FAIL | 180 | No email in the inbox after 3 minutes |
```
The replay shows the whole run, so whoever picks up the issue watches the failure instead of reproducing it.
## What it costs
An agent run is billed for the model's usage, browser time and proxy usage, per [pricing](/docs/billing/pricing). A typical run costs about 15 credits a minute all in, including the data through the residential exit. That is an estimate: the real figure depends on the task, and a heavy signup page costs more.
* A passing signup takes about 75 s, so **about 19 credits per country**.
* Three countries every hour: about 57 credits an hour, 1,368 a day, **about 41,040 a month**. That fits the Pro plan's 50,000.
* A run that waits the full 3 minutes for an email that never comes adds about 3 credits of browser time, plus the model's usage each time it checks the inbox. Each run's `caps.budget` is 100.
Run it every six hours instead of every hour and the same three countries cost about 6,840 credits a month, inside the Builder plan.
## Make it yours
* Test the paths people pay for: sign up, pick the paid plan, and stop at the card form. `Never make a payment.` is an enforced rule, so the run cannot submit it.
* Add countries by adding codes to `COUNTRIES`. Raise `PARALLEL` only while it stays below your plan's browser count.
* Delete the test accounts: have your backend remove any account created from your inbox provider's domain after an hour.
* Post the failure to Slack as well as GitHub with one more `fetch` to an incoming webhook.
## Related
Tasks, caps, schemas and exits.
Connect the inbox the agent reads.
Which rules are enforced and which are guidance.
The replay link in every issue.
# Research 500 companies into your CRM
Source: https://outcrawl.ai/docs/examples/research-companies-into-your-crm
Search, crawl and agents build a profile of every company on your list, deduplicated and written to HubSpot.
A list of 500 target accounts is worth little until someone knows what each company sells, what it charges, who runs it and whether it is growing. A researcher gets through perhaps twenty a day. This program runs the whole list in lanes: a search for recent news, a crawl of the company's own pages with structured extraction, and an agent only for what the crawl could not find. It dedupes by domain and writes every profile into HubSpot.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* `companies.csv` with a header row and `name,domain` per line: a CRM export, a conference attendee list, a portfolio page.
* A HubSpot private app token in `HUBSPOT_TOKEN` with the `crm.objects.companies.read` and `crm.objects.companies.write` scopes, and five text properties created once on the company object: `pricing_summary`, `decision_makers`, `open_roles`, `team_size_signal`, `recent_news`. The HubSpot calls are your own code, plain HTTPS.
* A plan with enough browsers for your lanes: Free 2, Builder 5, Pro 25, Scale 100. Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot.
## How it works
One [search](/docs/search/overview) per company for funding, hiring and layoff news. Five results, titles and links only, which is 2.5 credits.
A [crawl](/docs/crawl/overview) of up to 8 pages, limited to about, team, pricing and careers paths, with `formats: ['json']` and a schema. Every page comes back as the facts it holds: what they do, plans and prices, named people, open roles, team size.
If the crawl found no prices or no leaders, an [agent](/docs/agents/overview) visits the site to find them: it switches a monthly or yearly toggle, or opens a team page that loads as you scroll. An enforced [rule](/docs/account/rules) keeps it on the company's own domain, and a second tells it not to submit a form, book a demo, start a trial or sign in.
Four lanes, each holding at most one slot. A refused call (`ConcurrencyLimitError`, or `QuotaExceededError` on requests per minute) waits and retries: 20 seconds for a slot, or the `retryAfterSeconds` the error names. Running out of monthly credits stops. Every step reserves its maximum cost against a batch ceiling before it starts.
Domains are normalised before anything runs, so `www.acme.com` and `https://acme.com/` are one company. People are deduplicated by name. Each profile updates the HubSpot company with that domain, or creates it.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { readFile, writeFile } from 'node:fs/promises';
import {
ConcurrencyLimitError,
NoCapacityError,
Outcrawl,
QuotaExceededError,
type CrawlPage,
} from '@outcrawl/sdk';
const oc = new Outcrawl();
// Every key on your account shares one browser slot count (Free 2, Builder 5,
// Pro 25, Scale 100). Keep LANES below yours so other work still gets a slot.
const LANES = 4;
const BATCH_CEILING = 10_000; // credits, for the whole list
const SEARCH_RESULTS = 5; // 5 credits per 10 results
const CRAWL_PAGES = 8; // 1 credit per page, and the crawl's hard limit
const AGENT_BUDGET = '40.00'; // about 2.5 minutes of a typical run
const DECIDES = /founder|chief|\bce?o\b|\bc[tfor]o\b|president|\bvp\b|head of|director/i;
interface Plan { name: string; price: string }
interface Person { name: string; title: string }
interface Facts { whatTheyDo?: string; plans?: Plan[]; people?: Person[]; openRoles?: number; teamSize?: string }
interface Company {
name: string;
domain: string;
whatTheyDo: string;
plans: Plan[];
decisionMakers: Person[];
openRoles: number | null;
teamSize: string | null;
news: { title: string; url: string }[];
}
const PLAN = { type: 'object', properties: { name: { type: 'string' }, price: { type: 'string' } }, required: ['name', 'price'] };
const PERSON = { type: 'object', properties: { name: { type: 'string' }, title: { type: 'string' } }, required: ['name', 'title'] };
const PAGE_SCHEMA = {
type: 'object',
properties: {
whatTheyDo: { type: 'string', description: 'One sentence: what the company sells, and to whom. Empty if this page does not say.' },
plans: { type: 'array', items: PLAN, description: 'Each plan with its price as shown, e.g. "$49 per seat per month" or "Contact sales"' },
people: { type: 'array', items: PERSON, description: 'People this page names, with their job titles' },
openRoles: { type: 'integer', description: 'How many open roles this page lists, if it is a careers page' },
teamSize: { type: 'string', description: 'Any statement of team size, e.g. "a team of 40 across Berlin and Lisbon"' },
},
};
const sleep = (ms: number) => new Promise((wake) => setTimeout(wake, ms));
// A refused call is retried; anything else is the caller's to see.
async function withSlot(call: () => Promise): Promise {
for (let attempt = 1; ; attempt++) {
try {
return await call();
} catch (error) {
if (attempt === 8) throw error;
if (error instanceof ConcurrencyLimitError || error instanceof NoCapacityError) {
// Refused, not queued, and no Retry-After: a slot frees when one of your calls ends.
await sleep(20_000);
} else if (error instanceof QuotaExceededError && error.ceiling === 'requests_per_minute') {
await sleep(error.retryAfterSeconds * 1_000);
} else {
throw error; // out of monthly credits, or a bad request: retrying cannot help
}
}
}
}
// Reserve a step's maximum before it starts, settle to what it really cost.
// A step that throws is counted at its maximum.
let spent = 0;
let held = 0;
async function withinBudget(maximum: number, step: () => Promise, cost: (value: T) => number): Promise {
if (spent + held + maximum > BATCH_CEILING) return undefined;
held += maximum;
let actual = maximum;
try {
const value = await step();
actual = cost(value);
return value;
} finally {
held -= maximum;
spent += actual;
}
}
// At most LANES companies in flight: each lane takes the next one when it is free.
async function inLanes(items: readonly T[], work: (item: T) => Promise): Promise {
const results: R[] = [];
let next = 0;
const lane = async () => {
while (next < items.length) {
const index = next++;
results[index] = await work(items[index] as T);
}
};
await Promise.all(Array.from({ length: Math.min(LANES, items.length) }, lane));
return results;
}
// 0. Read the list, deduplicated by normalised domain.
const domain = (raw: string) => raw.trim().toLowerCase().replace(/^https?:\/\//, '').replace(/^www\./, '').split('/')[0] ?? '';
const targets = new Map();
for (const line of (await readFile('companies.csv', 'utf8')).split('\n').slice(1)) {
const cut = line.lastIndexOf(',');
const key = domain(line.slice(cut + 1));
if (cut > 0 && key !== '' && !targets.has(key)) targets.set(key, line.slice(0, cut).replace(/^"|"$/g, '').trim());
}
async function research([site, name]: [string, string]): Promise {
const company: Company = { name, domain: site, whatTheyDo: '', plans: [], decisionMakers: [], openRoles: null, teamSize: null, news: [] };
const people = new Map();
const addPeople = (found: Person[] = []) => {
for (const person of found) {
if (DECIDES.test(person.title) && !people.has(person.name.toLowerCase())) people.set(person.name.toLowerCase(), person);
}
};
try {
// 1. Search: news that says growing or shrinking.
const hits = await withinBudget(
SEARCH_RESULTS * 0.5,
() => withSlot(() => oc.search(`"${name}" funding OR hiring OR layoffs`, { limit: SEARCH_RESULTS })),
(found) => found.length * 0.5,
);
company.news = (hits ?? []).filter((hit) => !hit.url.includes(site)).slice(0, 3).map((hit) => ({ title: hit.title, url: hit.url }));
// 2. Crawl: the company's own pages, each read into the schema.
const pages = await withinBudget(
CRAWL_PAGES,
() =>
withSlot(async () => {
const crawl = await oc.crawl(`https://${site}`, {
limit: CRAWL_PAGES,
maxDepth: 2,
includePaths: ['/about*', '/about/**', '/company*', '/team*', '/leadership*', '/pricing*', '/plans*', '/careers*', '/jobs*'],
formats: ['json'],
schema: PAGE_SCHEMA,
});
const found: CrawlPage[] = [];
for await (const page of crawl) found.push(page);
return found;
}),
(found) => found.length,
);
for (const page of [...(pages ?? [])].sort((a, b) => a.depth - b.depth)) {
const facts = (page.json ?? {}) as Facts;
if (company.whatTheyDo === '' && facts.whatTheyDo) company.whatTheyDo = facts.whatTheyDo;
if ((facts.plans?.length ?? 0) > company.plans.length) company.plans = facts.plans ?? [];
if (facts.openRoles !== undefined) company.openRoles = Math.max(company.openRoles ?? 0, facts.openRoles);
company.teamSize ??= facts.teamSize ?? null;
addPeople(facts.people);
}
// 3. Agent: only for what the crawl could not find.
const wanted = [
company.plans.length === 0 && 'every plan and its price as shown (switch to monthly billing if there is a toggle; if prices are only on request, return one plan named "All plans" priced "Contact sales")',
people.size === 0 && 'the founders and leaders the site names, with their job titles',
].filter((want): want is string => typeof want === 'string');
if (wanted.length > 0) {
const run = await withinBudget(
Number(AGENT_BUDGET),
async () => {
// Only the submit is retried: a refusal arrives before any run exists.
const { job } = await withSlot(async () => {
const job = oc.agent({
task: `On ${site}, the website of ${name}, find: ${wanted.join('; and ')}. Use only pages on ${site}.`,
caps: { budget: AGENT_BUDGET, duration: '4m' },
schema: { type: 'object', properties: { plans: { type: 'array', items: PLAN }, people: { type: 'array', items: PERSON } } },
rules: [`Only visit ${site} and its subdomains.`, 'Never submit a form, book a demo, start a trial or sign in.'],
});
await job.submitted; // a refused submit throws here
return { job }; // wrapped, because a job is awaitable and returning it bare waits for the run
});
return await job;
},
(settled) => Number(settled.credits),
);
const found = (run?.data ?? {}) as Facts;
if (company.plans.length === 0) company.plans = found.plans ?? [];
addPeople(found.people);
}
} catch (error) {
console.error(`${site}: ${String(error)}`); // keep what was found before the error
}
company.decisionMakers = [...people.values()];
return company;
}
const companies = await inLanes([...targets.entries()], research);
await writeFile('companies.json', JSON.stringify(companies, null, 2));
// 4. Write to HubSpot: update the company with this domain, or create it.
async function hubspot(path: string, method: string, body: unknown): Promise<{ results?: { id: string }[] }> {
const response = await fetch(`https://api.hubapi.com/crm/v3/objects/companies${path}`, {
method,
headers: { authorization: `Bearer ${process.env.HUBSPOT_TOKEN}`, 'content-type': 'application/json' },
body: JSON.stringify(body),
});
if (!response.ok) throw new Error(`HubSpot ${response.status}: ${await response.text()}`);
return (await response.json()) as { results?: { id: string }[] };
}
let written = 0;
for (const company of companies) {
if (company.whatTheyDo === '' && company.plans.length === 0) continue; // nothing worth writing
const properties = {
name: company.name,
domain: company.domain,
description: company.whatTheyDo,
pricing_summary: company.plans.map((plan) => `${plan.name}: ${plan.price}`).join('\n'),
decision_makers: company.decisionMakers.map((person) => `${person.name}, ${person.title}`).join('\n'),
open_roles: company.openRoles === null ? '' : String(company.openRoles),
team_size_signal: company.teamSize ?? '',
recent_news: company.news.map((item) => `${item.title} ${item.url}`).join('\n'),
};
const found = await hubspot('/search', 'POST', {
filterGroups: [{ filters: [{ propertyName: 'domain', operator: 'EQ', value: company.domain }] }],
limit: 1,
});
const id = found.results?.[0]?.id;
await (id === undefined ? hubspot('', 'POST', { properties }) : hubspot(`/${id}`, 'PATCH', { properties }));
written += 1;
}
console.log(`${companies.length} companies researched, ${written} written to HubSpot, ${spent.toFixed(2)} credits`);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import json
import os
import re
import httpx
from outcrawl import ConcurrencyLimitError, NoCapacityError, Outcrawl, QuotaExceededError
# Every key on your account shares one browser slot count (Free 2, Builder 5,
# Pro 25, Scale 100). Keep LANES below yours so other work still gets a slot.
LANES = 4
BATCH_CEILING = 10_000 # credits, for the whole list
SEARCH_RESULTS = 5 # 5 credits per 10 results
CRAWL_PAGES = 8 # 1 credit per page, and the crawl's hard limit
AGENT_BUDGET = "40.00" # about 2.5 minutes of a typical run
DECIDES = re.compile(r"founder|chief|\bce?o\b|\bc[tfor]o\b|president|\bvp\b|head of|director", re.I)
PLAN = {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "string"}}, "required": ["name", "price"]}
PERSON = {"type": "object", "properties": {"name": {"type": "string"}, "title": {"type": "string"}}, "required": ["name", "title"]}
PAGE_SCHEMA = {
"type": "object",
"properties": {
"whatTheyDo": {"type": "string", "description": "One sentence: what the company sells, and to whom. Empty if this page does not say."},
"plans": {"type": "array", "items": PLAN, "description": 'Each plan with its price as shown, e.g. "$49 per seat per month" or "Contact sales"'},
"people": {"type": "array", "items": PERSON, "description": "People this page names, with their job titles"},
"openRoles": {"type": "integer", "description": "How many open roles this page lists, if it is a careers page"},
"teamSize": {"type": "string", "description": 'Any statement of team size, e.g. "a team of 40 across Berlin and Lisbon"'},
},
}
tally = {"spent": 0.0, "held": 0.0}
async def with_slot(call):
"""A refused call is retried; anything else is the caller's to see."""
for attempt in range(1, 9):
try:
return await call()
except (ConcurrencyLimitError, NoCapacityError):
# Refused, not queued, and no Retry-After: a slot frees when one of your calls ends.
if attempt == 8:
raise
await asyncio.sleep(20)
except QuotaExceededError as error:
# Out of monthly credits: retrying cannot help. Per-minute: wait as told.
if error.ceiling != "requests_per_minute" or attempt == 8:
raise
await asyncio.sleep(error.retry_after_seconds)
async def within_budget(maximum, step, cost):
"""Reserve a step's maximum before it starts, settle to what it really cost.
A step that raises is counted at its maximum."""
if tally["spent"] + tally["held"] + maximum > BATCH_CEILING:
return None
tally["held"] += maximum
actual = maximum
try:
value = await step()
actual = cost(value)
return value
finally:
tally["held"] -= maximum
tally["spent"] += actual
def normalise(raw: str) -> str:
raw = re.sub(r"^https?://", "", raw.strip().lower())
return re.sub(r"^www\.", "", raw).split("/")[0]
async def main() -> None:
# 0. Read the list, deduplicated by normalised domain.
targets: dict[str, str] = {}
with open("companies.csv") as source:
for line in source.read().splitlines()[1:]:
name, _, raw = line.rpartition(",")
site = normalise(raw)
if name and site and site not in targets:
targets[site] = name.strip().strip('"')
async with Outcrawl() as oc:
lanes = asyncio.Semaphore(LANES)
async def research(site: str, name: str) -> dict:
company = {"name": name, "domain": site, "whatTheyDo": "", "plans": [], "decisionMakers": [],
"openRoles": None, "teamSize": None, "news": []}
people: dict[str, dict] = {}
def add_people(found) -> None:
for person in found or []:
key = person["name"].lower()
if DECIDES.search(person["title"]) and key not in people:
people[key] = person
async with lanes:
try:
# 1. Search: news that says growing or shrinking.
hits = await within_budget(
SEARCH_RESULTS * 0.5,
lambda: with_slot(lambda: oc.search(f'"{name}" funding OR hiring OR layoffs', limit=SEARCH_RESULTS)),
lambda found: len(found) * 0.5,
)
company["news"] = [{"title": hit["title"], "url": hit["url"]}
for hit in (hits or []) if site not in hit["url"]][:3]
# 2. Crawl: the company's own pages, each read into the schema.
async def crawl_site():
crawl = await oc.crawl(
f"https://{site}",
limit=CRAWL_PAGES,
maxDepth=2,
includePaths=["/about*", "/about/**", "/company*", "/team*", "/leadership*",
"/pricing*", "/plans*", "/careers*", "/jobs*"],
formats=["json"],
schema=PAGE_SCHEMA,
)
return [page async for page in crawl] # a refusal raises on the first read
pages = await within_budget(CRAWL_PAGES, lambda: with_slot(crawl_site), len)
for page in sorted(pages or [], key=lambda p: p.get("depth", 0)):
facts = page.get("json") or {}
if not company["whatTheyDo"] and facts.get("whatTheyDo"):
company["whatTheyDo"] = facts["whatTheyDo"]
if len(facts.get("plans") or []) > len(company["plans"]):
company["plans"] = facts["plans"]
if facts.get("openRoles") is not None:
company["openRoles"] = max(company["openRoles"] or 0, facts["openRoles"])
company["teamSize"] = company["teamSize"] or facts.get("teamSize")
add_people(facts.get("people"))
# 3. Agent: only for what the crawl could not find.
wanted = []
if not company["plans"]:
wanted.append('every plan and its price as shown (switch to monthly billing if there is a '
'toggle; if prices are only on request, return one plan named "All plans" '
'priced "Contact sales")')
if not people:
wanted.append("the founders and leaders the site names, with their job titles")
if wanted:
async def submit():
job = oc.agent(
task=f"On {site}, the website of {name}, find: {'; and '.join(wanted)}. "
f"Use only pages on {site}.",
caps={"budget": AGENT_BUDGET, "duration": "4m"},
schema={"type": "object", "properties": {
"plans": {"type": "array", "items": PLAN},
"people": {"type": "array", "items": PERSON}}},
rules=[f"Only visit {site} and its subdomains.",
"Never submit a form, book a demo, start a trial or sign in."],
)
await job.submitted # a refused submit raises here
return job
async def run_agent():
# Only the submit is retried: a refusal arrives before any run exists.
job = await with_slot(submit)
return await job
run = await within_budget(float(AGENT_BUDGET), run_agent, lambda r: float(r["credits"]))
found = (run or {}).get("data") or {}
if not company["plans"]:
company["plans"] = found.get("plans") or []
add_people(found.get("people"))
except Exception as error:
print(f"{site}: {error}") # keep what was found before the error
company["decisionMakers"] = list(people.values())
return company
companies = await asyncio.gather(*(research(site, name) for site, name in targets.items()))
with open("companies.json", "w") as out:
json.dump(companies, out, indent=2, ensure_ascii=False)
# 4. Write to HubSpot: update the company with this domain, or create it.
written = 0
async with httpx.AsyncClient(
base_url="https://api.hubapi.com/crm/v3/objects",
headers={"authorization": f"Bearer {os.environ['HUBSPOT_TOKEN']}"},
) as hubspot:
for company in companies:
if not company["whatTheyDo"] and not company["plans"]:
continue # nothing worth writing
properties = {
"name": company["name"],
"domain": company["domain"],
"description": company["whatTheyDo"],
"pricing_summary": "\n".join(f"{p['name']}: {p['price']}" for p in company["plans"]),
"decision_makers": "\n".join(f"{p['name']}, {p['title']}" for p in company["decisionMakers"]),
"open_roles": "" if company["openRoles"] is None else str(company["openRoles"]),
"team_size_signal": company["teamSize"] or "",
"recent_news": "\n".join(f"{n['title']} {n['url']}" for n in company["news"]),
}
found = await hubspot.post("/companies/search", json={
"filterGroups": [{"filters": [{"propertyName": "domain", "operator": "EQ", "value": company["domain"]}]}],
"limit": 1,
})
found.raise_for_status()
existing = found.json().get("results") or []
if existing:
response = await hubspot.patch(f"/companies/{existing[0]['id']}", json={"properties": properties})
else:
response = await hubspot.post("/companies", json={"properties": properties})
response.raise_for_status()
written += 1
print(f"{len(companies)} companies researched, {written} written to HubSpot, {tally['spent']:.2f} credits")
asyncio.run(main())
```
The retry wraps only the call that can be refused. A crawl or search that is refused never started, so retrying it cannot bill twice, and an agent run is retried at its submit, never after it exists. A company whose steps fail keeps whatever the earlier steps found.
## What you get back
One entry per unique domain in `companies.json`, and the same fields on the HubSpot company record:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"name": "Brightloom Data",
"domain": "brightloom.example",
"whatTheyDo": "Brightloom sells a warehouse-native product analytics tool to B2B SaaS teams.",
"plans": [
{ "name": "Starter", "price": "$0 up to 10k events" },
{ "name": "Growth", "price": "$450 per month, billed monthly" },
{ "name": "Enterprise", "price": "Contact sales" }
],
"decisionMakers": [
{ "name": "Maya Okafor", "title": "Co-founder and CEO" },
{ "name": "Jonas Lindqvist", "title": "Co-founder and CTO" },
{ "name": "Priya Raman", "title": "VP of Sales" }
],
"openRoles": 7,
"teamSize": "a team of 38 across Berlin and Lisbon",
"news": [
{ "title": "Brightloom Data raises $14M Series A to take analytics into the warehouse", "url": "https://news.example/brightloom-series-a" }
]
}
```
And the last line printed:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
500 companies researched, 473 written to HubSpot, 7424.50 credits
```
## What it costs
Rates from [pricing](/docs/billing/pricing): search 5 credits per 10 results, crawl 1 credit a page. An agent run is billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in, proxy usage included. That is an estimate: the real figure depends on the task, and a page-heavy site costs more.
* Search: 5 results, **2.5 credits** a company.
* Crawl: at most 8 pages, **up to 8 credits** a company.
* Agent: needed for about a third of companies, at about 60 s, so about 15 credits a run, or **about 5 credits** averaged over the list.
* Per company, **about 15.5 credits**. For 500 companies, **about 7,750 credits**, under the 10,000 ceiling. Pro's 50,000 monthly credits covers it about six times.
* Wall clock: about a minute a company, so 500 over 4 lanes is about two hours. On Pro with 20 lanes it is about 25 minutes.
## Make it yours
* Build the list from search instead of a CSV: take the domain of every hit for `"Series A" developer tools Berlin`.
* Write to Salesforce, Attio or a Postgres table instead: only step 4 changes.
* Add fields to `PAGE_SCHEMA`, such as the tech stack named on the careers pages or the customer logos on the home page.
* Re-run monthly and write only the fields that changed, so your team sees a pricing change as a HubSpot property history entry.
## Related
Results for a query, optionally rendered.
Walk a site, stream every page with structured output.
Tasks, caps, schemas and what a run returns.
What `concurrency_limit` and `quota_exceeded` mean, and which are retryable.
# Script the sign-in, hand the rest to an agent
Source: https://outcrawl.ai/docs/examples/script-then-hand-to-the-agent
Playwright for the steps that never change, an agent for the pages that always do.
Every morning someone on an operations team signs in to the warehouse portal, opens the list of shipments with exceptions, and chases each one through a message thread and a carrier's tracking page. The first half is the same clicks every day. The second half is different every time: UPS, FedEx and a regional freight carrier each lay out tracking their own way. This program scripts the first half with Playwright and gives the second half to an agent, one shipment at a time.
This example is TypeScript only: `oc.browser()`, the browser you drive with Playwright, exists only in the TypeScript SDK.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* The portal's sign-in in `NORTHGATE_EMAIL` and `NORTHGATE_PASSWORD`. The script types them itself; the agent never sees them.
* `npm install @outcrawl/sdk playwright`. Playwright is only the driver; the browser runs on Outcrawl.
## How it works
`oc.browser({ profile })` opens a [browser](/docs/browsers/overview) in a [profile](/docs/account/profiles), so the portal's login is kept after the first run and the script skips the sign-in from then on.
Sign in with the SDK's human-input methods, then read the exceptions table row by row with Playwright through `page.raw`. The list is read exactly, not interpreted by a model.
An agent run cannot attach to a browser you opened. What it can share is the profile: closing the browser frees your browser slot and the profile, and the agent run starts signed in to the same account.
Each [agent run](/docs/agents/overview) reads one shipment's thread, follows its carrier's tracking page wherever it leads, and answers in a schema. A per-run [rule](/docs/account/rules) forbids messages, claims and changes to a shipment.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import type { Page } from 'playwright';
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const PORTAL = 'https://portal.northgate-3pl.example';
interface Exception {
shipment: string;
carrier: string;
flaggedOn: string;
trackingUrl: string;
}
interface Finding {
whatHappened: string;
lastScan: string;
newEstimate?: string;
askedOfUs?: string;
}
const findingSchema = {
type: 'object',
properties: {
whatHappened: { type: 'string', description: 'the exception in one sentence' },
lastScan: { type: 'string', description: 'the most recent carrier scan: place, time and event' },
newEstimate: { type: 'string', description: "the carrier's current delivery estimate, YYYY-MM-DD" },
askedOfUs: { type: 'string', description: 'what the carrier or the warehouse asks us to do, if anything' },
},
required: ['whatHappened', 'lastScan'],
};
// The profile keeps the portal's login between runs, and is what the agent inherits.
const [existing] = await oc.profiles.list({ label: 'northgate-ops' });
const profile = existing ?? (await oc.profiles.create({ label: 'northgate-ops' }));
// 1. The known part, scripted: sign in if needed and read the exceptions table.
const exceptions: Exception[] = [];
const browser = await oc.browser({ profile: profile.id });
try {
const page = await browser.newPage();
const pw = page.raw as unknown as Page; // page.raw is the real Playwright page
await page.goto(`${PORTAL}/shipments?status=exception`);
if (pw.url().includes('/login')) {
await page.fill('#email', process.env.NORTHGATE_EMAIL!);
await page.fill('#password', process.env.NORTHGATE_PASSWORD!);
await page.click('button[type=submit]');
await pw.waitForURL('**/shipments**');
}
await pw.waitForSelector('table.shipments');
for (const row of await pw.locator('table.shipments tbody tr').all()) {
const [shipment = '', carrier = '', flaggedOn = ''] = await row.locator('td').allInnerTexts();
const trackingUrl = (await row.locator('a.tracking').getAttribute('href')) ?? '';
exceptions.push({ shipment: shipment.trim(), carrier: carrier.trim(), flaggedOn: flaggedOn.trim(), trackingUrl });
}
} finally {
// Closing frees the browser slot and the profile. The login stays in the profile.
await browser.close();
}
console.log(`${exceptions.length} shipments flagged`);
// 2. The messy part, one agent run per shipment, in the same signed-in profile.
const report = [];
for (const exception of exceptions) {
const run = await oc.agent({
task:
`Open ${PORTAL}/shipments/${exception.shipment} and read the exception note and the message thread. ` +
`Then open the ${exception.carrier} tracking page at ${exception.trackingUrl} and read the most recent scan. ` +
"Report what went wrong, the last scan, the carrier's new delivery estimate, and anything we are asked to do.",
profile: profile.id,
schema: findingSchema,
caps: { budget: '25.00' },
rules: ['Never send a message, file a claim, or change, cancel or reship a shipment.'],
});
const finding = run.data as Finding | undefined;
report.push({
shipment: exception.shipment,
carrier: exception.carrier,
lastScan: finding?.lastScan ?? `(run ${run.status})`,
newEstimate: finding?.newEstimate ?? '',
askedOfUs: finding?.askedOfUs ?? '',
credits: run.credits,
});
}
console.table(report);
```
The browser closes in `finally`, whatever happens in between. A browser that is never closed keeps holding one of your account's browser slots until its lease runs out, six hours later. The agent runs go one after another because a profile holds one session at a time.
## What you get back
`console.table(report)` prints one row per exception (a sample):
| shipment | carrier | lastScan | newEstimate | askedOfUs | credits |
| - | - | - | - | - | - |
| NG-48213 | UPS | Memphis, TN, 25 Sep 06:12: held, address needs correcting | 2026-09-29 | Confirm the suite number for 1200 Harbor Blvd | 13.00 |
| NG-48190 | FedEx | Oakland, CA, 24 Sep 22:40: delivery exception, weather | 2026-09-26 | | 10.25 |
| NG-48177 | Estes | Reno, NV, 24 Sep 15:05: pallet short one carton | 2026-10-01 | The warehouse asks whether to ship the short pallet as is | 16.00 |
## What it costs
A browser you drive yourself is 1 credit a minute, plus proxy usage. The scripted part takes about 45 seconds: **under 1 credit**. An agent run is billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in, proxy usage included. A shipment takes about 50 seconds (the shipment, the carrier's tracking page, one detail view): **about 13 credits per exception**, with a `caps.budget` of 25 for each run. That is an estimate: the real figure depends on the task, and a heavy tracking page costs more.
Twelve exceptions a morning is about 157 credits, or about 3,450 over 22 working days. Reading the table with an agent as well would add about 30 seconds, 8 credits a day, and the list would come from a model instead of from the page.
## Make it yours
* Run the script from a scheduler at 07:00 and post the table to your team's channel with a Slack incoming webhook.
* Filter in the script: send only exceptions older than a day to an agent.
* Use `page.extract(instruction, schema)` inside the scripted session for a one-off read that is not worth a whole agent run.
* When the portal changes its sign-in form, only the four sign-in lines change; the agents do not care.
## Related
`oc.browser()`, `page.raw` and human input.
The login the script and the agents share.
Schemas, caps and statuses.
Guardrails for each run.
# Apply to the roles you picked, tailored and approved by you
Source: https://outcrawl.ai/docs/examples/tailored-job-applications
An agent reads each posting and drafts every answer from your resume; you edit and approve each draft; a second run submits exactly what you approved.
A good application answers the questions this company asked, in words that fit this role, and that is why each one takes an hour. This program sends an agent to read each posting you picked and its application form, and to draft every answer from your resume. Each draft lands in a file. You read it, change anything you like, and approve it; only then does a second run complete the form with your approved answers, attach your resume and submit.
It is for a handful of roles you chose and read. It does not look for jobs, and it is not for sending one application to hundreds of postings.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* Your resume twice: `resume.md`, the text the agent tailors from, and `resume.pdf`, the file it attaches. A run can attach a file to a page but cannot read it, so the text copy is what it works from.
* The postings. Greenhouse, Lever and Ashby forms need no account, which is what this program expects.
* A terminal. The program asks you about each draft.
## How it works
An [agent run](/docs/agents/overview) reads the posting and its form without typing into it, and answers a [schema](/docs/agents/overview#typed-results): every question as the form asks it, whether it is required, a drafted answer, a cover letter if the form takes one, and the requirements your resume does not show. Two [rules](/docs/account/rules) are enforced: `Only use job-boards.greenhouse.io.` (each posting's own site) and `Never send, post or upload anything.`
Every draft is saved as `applications/.json`. The program shows the role and the gaps, and waits while you edit the file. Type `submit` to approve it or press Enter to skip it. No run is waiting on you: the approval sits between two runs.
A second run reads the file as you left it and completes the form with those answers, word for word. `resume.pdf` goes to the run as a [file](/docs/agents/files) straight after submit; it is destroyed when the run ends. The rule `Never create an account or sign up.` is enforced, so a form that wants an account comes back to you unsubmitted.
Each run answers whether it submitted, the confirmation the site showed, or what is missing, with a replay link to watch exactly what it did.
The approval is the split between the two runs, not a rule. A button labelled "Submit application" is not one Outcrawl's irreversible check recognises, so no rule could block it. The only run told to submit is the one that starts after you type `submit`.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import { stdin, stdout } from 'node:process';
import { createInterface } from 'node:readline/promises';
import { setTimeout as sleep } from 'node:timers/promises';
import { ConcurrencyLimitError, Outcrawl, QuotaExceededError } from '@outcrawl/sdk';
import type { AgentRequest } from '@outcrawl/sdk';
const oc = new Outcrawl();
// The roles you picked. A handful you would apply to by hand, not a blast.
const postings = [
'https://job-boards.greenhouse.io/figma/jobs/5498127004',
'https://jobs.lever.co/plaid/6f2c1a9e-3b7d-4e0a-9c55-1d8e2b7f4a10',
'https://jobs.ashbyhq.com/linear/2b9d4c1e-7a3f-4f6b-8e21-5c0d9a7b3e64',
];
const me = {
name: 'Ada Park',
email: 'ada.park@example.com',
phone: '+1 415 555 0142',
location: 'San Francisco, CA',
linkedin: 'https://www.linkedin.com/in/adapark',
};
const resumeText = await readFile('resume.md', 'utf8'); // what the agent reads to tailor
const resumePdf = new Uint8Array(await readFile('resume.pdf')); // what it attaches to the form
// Runs at once. Keep it below your plan's browser count (Free 2, Builder 5, Pro 25, Scale 100).
const MAX_AT_ONCE = 2;
interface Draft {
company: string;
role: string;
fields: { label: string; required: boolean; answer: string }[];
coverLetter?: string;
gaps?: string[];
}
const draftSchema = {
type: 'object',
properties: {
company: { type: 'string' },
role: { type: 'string' },
fields: {
type: 'array',
description: 'Every question on the application form, in order, except name, contact details and resume upload',
items: {
type: 'object',
properties: {
label: { type: 'string', description: 'The question exactly as the form asks it' },
required: { type: 'boolean' },
answer: { type: 'string', description: 'Your drafted answer' },
},
required: ['label', 'required', 'answer'],
},
},
coverLetter: { type: 'string', description: 'Under 200 words, only if the form accepts one' },
gaps: { type: 'array', items: { type: 'string' }, description: 'Requirements the resume does not show' },
},
required: ['company', 'role', 'fields'],
};
const submittedSchema = {
type: 'object',
properties: {
submitted: { type: 'boolean' },
confirmation: { type: 'string', description: 'The confirmation message the site showed' },
leftForYou: { type: 'string', description: 'Why it was not submitted, and what is missing' },
},
required: ['submitted'],
};
// Submit a run and answer its id, waiting for a free browser slot instead of failing.
async function start(request: AgentRequest): Promise {
for (;;) {
try {
return (await oc.agent(request).submitted).id;
} catch (error) {
if (error instanceof ConcurrencyLimitError) {
await sleep(20_000); // no Retry-After: a slot frees when one of your runs ends
} else if (error instanceof QuotaExceededError && error.ceiling === 'requests_per_minute') {
await sleep(error.retryAfterSeconds * 1000); // the error says how long to wait
} else {
throw error; // out of monthly credits, or a bad request: retrying cannot help
}
}
}
}
async function inPool(items: readonly T[], size: number, work: (item: T) => Promise): Promise {
const results: R[] = new Array(items.length);
let next = 0;
const worker = async () => {
while (next < items.length) {
const index = next++;
results[index] = await work(items[index] as T);
}
};
await Promise.all(Array.from({ length: Math.min(size, items.length) }, worker));
return results;
}
// Pass 1: read each posting and its form, and draft. Nothing is typed into the form.
await mkdir('applications', { recursive: true });
const drafts = await inPool(postings, MAX_AT_ONCE, async (url) => {
const run = await oc.agent.job(await start({
task:
`Open ${url}. Read the job posting and its application form. Do not type into the form or submit anything. ` +
'Using my resume below, draft an answer to every question on the form, tailored to this role: plain, ' +
'specific, and true to the resume, never claiming anything it does not support. List any requirement the ' +
`resume does not show.\n\nMy resume:\n${resumeText}`,
caps: { budget: '75.00', duration: '5m' },
rules: [`Only use ${new URL(url).hostname}.`, 'Never send, post or upload anything.'],
schema: draftSchema,
})).settled();
if (run.data === undefined) {
console.log(`No draft for ${url}: run ended ${run.stopReason ?? run.status}, replay ${run.replayUrl ?? 'none'}`);
return undefined;
}
const file = `applications/${new URL(url).pathname.split('/').filter(Boolean).join('-')}.json`;
await writeFile(file, JSON.stringify({ url, ...(run.data as Draft) }, null, 2));
return file;
});
// You read every draft, edit the file if you want, and approve it or skip it.
const prompt = createInterface({ input: stdin, output: stdout });
const approved: string[] = [];
for (const file of drafts) {
if (file === undefined) continue;
const draft = JSON.parse(await readFile(file, 'utf8')) as Draft & { url: string };
console.log(`\n${draft.role} at ${draft.company}: ${draft.fields.length} answers in ${file}`);
if (draft.gaps?.length) console.log(` The resume does not show: ${draft.gaps.join('; ')}`);
const answer = await prompt.question('Edit the file if you like, then type "submit" to send it, or Enter to skip: ');
if (answer.trim() === 'submit') approved.push(file);
}
prompt.close();
// Pass 2: only what you approved, with the answers as they are in the file now.
const outcomes = await inPool(approved, MAX_AT_ONCE, async (file) => {
const draft = JSON.parse(await readFile(file, 'utf8')) as Draft & { url: string };
const job = oc.agent.job(await start({
task:
`Open ${draft.url} and apply. Personal details: ${JSON.stringify(me)}. Attach the file resume.pdf where the ` +
`form asks for a resume or CV. Answer the questions with exactly these answers, word for word: ` +
`${JSON.stringify(draft.fields)}.${draft.coverLetter ? ` Cover letter, where the form accepts one: ${draft.coverLetter}` : ''} ` +
'If the form asks a required question that is not answered here, or wants me to create an account, stop ' +
'without submitting and say what is missing. Otherwise submit the application.',
caps: { budget: '115.00', duration: '8m' },
rules: [`Only use ${new URL(draft.url).hostname}.`, 'Never create an account or sign up.'],
schema: submittedSchema,
}));
await job.addFile({ name: 'resume.pdf', bytes: resumePdf, contentType: 'application/pdf' });
const run = await job.settled();
return { role: `${draft.role} at ${draft.company}`, ...(run.data as object | undefined), replay: run.replayUrl };
});
console.table(outcomes);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import json
from pathlib import Path
from urllib.parse import urlparse
from outcrawl import ConcurrencyLimitError, Outcrawl, QuotaExceededError
# The roles you picked. A handful you would apply to by hand, not a blast.
POSTINGS = [
"https://job-boards.greenhouse.io/figma/jobs/5498127004",
"https://jobs.lever.co/plaid/6f2c1a9e-3b7d-4e0a-9c55-1d8e2b7f4a10",
"https://jobs.ashbyhq.com/linear/2b9d4c1e-7a3f-4f6b-8e21-5c0d9a7b3e64",
]
ME = {
"name": "Ada Park",
"email": "ada.park@example.com",
"phone": "+1 415 555 0142",
"location": "San Francisco, CA",
"linkedin": "https://www.linkedin.com/in/adapark",
}
RESUME_TEXT = Path("resume.md").read_text() # what the agent reads to tailor
RESUME_PDF = Path("resume.pdf").read_bytes() # what it attaches to the form
# Runs at once. Keep it below your plan's browser count (Free 2, Builder 5, Pro 25, Scale 100).
MAX_AT_ONCE = 2
DRAFT_SCHEMA = {
"type": "object",
"properties": {
"company": {"type": "string"},
"role": {"type": "string"},
"fields": {
"type": "array",
"description": "Every question on the application form, in order, except name, contact details and resume upload",
"items": {
"type": "object",
"properties": {
"label": {"type": "string", "description": "The question exactly as the form asks it"},
"required": {"type": "boolean"},
"answer": {"type": "string", "description": "Your drafted answer"},
},
"required": ["label", "required", "answer"],
},
},
"coverLetter": {"type": "string", "description": "Under 200 words, only if the form accepts one"},
"gaps": {"type": "array", "items": {"type": "string"}, "description": "Requirements the resume does not show"},
},
"required": ["company", "role", "fields"],
}
SUBMITTED_SCHEMA = {
"type": "object",
"properties": {
"submitted": {"type": "boolean"},
"confirmation": {"type": "string", "description": "The confirmation message the site showed"},
"leftForYou": {"type": "string", "description": "Why it was not submitted, and what is missing"},
},
"required": ["submitted"],
}
async def start(oc: Outcrawl, **request) -> str:
"""Submit a run and answer its id, waiting for a free browser slot instead of failing."""
while True:
try:
return (await oc.agent(**request).submitted)["id"]
except ConcurrencyLimitError:
await asyncio.sleep(20) # no Retry-After: a slot frees when one of your runs ends
except QuotaExceededError as error:
if error.ceiling != "requests_per_minute":
raise # out of monthly credits: retrying cannot help
await asyncio.sleep(error.retry_after_seconds) # the error says how long to wait
async def main() -> None:
Path("applications").mkdir(exist_ok=True)
async with Outcrawl() as oc:
slots = asyncio.Semaphore(MAX_AT_ONCE)
# Pass 1: read each posting and its form, and draft. Nothing is typed into the form.
async def draft(url: str) -> Path | None:
async with slots:
run_id = await start(
oc,
task=(
f"Open {url}. Read the job posting and its application form. Do not type into the form or "
"submit anything. Using my resume below, draft an answer to every question on the form, "
"tailored to this role: plain, specific, and true to the resume, never claiming anything it "
f"does not support. List any requirement the resume does not show.\n\nMy resume:\n{RESUME_TEXT}"
),
caps={"budget": "75.00", "duration": "5m"},
rules=[f"Only use {urlparse(url).hostname}.", "Never send, post or upload anything."],
schema=DRAFT_SCHEMA,
)
run = await oc.agent.job(run_id).settled()
if not run.get("data"):
print(f"No draft for {url}: run ended {run.get('stopReason') or run['status']}, replay {run.get('replayUrl')}")
return None
file = Path("applications") / ("-".join(part for part in urlparse(url).path.split("/") if part) + ".json")
file.write_text(json.dumps({"url": url, **run["data"]}, indent=2))
return file
drafts = await asyncio.gather(*(draft(url) for url in POSTINGS))
# You read every draft, edit the file if you want, and approve it or skip it.
approved = []
for file in drafts:
if file is None:
continue
content = json.loads(file.read_text())
print(f"\n{content['role']} at {content['company']}: {len(content['fields'])} answers in {file}")
if content.get("gaps"):
print(f" The resume does not show: {'; '.join(content['gaps'])}")
answer = await asyncio.to_thread(input, 'Edit the file if you like, then type "submit" to send it, or Enter to skip: ')
if answer.strip() == "submit":
approved.append(file)
# Pass 2: only what you approved, with the answers as they are in the file now.
async def apply(file: Path) -> dict:
content = json.loads(file.read_text())
cover = f" Cover letter, where the form accepts one: {content['coverLetter']}" if content.get("coverLetter") else ""
async with slots:
run_id = await start(
oc,
task=(
f"Open {content['url']} and apply. Personal details: {json.dumps(ME)}. Attach the file "
"resume.pdf where the form asks for a resume or CV. Answer the questions with exactly these "
f"answers, word for word: {json.dumps(content['fields'])}.{cover} If the form asks a required "
"question that is not answered here, or wants me to create an account, stop without "
"submitting and say what is missing. Otherwise submit the application."
),
caps={"budget": "115.00", "duration": "8m"},
rules=[f"Only use {urlparse(content['url']).hostname}.", "Never create an account or sign up."],
schema=SUBMITTED_SCHEMA,
)
job = oc.agent.job(run_id)
await job.add_file(name="resume.pdf", content=RESUME_PDF, content_type="application/pdf")
run = await job.settled()
return {"role": f"{content['role']} at {content['company']}", **(run.get("data") or {}), "replay": run.get("replayUrl")}
for outcome in await asyncio.gather(*(apply(file) for file in approved)):
print(outcome)
asyncio.run(main())
```
## What you get back
A draft, `applications/figma-jobs-5498127004.json`, before you edit it:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"url": "https://job-boards.greenhouse.io/figma/jobs/5498127004",
"company": "Figma",
"role": "Product Designer, Editor",
"fields": [
{
"label": "Why are you interested in working at Figma?",
"required": true,
"answer": "I have spent four years designing editing tools for a collaborative whiteboard, most recently leading the redesign of its selection and alignment model, which cut layout-related support tickets by a third. Figma's editor is where that work matters most, and I want to work on it with the people who set the standard."
},
{
"label": "Link to your portfolio",
"required": true,
"answer": "https://adapark.design"
},
{
"label": "Will you now or in the future require visa sponsorship?",
"required": true,
"answer": "No"
}
],
"gaps": ["Shipping design systems at the scale of a multi-product company"]
}
```
After pass 2, one row per approved application:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
[
{
"role": "Product Designer, Editor at Figma",
"submitted": true,
"confirmation": "Thank you for applying. Your application has been received.",
"replay": "https://outcrawl.ai/play/ses_7Kq2mD"
},
{
"role": "Senior Product Designer at Linear",
"submitted": false,
"leftForYou": "The form asks for a salary expectation, which is required and was not in the approved answers. Nothing was submitted.",
"replay": "https://outcrawl.ai/play/ses_9TmW4c"
}
]
```
Add the missing answer to that file and run pass 2 for it again.
## What it costs
An agent run is billed for the model's usage, browser time and proxy usage, per [pricing](/docs/billing/pricing). A typical run costs about 15 credits a minute all in, proxy usage included. That is an estimate: the real figure depends on the task, and a page-heavy form costs more.
* **A draft** takes about 75 s: **about 19 credits**.
* **A submission** takes about 2.5 minutes: **about 38 credits**.
So **about 57 credits an application** you send, and 19 for one you skip. Each draft's `caps.budget` is 75 and each submission's is 115. On the Free plan a run takes one file of up to 2 MiB; Builder takes 5 files of up to 4 MiB.
## Make it yours
* Watch a submission as it happens: log the `watchUrl` from `job.submitted`, open it, and press take control in the live viewer to correct a field yourself before handing back. While you hold the wheel, the run bills browser time (1 credit a minute) and proxy usage only, with no model usage.
* Keep one resume per kind of role (`resume-design.md`, `resume-research.md`) and pick one per posting.
* Keep a log: append each outcome, with the date and replay link, to a CSV of everything you applied to.
* Ask pass 1 for a short note on each gap, so you can decide whether to apply at all before approving.
## Related
Attach a file to a run.
Runs, caps and typed results.
Enforced rules and guidance.
Take and release control of a live run.
# Turn a website into an API
Source: https://outcrawl.ai/docs/examples/turn-a-website-into-an-api
A signed-in portal with no API, served as typed JSON from an endpoint of your own.
Plenty of the systems a business runs on are portals with a login and no API: a supplier's dealer site, a carrier's account page, a government filing system. Someone on your team signs in and copies numbers out by hand. This program puts a small HTTP endpoint in front of one: `GET /orders/:po` runs an Outcrawl agent in a signed-in profile, and answers with typed JSON your other systems can call.
## What you need
* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* A [profile](/docs/account/profiles) signed in to the portal, its id in `PORTAL_PROFILE_ID`. The one-time script below makes it, with the password kept as a [secret](/docs/account/secrets).
* `npm install @outcrawl/sdk hono @hono/node-server` or `pip install outcrawl fastapi uvicorn`.
The agent types `{{secret:harbor_password}}` and the value is substituted below the model, so the password never appears in a prompt or a log. If the portal emails a sign-in code, connect an inbox [integration](/docs/account/integrations) and name it in `integrations` on this run.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const profile = await oc.profiles.create({ label: 'harbor-supply' });
await oc.secrets.create({ handle: 'harbor_password', kind: 'text', value: process.env.HARBOR_PASSWORD! });
const run = await oc.agent({
task:
'Go to https://dealers.harbor-supply.example/login and sign in as purchasing@yourco.com ' +
'with the password {{secret:harbor_password}}. Stop when the dashboard shows.',
profile: profile.id,
secrets: ['harbor_password'],
caps: { budget: '20.00' },
});
console.log(run.status, `PORTAL_PROFILE_ID=${profile.id}`);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import os
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
profile = await oc.profiles.create(label="harbor-supply")
await oc.secrets.create(handle="harbor_password", kind="text", value=os.environ["HARBOR_PASSWORD"])
run = await oc.agent(
task="Go to https://dealers.harbor-supply.example/login and sign in as purchasing@yourco.com "
"with the password {{secret:harbor_password}}. Stop when the dashboard shows.",
profile=profile["id"],
secrets=["harbor_password"],
caps={"budget": "20.00"},
)
print(run["status"], f"PORTAL_PROFILE_ID={profile['id']}")
asyncio.run(main())
```
## How it works
The [profile](/docs/account/profiles) holds the portal's cookies between runs, so each lookup starts signed in.
The endpoint submits an [agent run](/docs/agents/overview) with a JSON schema for an order. `found: false` is part of the schema, so a missing order is a 404 and not a guess.
The task only reads, and the per-run [rule](/docs/account/rules) `Never place, change, cancel or reorder an order.` is enforced, so the run cannot place an order. The PO number is checked against a strict pattern before it goes anywhere near a task.
Answers are cached for 10 minutes. Two callers asking for the same order share one run. A profile holds one session at a time, so lookups on it run one after another.
## The program
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { serve } from '@hono/node-server';
import { Hono } from 'hono';
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const PORTAL = 'https://dealers.harbor-supply.example';
const PROFILE_ID = process.env.PORTAL_PROFILE_ID!; // the profile that is signed in to the portal
const FRESH_FOR_MS = 10 * 60_000; // how long an answer is served from cache
interface Order {
found: boolean;
po: string;
status?: string;
orderedOn?: string;
shipBy?: string;
carrier?: string;
tracking?: string;
lines?: { sku: string; description: string; quantity: number; unitPrice: string }[];
}
const orderSchema = {
type: 'object',
properties: {
found: { type: 'boolean', description: 'false when the portal has no order with this number' },
po: { type: 'string' },
status: { type: 'string', description: 'the status exactly as the portal shows it' },
orderedOn: { type: 'string', description: 'YYYY-MM-DD' },
shipBy: { type: 'string', description: 'YYYY-MM-DD' },
carrier: { type: 'string' },
tracking: { type: 'string' },
lines: {
type: 'array',
items: {
type: 'object',
properties: {
sku: { type: 'string' },
description: { type: 'string' },
quantity: { type: 'integer' },
unitPrice: { type: 'string', description: 'with currency, e.g. $14.20' },
},
required: ['sku', 'quantity'],
},
},
},
required: ['found', 'po'],
};
const cache = new Map();
const inFlight = new Map>();
// A profile holds one session at a time, so runs on it go one after another.
let lane: Promise = Promise.resolve();
function lookUp(po: string): Promise {
const next = lane.then(async () => {
const run = await oc.agent({
task:
`On ${PORTAL}, open Orders, search for purchase order ${po} and open it. ` +
'Report its status, order date, ship-by date, carrier, tracking number and every line item. ' +
'If no order has that number, set found to false.',
profile: PROFILE_ID,
schema: orderSchema,
caps: { budget: '30.00' },
rules: ['Never place, change, cancel or reorder an order.'],
});
if (run.data === undefined) {
throw new Error(`run ${run.id} ended ${run.status} (${run.stopReason ?? 'no reason given'})`);
}
return run.data as Order;
});
lane = next.catch(() => undefined);
return next;
}
const app = new Hono();
app.get('/orders/:po', async (c) => {
const po = c.req.param('po').toUpperCase();
if (!/^[A-Z0-9-]{4,20}$/.test(po)) {
return c.json({ error: 'A purchase order number is 4 to 20 letters, digits or dashes.' }, 400);
}
const hit = cache.get(po);
if (hit !== undefined && Date.now() - hit.at < FRESH_FOR_MS) {
return c.json({ ...hit.order, cachedAt: new Date(hit.at).toISOString() }, hit.order.found ? 200 : 404);
}
// Two callers asking for the same order at once share one run.
let pending = inFlight.get(po);
if (pending === undefined) {
pending = lookUp(po).finally(() => inFlight.delete(po));
inFlight.set(po, pending);
}
try {
const order = await pending;
cache.set(po, { at: Date.now(), order });
return c.json(order, order.found ? 200 : 404);
} catch (err) {
return c.json({ error: String(err) }, 502);
}
});
serve({ fetch: app.fetch, port: 8787 });
console.log('GET http://localhost:8787/orders/PO-20931');
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
import os
import re
import time
from fastapi import FastAPI, HTTPException
from outcrawl import Outcrawl
PORTAL = "https://dealers.harbor-supply.example"
PROFILE_ID = os.environ["PORTAL_PROFILE_ID"] # the profile that is signed in to the portal
FRESH_FOR = 10 * 60 # seconds an answer is served from cache
ORDER_SCHEMA = {
"type": "object",
"properties": {
"found": {"type": "boolean", "description": "false when the portal has no order with this number"},
"po": {"type": "string"},
"status": {"type": "string", "description": "the status exactly as the portal shows it"},
"orderedOn": {"type": "string", "description": "YYYY-MM-DD"},
"shipBy": {"type": "string", "description": "YYYY-MM-DD"},
"carrier": {"type": "string"},
"tracking": {"type": "string"},
"lines": {
"type": "array",
"items": {
"type": "object",
"properties": {
"sku": {"type": "string"},
"description": {"type": "string"},
"quantity": {"type": "integer"},
"unitPrice": {"type": "string", "description": "with currency, e.g. $14.20"},
},
"required": ["sku", "quantity"],
},
},
},
"required": ["found", "po"],
}
oc = Outcrawl() # one long-lived client for the server
app = FastAPI()
cache: dict[str, tuple[float, dict]] = {}
in_flight: dict[str, asyncio.Task] = {}
lane = asyncio.Lock() # a profile holds one session at a time, so runs on it go one after another
async def look_up(po: str) -> dict:
async with lane:
run = await oc.agent(
task=f"On {PORTAL}, open Orders, search for purchase order {po} and open it. "
"Report its status, order date, ship-by date, carrier, tracking number and every line item. "
"If no order has that number, set found to false.",
profile=PROFILE_ID,
schema=ORDER_SCHEMA,
caps={"budget": "30.00"},
rules=["Never place, change, cancel or reorder an order."],
)
data = run.get("data")
if data is None:
raise RuntimeError(f"run {run['id']} ended {run['status']} ({run.get('stopReason', 'no reason given')})")
return data
@app.get("/orders/{po}")
async def get_order(po: str) -> dict:
po = po.upper()
if not re.fullmatch(r"[A-Z0-9-]{4,20}", po):
raise HTTPException(400, "A purchase order number is 4 to 20 letters, digits or dashes.")
hit = cache.get(po)
if hit is None or time.monotonic() - hit[0] >= FRESH_FOR:
# Two callers asking for the same order at once share one run.
task = in_flight.get(po)
if task is None:
task = asyncio.create_task(look_up(po))
in_flight[po] = task
task.add_done_callback(lambda _: in_flight.pop(po, None))
try:
order = await asyncio.shield(task) # a caller hanging up does not cancel the shared run
except RuntimeError as err:
raise HTTPException(502, str(err)) from err
hit = (time.monotonic(), order)
cache[po] = hit
order = hit[1]
if not order["found"]:
raise HTTPException(404, f"No order {po} on the portal.")
return order
# Run it: uvicorn orders:app --port 8787
```
## What you get back
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl http://localhost:8787/orders/PO-20931
```
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"found": true,
"po": "PO-20931",
"status": "Partially shipped",
"orderedOn": "2026-09-14",
"shipBy": "2026-09-29",
"carrier": "Old Dominion",
"tracking": "07731194625",
"lines": [
{ "sku": "HS-4410-BLK", "description": "Deck screw 3in, black, box of 350", "quantity": 40, "unitPrice": "$38.90" },
{ "sku": "HS-2208", "description": "Joist hanger 2x8, galvanised", "quantity": 250, "unitPrice": "$1.42" },
{ "sku": "HS-9001-50", "description": "Construction adhesive 28oz", "quantity": 60, "unitPrice": "$7.15" }
]
}
```
An unknown order answers `404`, a malformed number `400`, and a run that ended without data `502` with its status and stop reason.
## What it costs
An agent run is billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in, proxy usage included. A lookup takes about 45 seconds (orders list, search, the order, its shipment tab): **about 11 credits per lookup**, with a `caps.budget` of 30 for each run. That is an estimate: the real figure depends on the task, and a heavy page costs more. A cache hit costs nothing, because it never reaches Outcrawl.
At 500 distinct lookups a month that is about 5,500 credits, inside a Builder plan's 10,000.
## Make it yours
* When the order page has a stable URL, swap the agent for `oc.scrape(url, { profile, formats: ['json'], schema })`: one page, 1 credit, a few seconds.
* For more throughput, sign three profiles in to the same account and give each its own queue.
* Add a second endpoint, `GET /invoices/:number`, with its own schema on the same profile and queue.
* Put the answer in a database instead of memory so the cache survives a restart.
## Related
Logins that last between runs.
Passwords the model never sees.
Schemas, caps and statuses.
The `json` format for pages you can address directly.
# Billing FAQ
Source: https://outcrawl.ai/docs/faq/billing
Frequently asked billing questions.
One credit balance. Your plan adds its allowance once per paid billing period (on Free, on the 1st of each month, UTC), and every scrape, crawl, search, browser session, agent run and monitor check draws from the same number. Credits you buy on top never expire. See [pricing](/docs/billing/pricing) and [credits](/docs/billing/credits).
Whatever expires soonest. In practice: this period's plan allowance first, then grants with an expiry, soonest first (one that ends before your period does is spent before the allowance), then the credits you bought, which never expire. Unused plan allowance lapses 24 hours after its period ends, the time the renewal has to be charged; bought credits carry over until spent.
For the AI model's usage, browser time and proxy usage, in credits. Browser time is 1 credit a minute for as long as the run is open, paused or waiting time included; proxy usage is 2,000 credits per GB. A typical run costs about 15 credits a minute all in. That is an estimate, and the actual cost depends on the task. Drive a browser yourself with the SDK, without an agent, and you pay only browser time and proxy usage.
1 credit per check, whether or not the page changed. A monitor with `judge: true` also bills the model's usage when it judges a change.
No. One credit balance, no separate proxy, browser or additional credit type. Captcha and challenge solving is bundled into the published rates and is never billed as its own line item.
New requests are refused once your spendable credits reach zero. Buy credits from **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)** on any plan (Free and paused workspaces buy at the Builder rate), or turn on auto top-up so it never gets there. `credits` and `usage` show what has been granted, spent and remains, so you can see it coming before a request is refused.
A workspace owner or admin chooses a threshold, an amount and a monthly cap on **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)**. Whenever spendable credits drop below the threshold (checked every minute), the default card is charged for the amount at your plan's price per credit, one charge at a time and never past the cap in a calendar month (UTC). It waits while your plan's renewal is being charged, since the new period's credits are about to arrive. The credits never expire. If a charge is declined, the bank asks to confirm it, or one top-up comes to cost more than the cap, auto top-up turns itself off and every owner gets an email; nothing is retried until someone turns it back on. See [auto top-up](/docs/billing/credits#auto-top-up).
A refund takes back the share of credits that payment added, in proportion to the amount refunded and rounded in your favour; plan credits whose period has already ended are not touched. A disputed payment's credits are removed while the dispute is open and added back if it is resolved in your favour. The share comes out of credits still on the balance first: what is left of that payment's credits (the refunded period's allowance, for a plan invoice), then your other credits, purchased credits included. Only what the balance cannot cover is owed, and the next credits to arrive pay that first. A refunded auto top-up, or a dispute on any payment, also turns auto top-up off and emails the owners.
From **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)**, pick a plan and confirm. An upgrade applies immediately and starts a new monthly billing period today: before you confirm, you see the exact amount due today (the new plan's full month, less a credit for the unused time on your current plan) and the new renewal date, and it is charged to the card on file. A downgrade takes effect at the end of the period you already paid for, shown before you confirm, and charges nothing today.
From **[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)**, Cancel plan. You are asked for a reason, then offered the plan one step down, a pause, and a one-time discount, in turn; a pause or a discount you have already used is not offered again. Take any of them and cancellation stops there. Decline them all and the plan is set to end at the close of the current period, with a one-click Resume on the billing page until then.
Nothing is charged while paused, for 1 to 3 months, and a plan can pause once every 12 months. The plan's allowance does not renew while paused; the credits already on the workspace stay. Your first workspace runs on Free's limits and can buy credits at the Builder rate; any other workspace has no plan while paused and runs nothing until the plan returns. On the date you chose, billing resumes: the plan returns with the next renewal on your usual billing date, or at once if you pay the restart shown on the billing page (one month today, restarting the monthly period from today). Resuming early is one click, and charges nothing unless the pause has already skipped a renewal, in which case it is that same restart; the billing page shows the amount before anything is charged.
Cancelling offers a one-time 30% discount for the next 3 months, once per workspace, as one of the steps before the cancellation itself goes through.
**[https://outcrawl.ai/console/billing](https://outcrawl.ai/console/billing)** lists every invoice with a link to its PDF, alongside the credit grant history for accounts that have one.
# Capabilities FAQ
Source: https://outcrawl.ai/docs/faq/capabilities
Frequently asked questions about what Outcrawl can do.
Scrape (one URL, clean content), crawl (walk a site from a seed, stream every page), search (results you can act on, optionally rendered), browsers (a real browser your agent drives over plain CDP), and agents (hand over a task, it finishes in a real browser). Monitors watch a page on a schedule alongside them.
No. Live browser control needs a CDP connection held open in the caller's own process, so it exists only in the TypeScript SDK. Python, the CLI and MCP reach the same underlying page work through `agent` and `scrape`, which run the loop on the platform's own side instead.
A pinned exit and persistent storage that keeps a login across sessions, rather than signing in fresh every time. See [profiles](/docs/account/profiles).
It asks, or a person takes the wheel. When a run needs something only you know, it pauses with the question on its status, and `agent.answer` resumes it from where it stopped. `agent.control` pauses the run with the browser still open and hands you a URL to drive directly; releasing it resumes the run on whatever page you left it on. See [human in the loop](/docs/agents/human-in-the-loop).
No. No route sends a running agent new instructions. A person acts on a live run by answering a question it asked, or with `control`, taking the browser and handing it back; `cancel` stops it. None of them changes what it was asked to do.
# General FAQ
Source: https://outcrawl.ai/docs/faq/general
Frequently asked questions about Outcrawl.
Web infrastructure for AI agents: one account, one credit balance, five capabilities, scrape, crawl, search, browsers and agents, plus monitors for watching a page on a schedule. What you get back is a request that returns the page.
Developers and teams building agents or applications that touch the web: fetching pages reliably, walking a site, searching, driving a browser directly, or handing a task to an agent that finishes it in a real browser.
Yes. The Free plan includes 1,000 credits a month at no cost. See [pricing](/docs/billing/pricing) for every tier.
Sign in and connect an assistant with one command, no key needed, or generate an API key and make a request yourself. See the [quickstart](/docs/getting-started/quickstart).
No, for the REST API, curl or any HTTP client works. For a typed client, `npm install @outcrawl/sdk` or `pip install outcrawl`. For a terminal, the CLI is a single static binary.
Email **[hello@outcrawl.ai](mailto:hello@outcrawl.ai)**. Security reports go to **[security@outcrawl.ai](mailto:security@outcrawl.ai)** specifically.
# Security FAQ
Source: https://outcrawl.ai/docs/faq/security
Frequently asked security questions.
Email **[security@outcrawl.ai](mailto:security@outcrawl.ai)** with what you found, the steps to reproduce it, and its impact as you understand it.
A secret is stored under a name an agent can reference but never read back. No route on any surface returns a stored value. An agent reaches one only by naming it at runtime, and the substitution happens below the model.
No, deliberately. Secret and integration management both carry no MCP tool: an assistant must never gain the ability to create, list or delete a credential, or to grant itself reach outside the browser. Both stay human-only, through the CLI, an SDK or the console. A secret's value only ever reaches the CLI or an SDK; the console lists what is stored and revokes it.
A person, in a browser. `sessions.get` returns a replay URL for exactly that purpose, verifying what a charge actually bought, and it carries no MCP tool or CLI command for the same reason: a model cannot watch a replay. A replay never autoplays, a person presses play.
# Key concepts
Source: https://outcrawl.ai/docs/getting-started/concepts
Core terminology and concepts on Outcrawl.
## The five things
Everything is one account, one API, one credit balance, five capabilities.
| Capability | What it is | Endpoint |
| - | - | - |
| **Scrape** | One URL, clean structured content instead of markup. | `POST /v1/scrape` |
| **Crawl** | Walk a site from a seed URL, stream every page as it is fetched. | `POST /v1/crawl` |
| **Search** | Web search, optionally with every hit rendered through the scrape pipeline. | `POST /v1/search` |
| **Browsers** | A real browser your agent drives directly, over plain CDP. TypeScript SDK only. | No REST route, SDK only |
| **Agents** | Hand over a natural-language task, it finishes in a real browser. | `POST /v1/agent` |
Alongside them, **monitors** watch a page on a schedule and notify on real change, and a handful of platform primitives (profiles, secrets, rules, integrations, sessions) support all five without being one of them.
## Jobs, and the shape they share
Three capabilities (crawl, agent, and a monitor's schedule) do not answer in one round trip. `crawl` and `agent` both follow the same shape: `POST` returns an id in milliseconds, and the work continues after the call returns.
```
submit -> { id, status: "running" }
|
|-- GET .../{id} status, spend, progress
|-- GET .../{id}/results what it produced so far, paginated
|-- POST .../{id}/cancel stop it, keep what was already billed
```
`crawl` streams as newline-delimited JSON while it runs; `agent` answers plain JSON with the id and returns at once, and you follow the run with the calls above. Either way, the id you get first is what every later call about that job needs, store it.
## Workspaces and organizations
A workspace is a billing and access boundary: its own credit balance, its own members, its own plan and its own payment method. Signing up gives you one, and **+ New workspace** in the console switcher opens another, up to five per person.
Three roles: **owner** (one at a time, though ownership can hand to someone else), **admin**, and **member**. All three can use the five capabilities; owner and admin can invite, remove, change a plan and see what it costs.
A workspace starts as a **personal workspace**: invoices name the person who opened it. **Switch to organization** in Settings moves invoicing to a company instead, its own legal name, address and tax ID, without moving anything else. The credits, the members and the plan stay exactly where they were.
Only a person's very first workspace ever starts on the Free plan's monthly credits. Every workspace after that starts with none: the Free allowance is a monthly ceiling, not a one-time gift, and starting a second workspace on it would mint a second free allowance out of the same verification check the first one already used. A later workspace works the moment it has a paid plan.
A workspace can be deleted by its owner once nothing paid is still active on it and it is not the account's only one; see [Settings](/docs/account/settings#danger-zone) for exactly what that takes and what it costs.
## Credits
One prepaid-style balance, shared across every capability. Every response carries a `usage` object stating exactly what that call cost, so nothing on a bill is a surprise from a number you could not already see. See [pricing](/docs/billing/pricing) for the exact rate per kind.
## Profiles
A pinned exit and persistent storage (cookies, logins) that survives across sessions. Pass a profile to a browser session or an agent run to keep a login rather than signing in fresh every time. A profile is leased while it is open, a second concurrent use of the same profile is refused rather than silently forking its state.
## Sessions
Every browser-driven run, whether a live `oc.browser()` session or an agent run, produces a session with a replay. `sessions.get` returns the replay URL, duration, page count and usage, the way a charge gets disputed: watch what it bought. A replay never autoplays, a person presses play.
## Secrets, rules and integrations
Three platform primitives that shape what an agent is allowed to do:
* **Secrets** store a credential under a handle an agent can reference but never read back. No route on any surface returns a stored value.
* **Rules** are plain-language sentences a workspace sets; each one comes back classified as `enforced` (a hard gate) or `guidance` (in the prompt only), so you know which kind you actually have.
* **Integrations** connect an external MCP server (Gmail, Slack, and similar) so a run can call its tools mid-task. Connecting one is a human-only action; a run only reaches an integration it is explicitly given at submit time.
## Authentication
Two bearer credentials, either works everywhere: an API key, `ock_.`, for code you write yourself; or an OAuth access token, `oat_<...>`, from signing in, what `outcrawl login` and an MCP client's first use both give you. See [API authentication](/docs/api/authentication).
# Quickstart
Source: https://outcrawl.ai/docs/getting-started/quickstart
Make your first API call in under five minutes.
Get from zero to your first scrape in under five minutes.
## Step 1: sign in
Go to [https://outcrawl.ai/console](https://outcrawl.ai/console) and sign in with Google, GitHub, an email code, or a magic link.
Every plan, including Free, starts with a credit balance. Scrape, crawl, search, browsers and agents all draw from the same one.
## Step 2: connect
The fastest path is connecting an AI assistant, one command, sign in over OAuth, no key to copy:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
claude mcp add --transport http outcrawl https://api.outcrawl.ai/mcp
```
See [MCP setup](/docs/mcp/setup) for Cursor, Codex, and any other client.
Writing your own code instead? Create an API key from **[https://outcrawl.ai/console/keys](https://outcrawl.ai/console/keys)**, shown once, copy it immediately. Your key looks like `ock_.`.
The base URL is `https://api.outcrawl.ai/v1`. Every request carries your key as a bearer token.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY
const doc = await oc.scrape('https://example.com', { formats: ['markdown'] });
console.log(doc.markdown);
console.log(doc.usage); // what this call cost
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc: # reads OUTCRAWL_API_KEY
doc = await oc.scrape("https://example.com", formats=["markdown"])
print(doc.markdown)
print(doc.usage)
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/scrape \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"]
}'
```
Install a client first: `npm install @outcrawl/sdk` (TypeScript) or `pip install outcrawl` (Python). Both read `OUTCRAWL_API_KEY` from the environment, so you never have to paste the key into code.
The response comes back with the page content in the format you asked for, plus a `usage` object stating exactly what the call cost. One credit for that page, the same rate whether you called it directly, through crawl, or through search.
## Step 3: try the console playground
Prefer not to write code yet? The console has a playground for scrape, crawl and search at **[https://outcrawl.ai/console/playground](https://outcrawl.ai/console/playground)**. Run a request from the browser and it shows the equivalent curl, TypeScript and Python for what you just did.
## What else can you do?
Crawl streams every page from a seed URL, not just one.
Get results you can act on, optionally rendered.
Submit a natural-language task and it finishes in a real browser.
Live CDP control from the TypeScript SDK.
# MCP server
Source: https://outcrawl.ai/docs/mcp/overview
Connect Claude, Cursor or any MCP client to Outcrawl.
The Outcrawl MCP server exposes 29 tools, one per capability that carries one, generated from the same registry that drives the REST API and the CLI. An assistant can scrape, crawl, search, run and manage agents, get the files a run downloaded, manage monitors and profiles, read sessions and their network, and read usage and credits, directly from a conversation.
33 of the 62 capabilities deliberately carry no MCP tool. Session replays are watched, exported and shared by a person, not a model. Secret management never reaches an assistant, only a handle does, at runtime, inside a run it was already given. Connecting an integration grants an agent reach outside the browser, and that consent is a human's to give, not a model's to grant itself.
One command per client, sign in over OAuth, nothing to install.
The full reference of all 29 tools.
# Setup
Source: https://outcrawl.ai/docs/mcp/setup
Connect the Outcrawl MCP server to Claude Code, Cursor, Codex, or any MCP client.
Outcrawl runs the MCP server for you at **[https://api.outcrawl.ai/mcp](https://api.outcrawl.ai/mcp)**. There is no local package to install. Most clients connect with one command and sign in over OAuth on first use, no key to copy.
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
claude mcp add --transport http outcrawl https://api.outcrawl.ai/mcp
```
The first tool call opens a browser to sign in and pick a workspace. Nothing else to configure.
One click: Add to Cursor. It opens Cursor with the server filled in; the first tool call signs you in.
Or add it manually in Cursor's MCP settings:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"mcpServers": {
"outcrawl": {
"url": "https://api.outcrawl.ai/mcp"
}
}
}
```
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
codex mcp add outcrawl --url https://api.outcrawl.ai/mcp
codex mcp login outcrawl
```
Or add the block directly to `config.toml`:
```toml theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
[mcp_servers.outcrawl]
url = "https://api.outcrawl.ai/mcp"
```
Any client that speaks streamable HTTP and the MCP authorization spec needs nothing more than the URL:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"mcpServers": {
"outcrawl": {
"url": "https://api.outcrawl.ai/mcp"
}
}
}
```
For a client with no OAuth support at all, fall back to an API key in a header instead:
```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
"mcpServers": {
"outcrawl": {
"url": "https://api.outcrawl.ai/mcp",
"headers": { "Authorization": "Bearer ock_YOUR_KEY_ID.YOUR_SECRET" }
}
}
}
```
Generate a key from the console at **[https://outcrawl.ai/console/keys](https://outcrawl.ai/console/keys)**. This is the one place an API key belongs in an MCP config; every client above signs in instead.
## How sign-in works
MCP clients sign in with the browser flow OAuth 2.1 describes for public clients: an authorization code with PKCE and no client secret, and an access token scoped to the workspace you pick on the consent page. (The CLI's `outcrawl login` reaches the same account through the device flow instead, because a terminal has no browser redirect to catch.) A client that has never connected before registers itself automatically; there is nothing to create by hand. Removing a member from a workspace cuts off every token their sign-in produced, immediately.
## Rate limiting
Shared with the REST API: every request against your plan's `requestsPerMinute` ceiling, the MCP server included. See [plans](/docs/billing/pricing) for the figure per tier.
# Tools
Source: https://outcrawl.ai/docs/mcp/tools
Complete reference of all 29 MCP tools.
### Scrape and search
| Tool | Description |
| - | - |
| `outcrawl_scrape` | Fetch one URL, return clean content in the requested formats |
| `outcrawl_search` | Web search, optionally rendered through the scrape pipeline |
### Crawl
| Tool | Description |
| - | - |
| `outcrawl_crawl` | Walk a site from a seed URL; the whole stream folds into one tool response |
### Monitors
| Tool | Description |
| - | - |
| `outcrawl_monitor_create` | Watch a URL on a schedule, notify on real change |
| `outcrawl_monitor_list` | List monitors with their last check status |
| `outcrawl_monitor_delete` | Delete a monitor, stop its schedule |
### Agent
| Tool | Description |
| - | - |
| `outcrawl_agent` | Submit a task, get an id back in milliseconds |
| `outcrawl_agent_list` | Your runs, newest first, filterable by status and time |
| `outcrawl_agent_status` | Status, plan, spend, and the answer once there is one |
| `outcrawl_agent_events` | The step-by-step log from a cursor, every record in one response, with the cursor to resume from |
| `outcrawl_agent_results` | Records produced so far |
| `outcrawl_agent_cancel` | Stop a run |
| `outcrawl_agent_control` | Take the wheel of a live run, or hand it back |
| `outcrawl_agent_answer` | Answer the question a paused run asked |
| `outcrawl_agent_file_add` | Attach a file to a run, carried as base64 |
| `outcrawl_agent_downloads` | Every file the run downloaded, each with a link to its bytes that works for an hour |
A run pauses on its own when it needs something only you know: `outcrawl_agent_status` shows `paused` and the question, and `outcrawl_agent_answer` resumes it from where it stopped. `outcrawl_agent_control` is for a step you would rather do yourself: it takes the wheel with the browser still open and hands it back on whatever page you left it on.
### Profiles
| Tool | Description |
| - | - |
| `outcrawl_profile_create` | Create a profile that keeps logins |
| `outcrawl_profile_list` | List profiles |
| `outcrawl_profile_get` | Fetch one profile |
| `outcrawl_profile_delete` | Delete a profile |
### Sessions
| Tool | Description |
| - | - |
| `outcrawl_sessions_list` | Past sessions, newest first, filterable by status, task, profile and time |
| `outcrawl_sessions_get` | One session: status, pages, cost, replay link |
| `outcrawl_session_network` | What went wrong in a session's network, briefly: failed and slow requests |
| `outcrawl_session_request` | One request from a session in full, with its body |
| `outcrawl_session_downloads` | Every file a browser session you drove downloaded, each with a link to its bytes that works for an hour |
### Usage and credits
| Tool | Description |
| - | - |
| `outcrawl_usage` | Aggregate usage |
| `outcrawl_credits` | Granted, spent, remaining |
### Rules
| Tool | Description |
| - | - |
| `outcrawl_rules_get` | The workspace rules, enforced or guidance |
| `outcrawl_rules_set` | Replace the rules, get the classification back on the same call |
33 of the registry's 62 capabilities carry no MCP tool, deliberately. **Crawl job reads** (`outcrawl_crawl` already returns the whole crawl in one response). **Session replay streams, exports and sharing**: a model reads a session through the five session tools above; watching, exporting and sharing a replay is a person's act. **Secret management**, an assistant must never manage a credential, only address a handle a run was already given. **Integration management**, connecting a server grants an agent reach outside the browser, and that consent is a human's alone to give. An agent still calls an already-connected integration's tools mid-run once a person has granted a specific run access to it, that is a different act, made under the run's own submit-time grant, not one of the tools above.
# Overview
Source: https://outcrawl.ai/docs/monitors/overview
Watch a URL on a schedule, notify on real change.
A monitor watches a URL on a schedule and notifies you when it genuinely changes. Use it for recurring checks, a pricing page, a status page, a listing; not for a one-off comparison, that is just two scrapes.
The rate is **1 credit per check**, whether or not the page changed, so the schedule sets the floor: an hourly monitor is about 720 credits a month. A monitor created with `judge: true` runs a model over each change it finds, and also bills that model's usage in credits. See [pricing](/docs/billing/pricing).
## Notify targets
Three, and only for a monitor: Discord, Slack, or a bare webhook URL.
Monitor notifications are sent once. They are **not retried**. If your endpoint is down when a change fires, that notification is gone. `monitors.list` shows only when each monitor last checked and what it found (`lastCheckedAt`, `lastStatus`), not the diff, and there is no call that reads past checks, so keep your endpoint up. This is different from an agent run's webhook, which does retry, see [agent monitoring](/docs/agents/monitoring) for the contrast.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const monitor = await oc.monitors.create({
url: 'https://example.com/pricing',
schedule: '0 * * * *',
notify: { webhook: 'https://yourapp.example/hooks/outcrawl' },
});
const monitors = await oc.monitors.list();
console.log(monitor.id, monitors.length);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main():
async with Outcrawl() as oc:
monitor = await oc.monitors.create(
url="https://example.com/pricing",
schedule="0 * * * *",
notify={"webhook": "https://yourapp.example/hooks/outcrawl"},
)
monitors = await oc.monitors.list()
print(monitor["id"], len(monitors))
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/monitors \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/pricing",
"schedule": "0 * * * *",
"notify": { "webhook": "https://yourapp.example/hooks/outcrawl" }
}'
curl https://api.outcrawl.ai/v1/monitors -H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET"
```
Agent-run webhooks are a different mechanism, and unlike a monitor's, they do retry: up to 8 attempts over about 3 hours 40 minutes.
# Browsers
Source: https://outcrawl.ai/docs/scout/browsers
Run Scout on Outcrawl's cloud, on a local Chromium it launches, or on any browser you already run over CDP.
Scout drives any Chrome or Chromium, so moving between browsers is one line. Your models and your task stay the same. Pass exactly one of the three.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
// Outcrawl's cloud browser (recommended). Reads OUTCRAWL_API_KEY.
browser: 'outcrawl'
// A local Chromium, launched and closed by Scout.
browser: { launch: {} }
// Any browser you already run, over CDP.
browser: { cdpUrl: 'http://127.0.0.1:9222' }
```
Whichever you pick, Scout opens a fresh tab of its own for the run. When the run ends it closes that tab and its connection, and closes the browser only when Scout launched it or allocated it. A browser you passed in stays running.
## Outcrawl's cloud
**The best way to run Scout.** Many sites stop a browser that runs from a data center or behaves like a script; on the cloud, Scout gets a browser built to get through, with nothing to install. See [Outcrawl's cloud](/docs/scout/cloud) for what it adds and how a session is billed.
`browser: 'outcrawl'` uses the defaults. To set anything, pass `browser: { outcrawl: { ... } }`:
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
browser: {
outcrawl: { apiKey, proxy: 'http://user:pass@host:port', record: false },
}
```
| Option | Default | What it does |
| - | - | - |
| `apiKey` | `OUTCRAWL_API_KEY` from the environment | Your Outcrawl API key. Without one, the run throws `ScoutConfigError` before it opens anything. |
| `proxy` | an Outcrawl exit | Your own proxy URL for the session's traffic, instead of an Outcrawl exit. |
| `record` | `true` | Records the session for replay. `false` records nothing, and the result carries `recordingDeclined: true` instead of a `replayUrl`. |
| `baseUrl` | `https://api.outcrawl.ai` | The API's base URL. |
On the cloud the browser plans every pointer move, scroll and keystroke itself, so Scout sends it plain actions.
## A local Chromium
`browser: { launch: {} }`, or an empty `browser: {}`, starts a local browser for the run and closes it afterwards.
Scout uses `playwright-core` if it is installed, with the Chromium that Playwright downloads.
Otherwise it uses `puppeteer-core`
with the Chrome you have installed. Both are optional peer dependencies; install one.
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
npm i playwright-core && npx playwright install chromium
# or, to drive your installed Chrome
npm i puppeteer-core
```
| Option | Default | What it does |
| - | - | - |
| `headless` | `true` | Run without a window. `false` lets you watch it work. |
| `executablePath` | the Chromium Playwright downloads, or your installed browser | Path to the browser binary to run. |
| `args` | none | Extra command-line switches for the browser. Without a `--window-size` switch, Scout adds `--window-size=1280,800`. |
| `driver` | `playwright-core` if installed, then `puppeteer-core` | Which installed library launches it: `'playwright'` or `'puppeteer'`. |
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
browser: {
launch: {
headless: false,
executablePath: '/path/to/browser',
args: ['--lang=en-GB'],
driver: 'puppeteer',
},
}
```
If neither library is installed, or the one `driver` names is missing, the run throws `ScoutBrowserError` with the install command.
## A browser you already run
`browser: { cdpUrl }` connects to any browser that exposes the DevTools protocol. Pass a `ws://` or `wss://` WebSocket URL, or an `http://` or `https://` address such as `http://127.0.0.1:9222`, from which Scout reads the WebSocket URL at `/json/version`. Scout opens its own tab, closes it after the run, and leaves your browser running.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
browser: { cdpUrl: 'http://127.0.0.1:9222' } // or the ws:// webSocketDebuggerUrl
```
## Input
On a local or CDP browser, Scout plans pointer and keyboard input with its bundled input planner, `agenthands`. On the cloud it sends plain actions, because the browser plans input itself. To plan input your own way, pass a `HandsPort` as `hands` in the `Scout` options; it replaces the default on every browser.
Residential connections, captcha solving and replays.
Name a model for each job.
# Outcrawl's cloud
Source: https://outcrawl.ai/docs/scout/cloud
The best way to run Scout: one line gives it a browser sites treat like anyone at home, with captcha solving and a replay of every run.
Many sites stop a browser that runs from a data center or behaves like a script, and a task that stops halfway is no answer at all. On Outcrawl's cloud, Scout gets a browser built to get through. It is one line, and your models and your task stay the same:
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
// Reads OUTCRAWL_API_KEY from the environment.
const scout = new Scout({ browser: 'outcrawl', models });
```
![Your agent on the left reaches a browser on your own laptop, or a browser on Outcrawl's cloud on the right.]()
![Your agent on the left reaches a browser on your own laptop, or a browser on Outcrawl's cloud on the right.]()
## What the cloud adds
* **Residential connections.** Traffic leaves through ordinary home connections, so sites load the way they do for anyone at home. Pass `proxy` to send it through your own proxy instead.
* **Captcha solving.** Challenges are answered inside the browser, and the run carries on.
* **Natural input.** The browser plans every pointer move, scroll and keystroke itself, so Scout sends it plain actions.
* **Recordings.** Sessions are recorded by default, and `result.replayUrl` links to a step-by-step [replay](/docs/browsers/sessions) once the run ends. `record: false` turns it off.
* **Nothing to install.** No browser to download, update or keep running on your machine.
**Measured:** runs on Outcrawl's cloud reached 92.5% of pages on an 80-site benchmark, scored with the benchmark's own code; the next best reached 81.4%. Three full runs, 7 October 2026. [Method and every run](https://outcrawl.ai/resources/benchmarks).
## Options
| Option | Default | What it does |
| - | - | - |
| `apiKey` | `OUTCRAWL_API_KEY` from the environment | Your Outcrawl API key. |
| `proxy` | an Outcrawl exit | Your own proxy URL for the session's traffic. |
| `record` | `true` | Records the session for replay; `false` records nothing. |
| `baseUrl` | `https://api.outcrawl.ai` | The API's base URL. |
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
browser: {
outcrawl: { apiKey, proxy: 'http://user:pass@host:port', record: false },
}
```
Create a key from **[https://outcrawl.ai/console/keys](https://outcrawl.ai/console/keys)**. See [API keys](/docs/account/api-keys).
## How it is billed
A cloud run is billed as an ordinary Outcrawl browser session: browser session time at 1 credit per minute, plus proxy usage at 2,000 credits per GB. Captcha and challenge solving is bundled in and is never a separate line item. Your model calls still go to your own endpoints, and are not billed by Outcrawl.
Every account gets 1,000 free credits a month, no card needed. See [pricing](/docs/billing/pricing) for plans and the price of a credit.
A run on your own browser, launched locally or reached over CDP, makes no call to Outcrawl and is not billed.
A first task on the cloud in three steps.
Watch or export a session's replay.
# Examples
Source: https://outcrawl.ai/docs/scout/examples
Five working Scout examples: research, a form fill, a compare, a sign-in, and the same research on Outcrawl's cloud.
Tasks you would do in a browser yourself, written the way you would ask a colleague. Each is a working file in [`packages/scout/examples`](https://github.com/outcrawlai/scout/tree/main/packages/scout/examples), and each reads its models from the environment through [`examples/models.ts`](https://github.com/outcrawlai/scout/blob/main/packages/scout/examples/models.ts) (see [models from the environment](/docs/scout/models#models-from-the-environment)).
Each file's header gives the command that runs it from `packages/scout`, for example:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
SCOUT_BASE_URL=https://your-endpoint/v1 SCOUT_API_KEY=... SCOUT_HANDS_MODEL=your-model-id \
npx tsx examples/research.ts
```
The first four run on a local Chromium, `browser: { launch: {} }`. The last runs the research task on Outcrawl's cloud, the one line that changes.
## Research
The three most-commented stories on the Hacker News front page, as structured results. [`research.ts`](https://github.com/outcrawlai/scout/blob/main/packages/scout/examples/research.ts)
```ts wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const scout = new Scout({ browser: { launch: {} }, models: modelsFromEnv() });
const result = await scout.run(
'Go to https://news.ycombinator.com and report the three stories with the most comments on the front page: ' +
'title, link and comment count for each.',
{
caps: { steps: 15, duration: '8m' },
schema: { /* stories: [{ title, url, comments }], at most 3 */ },
onStep: (step) => console.log(`step ${String(step.index)}: ${step.action.description}`),
},
);
```
## Fill in a form
Text fields, radio buttons, checkboxes and a submit, then read back what the server received. [`form-fill.ts`](https://github.com/outcrawlai/scout/blob/main/packages/scout/examples/form-fill.ts)
```ts wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const result = await scout.run(
'Open https://httpbin.org/forms/post and order a pizza: customer name Ada Lovelace, telephone 555-0100, ' +
'email ada@example.com, size Large, toppings Bacon and Extra Cheese, delivery time 19:30, ' +
'delivery instructions "Ring twice". Submit the order, then report the customer name, size and toppings ' +
'exactly as the page that loads after submitting shows them.',
{
caps: { steps: 25, duration: '10m' },
schema: { /* customer, size, toppings */ },
},
);
```
## Compare
Open two books and compare price and stock in one answer. A run that moves across several pages is where a `brain` model helps, so this one suggests setting `SCOUT_BRAIN_MODEL` as well. [`compare.ts`](https://github.com/outcrawlai/scout/blob/main/packages/scout/examples/compare.ts)
```ts wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const result = await scout.run(
'On https://books.toscrape.com find the books "A Light in the Attic" and "Tipping the Velvet". ' +
"Open each book's own page and report its price and how many copies are in stock, then say which is cheaper.",
{
caps: { steps: 30, duration: '15m' },
// One named field per book rather than a list: Scout merges what each page
// gives it field by field, so the first book's details survive while it
// reads the second book's page. A list is replaced whole by each new read.
schema: { /* lightInTheAttic, tippingTheVelvet, cheaper */ },
},
);
```
## Sign in
Sign in, then read the page that is only there once you are signed in. The site is a public test site whose demo credentials are printed on its own login page. For a real site, read the credentials from your environment rather than writing them into the task. [`login.ts`](https://github.com/outcrawlai/scout/blob/main/packages/scout/examples/login.ts)
```ts wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const username = process.env['DEMO_USERNAME'] ?? 'student';
const password = process.env['DEMO_PASSWORD'] ?? 'Password123';
const result = await scout.run(
`Go to https://practicetestautomation.com/practice-test-login/ and sign in with username "${username}" and ` +
`password "${password}". Once you are signed in, extract the heading and the message of the page that welcomes you.`,
{
caps: { steps: 25, duration: '10m' },
schema: { /* heading, message */ },
},
);
```
## On Outcrawl's cloud
The research task again, on Outcrawl's cloud. One option changes, `browser: 'outcrawl'`: same agent, same models, same result shape, plus a replay link. Set `OUTCRAWL_API_KEY` as well. [`outcrawl.ts`](https://github.com/outcrawlai/scout/blob/main/packages/scout/examples/outcrawl.ts)
```ts wrap theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
// Reads OUTCRAWL_API_KEY. Pass { outcrawl: { apiKey, proxy, record } } to set them in code.
const scout = new Scout({ browser: 'outcrawl', models: modelsFromEnv() });
const result = await scout.run(/* the research task and schema */);
console.log(result.status, result.stopReason);
console.log(JSON.stringify(result.data, null, 2));
console.log('watch the run:', result.replayUrl ?? '(not recorded)');
```
Want the answer in a fixed shape, such as a list of names and prices? Pass a JSON Schema with the task, as every example here does, and the answer comes back in exactly that shape. See [running tasks](/docs/scout/runs#schema).
# Models
Source: https://outcrawl.ai/docs/scout/models
Name a model for each job Scout does, on any endpoint that speaks the chat completions API.
Scout has no built-in model and no default: no model id, no provider and no base URL. You name a model for each job you use, on any endpoint that speaks the chat completions API (`POST /chat/completions`): a hosted provider, a gateway, or a server on your own machine. Each job can use its own endpoint and its own key.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const scout = new Scout({
browser: 'outcrawl',
models: {
hands: { model: 'your-fast-model', baseURL: urlA, apiKey: keyA },
brain: { model: 'your-strong-model', baseURL: urlB, apiKey: keyB, effort: 'low' },
eyes: { model: 'your-vision-model', baseURL: urlA, apiKey: keyA },
extract: { model: 'your-local-model', baseURL: localUrl, apiKey: '' },
},
});
```
## The jobs
| Job | What it does | What matters |
| - | - | - |
| `hands`, required | Reads the page's accessibility tree and picks each action: click, type, scroll, navigate, read. | Called on every step, so it sets most of a run's speed and cost. A fast model that follows a JSON schema reliably. |
| `brain`, optional | Decides every step from a compact view of the page and the run so far; `hands` carries the decision out. | Helps long tasks that cross several pages or need a plan. A stronger reasoning model. |
| `eyes`, optional | Answers questions about a screenshot when the agent asks to look at the page. | Must accept images. Without it the agent works from the accessibility tree alone. |
| `extract`, optional | Reads a page into the answer or into your schema. | Long pages and exact fields. Without it `hands` extracts. |
`models` takes only these four keys and `rates`. Anything else throws `ScoutConfigError`.
## What each job takes
| Field | Required | What it is |
| - | - | - |
| `model` | yes | The model id, exactly as your endpoint expects it. |
| `baseURL` | yes | The endpoint's `http` or `https` base URL, the part before `/chat/completions`, for example `https://api.example.com/v1` or `http://localhost:11434/v1`. |
| `apiKey` | yes | Sent as a bearer token. `''` sends none, for a local server. |
| `effort` | no | How hard the model may think, for a model that takes a reasoning effort: `'minimal'`, `'low'`, `'medium'` or `'high'`. Absent sends none. |
| `structuredOutput` | no | How a JSON reply is asked for. Default `'tool_choice'`. See below. |
| `headers` | no | Extra headers sent with every request to this endpoint, as a map of header names to strings. |
### Structured replies
Scout asks each model for JSON. `structuredOutput` sets how:
* `'tool_choice'` (the default) forces one function call whose parameters are the schema. Function calling is the most widely supported way to get structured output across chat completions endpoints.
* `'json_object'` asks for any JSON and spells the schema out in the prompt, for an endpoint without function calling.
* `'strict'` sends a strict `json_schema` response format, for an endpoint that supports it.
If your endpoint refuses the mode, it answers 400 and the run's `error` names the job, its model and the two other settings to try, for example `set models.hands.structuredOutput to 'json_object' or 'strict'`. Scout does not switch modes on its own: you decide which one your endpoint takes.
## Cost
When your endpoint states a cost per call, `result.costUsd` is the run's model spend in USD. To price models yourself, and to hold a run to a [`caps.budget`](/docs/scout/runs#caps), pass `models.rates` with every job's model:
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
models: {
hands: { ... },
rates: {
'your-model-id': {
input: '0.30',
output: '1.20',
cacheRead: null,
cacheWrite: null,
inputModalities: ['text'],
},
},
}
```
Prices are USD per million tokens as decimal strings, keyed by model id. `cacheRead` and `cacheWrite` are `null` when the endpoint bills no such line, which is not the same as billing zero. `inputModalities` lists what the model accepts: `'text'`, and `'image'` for a model that reads screenshots. Once you pass `rates`, every job's model needs an entry, or `new Scout` throws `ScoutConfigError` rather than report a cost that reads as zero. Without rates and without a stated cost, `costUsd` is absent rather than zero.
## Models from the environment
The [examples](/docs/scout/examples) read their models from environment variables through a small helper, [`examples/models.ts`](https://github.com/outcrawlai/scout/blob/main/packages/scout/examples/models.ts). Scout itself reads none of these: the only variable it reads is `OUTCRAWL_API_KEY`, for the cloud browser. Start from the helper if the pattern suits you.
| Variable | What it sets |
| - | - |
| `SCOUT_BASE_URL` | Your endpoint's base URL, the part before `/chat/completions`. |
| `SCOUT_API_KEY` | Its key. Leave it empty for a local server that takes none. |
| `SCOUT_HANDS_MODEL` | Required: the model that reads the page and acts. |
| `SCOUT_BRAIN_MODEL` | Optional: decides each step on long, multi-page tasks. |
| `SCOUT_EYES_MODEL` | Optional: an image-capable model for questions about a screenshot. |
| `SCOUT_EXTRACT_MODEL` | Optional: reads pages into your schema. |
Any job can use its own endpoint with `SCOUT__BASE_URL` and `SCOUT__API_KEY`, for example `SCOUT_BRAIN_BASE_URL`.
# Open source
Source: https://outcrawl.ai/docs/scout/open-source
Scout's repository, how it stays in step with production, how to contribute, and its licence.
Scout's source, issues and pull requests are at [github.com/outcrawlai/scout](https://github.com/outcrawlai/scout). The package is `@outcrawl/scout` on npm.
## Inside the repository
| Package | What it is |
| - | - |
| [`packages/scout`](https://github.com/outcrawlai/scout/tree/main/packages/scout) | The front door: `new Scout({ browser, models }).run(task)`. |
| [`packages/runtime`](https://github.com/outcrawlai/scout/tree/main/packages/runtime) | The agent loop: look at the page, pick an action, act, check, repeat. |
| [`packages/model`](https://github.com/outcrawlai/scout/tree/main/packages/model) | The model side of the loop: prompts, the job split, structured replies, provider-neutral clients. |
| [`packages/scout-core`](https://github.com/outcrawlai/scout/tree/main/packages/scout-core) | Shared types and helpers. |
| [`packages/scout-cdp`](https://github.com/outcrawlai/scout/tree/main/packages/scout-cdp) | Drives a real browser over the DevTools protocol: navigation, clicks, typing, scrolling, tabs, snapshots. |
| [`packages/agenthands`](https://github.com/outcrawlai/scout/tree/main/packages/agenthands) | Pointer and keyboard input planning for local browsers, under its own MIT licence. |
To work on Scout itself:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
bun install
cd packages/runtime && bun test ./test
```
## How it stays in sync
The repository is an automatic mirror of the agent code Outcrawl runs in production. Every change lands first in the repository behind Outcrawl's hosted agent, ships with its tests, and then arrives on GitHub in a sync commit. What you install is the agent as it runs there.
## Contributing
Pull requests are welcome: bug reports, fixes and new examples alike. Because the repository is a mirror, nobody commits to it directly, and a pull request lands like this:
Open it on [outcrawlai/scout](https://github.com/outcrawlai/scout) as usual. Review comments happen there.
A maintainer imports it into the upstream repository as a branch. Your commits keep you as their author.
It is reviewed and tested there like any internal change, and ships to Outcrawl's fleet with its tests.
The next sync brings it back to GitHub inside a sync commit, and your pull request is closed with a link to that commit.
Before you open one:
* Run the tests of the package you changed: `cd packages/ && bun test ./test`.
* Typecheck it: `bunx tsc --noEmit -p tsconfig.test.json`.
* Keep model choices out of code, tests and examples. Scout names models by job (`brain`, `hands`, `eyes`, `extract`), and the user supplies the model for each.
* When you fix a bug, add a test that fails without your change.
### Sign your commits
Scout uses the [Developer Certificate of Origin](https://developercertificate.org/) instead of a CLA. Every commit carries a sign-off line certifying that you wrote the change or otherwise have the right to submit it under the project's licence:
```text theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
Signed-off-by: Your Name
```
`git commit -s` adds it for you. A pull request with an unsigned commit cannot be imported: amend with `git commit --amend -s`, or `git rebase --signoff main` for several commits, and push again.
### Security issues
Please do not open a public issue. Email [security@outcrawl.ai](mailto:security@outcrawl.ai) with the details.
## Licence
Apache-2.0, see [LICENSE](https://github.com/outcrawlai/scout/blob/main/LICENSE). By contributing you agree that your contribution is licensed under Apache-2.0, as section 5 of the licence describes. `packages/agenthands` keeps its MIT licence, and changes to it are licensed under MIT. Copyright 2026 Aeonmind LLC.
# Overview
Source: https://outcrawl.ai/docs/scout/overview
Scout is the open-source browser agent behind Outcrawl's hosted agent. Tell it the task, it opens a browser and gets it done.
Describe what you need in plain words. Scout opens a real browser, reads the page, clicks, types and scrolls until the job is done, then hands back the answer. If it cannot finish, it tells you what is missing and returns what it found.
Scout is the agent behind Outcrawl's [hosted agent](/docs/agents/overview), published as it runs there: the same loop, the same model harness and the same browser binding. It is open source under Apache-2.0 at [github.com/outcrawlai/scout](https://github.com/outcrawlai/scout), and installs from npm as `@outcrawl/scout`. You bring the browser and the models.
![A recorded Scout run on Outcrawl's cloud. The task: go to books.toscrape.com, open the first book in the Poetry category and report its title and price. Scout opens the shop, clicks Poetry, opens the first book and answers: title A Light in the Attic, price £51.77. Completed in 5 steps.]()
![A recorded Scout run on Outcrawl's cloud. The task: go to books.toscrape.com, open the first book in the Poetry category and report its title and price. Scout opens the shop, clicks Poetry, opens the first book and answers: title A Light in the Attic, price £51.77. Completed in 5 steps.]()
A recorded run on Outcrawl's cloud, 13 October 2026, shown at 6x speed with nothing cut.
* **Best on Outcrawl's cloud.** One line, `browser: 'outcrawl'`, gives Scout a browser that sites treat like anyone at home, with captcha solving and a replay of every run. It also drives a local browser it launches for you, or any browser you already run over the DevTools protocol.
* **Your models, one per job.** Any endpoint that speaks the chat completions API: a hosted provider, a gateway, or a server on your own machine. There is no default model.
* **An answer even when a run falls short.** A run that hits a limit or cannot finish still returns what it found, and says what is missing.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Scout } from '@outcrawl/scout';
const scout = new Scout({
browser: 'outcrawl',
models: {
hands: {
model: 'your-model-id',
baseURL: 'https://your-endpoint/v1',
apiKey: process.env.SCOUT_API_KEY ?? '',
},
},
});
const result = await scout.run(
'Find the top three stories on Hacker News right now',
);
console.log(result.data);
console.log(result.replayUrl); // watch the whole run
```
## Scout or the hosted agent
Both run the same agent. Pick by where you want the loop to live.
* **Scout** runs in your own process, on Node 22 or later. You name a model for each job and pay your model endpoints directly. On Outcrawl's cloud the browser session is billed in credits; on your own browser Outcrawl bills nothing.
* **The [hosted agent](/docs/agents/overview)** runs on Outcrawl's side behind one API call, from the SDKs, the CLI or MCP. A run is billed in credits for model usage, browser time and proxy usage, and you read its status and results separately.
Install, add your keys and run a first task on the cloud.
Outcrawl's cloud, a local Chromium, or any browser over CDP.
One model per job, on any chat completions endpoint.
Caps, schemas, live steps, the result and the errors.
Research, a form, a compare, a sign-in, and the cloud.
What the cloud adds, and how a session is billed.
# Quickstart
Source: https://outcrawl.ai/docs/scout/quickstart
Install Scout, add your keys and run a first task on Outcrawl's cloud.
Three steps to a first answer. On Outcrawl's cloud there is no browser to install.
You need Node 22 or later, or Bun.
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
npm i @outcrawl/scout
```
An Outcrawl key for the browser, free at [outcrawl.ai](https://outcrawl.ai) with 1,000 credits every month and no card needed. Then your model endpoint's base URL (the part before `/chat/completions`), its key, and the model id it expects.
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
export OUTCRAWL_API_KEY=your-outcrawl-key
export SCOUT_BASE_URL=https://your-endpoint/v1
export SCOUT_API_KEY=your-model-key
export SCOUT_HANDS_MODEL=your-model-id
```
Save this as `task.ts` and run `npx tsx task.ts`.
```ts task.ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Scout } from '@outcrawl/scout';
const scout = new Scout({
browser: 'outcrawl',
models: {
hands: {
model: process.env.SCOUT_HANDS_MODEL!,
baseURL: process.env.SCOUT_BASE_URL!,
apiKey: process.env.SCOUT_API_KEY ?? '',
},
},
});
const result = await scout.run(
'Go to https://books.toscrape.com, open the first book in the Poetry category and report its title and price.',
{
caps: { steps: 15, duration: '8m' },
schema: {
type: 'object',
properties: { title: { type: 'string' }, price: { type: 'string' } },
required: ['title', 'price'],
},
onStep: (step) => console.log(step.index, step.action.description),
},
);
console.log(result.status, result.data);
console.log(result.replayUrl); // the whole run, step by step
```
You see each step as it happens, then something like `completed { title: '...', price: '£..' }` and the replay link.
`browser: 'outcrawl'` reads `OUTCRAWL_API_KEY` from the environment. Scout has no default model, so `models.hands` is the one thing it cannot run without. See [models](/docs/scout/models) for the other jobs.
## Run on your own machine instead
Install `playwright-core` (or `puppeteer-core`) and a browser, then change the one line:
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
npm i playwright-core && npx playwright install chromium
```
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const scout = new Scout({
browser: { launch: {} }, // was: browser: 'outcrawl'
models: { hands: { /* unchanged */ } },
});
```
Pass `browser: { launch: { headless: false } }` to watch it work. Your models and your task stay the same. Replays come with the cloud, so `replayUrl` stays empty on your own browser.
Not writing the code yourself? Give this page to your coding agent and ask it to set Scout up and run your task.
Every browser option, and what the cloud adds.
Caps, schemas, cancelling, and everything a result carries.
# Running tasks
Source: https://outcrawl.ai/docs/scout/runs
scout.run(task, options): caps, schemas, cancelling, live steps, what a run returns, and the two errors it can throw.
`scout.run(task, options)` runs one task to an end and returns what it found. A run that falls short or reaches a cap still returns, with the data it collected. It throws only when it cannot start.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const controller = new AbortController();
const result = await scout.run(task, {
caps: { steps: 30, duration: '5m', budget: '0.50' },
schema: { ... },
signal: controller.signal,
onStep: (step) => console.log(step.action.description),
});
```
`task` is the task in words, as a non-empty string. Every option is optional. One `Scout` can run many tasks: each run opens its own browser, or its own tab on a browser you passed in, and closes it at the end.
## Caps
`caps` says when the run stops. Reaching a cap is a normal ending, not an error: the result carries what was found.
| Cap | Default | What it limits |
| - | - | - |
| `steps` | `40` | Most actions, as a positive whole number. |
| `duration` | `'10m'` | Wall clock, written as `ms`, `s`, `m` or `h`. |
| `budget` | none | Most model spend, in USD as a decimal string such as `'0.50'`. Needs [`models.rates`](/docs/scout/models#cost); without it, `run` throws `ScoutConfigError`. |
`steps` and `duration` always apply. A run that ends on a cap you set has `status: 'capped'`.
## Schema
`schema` is a JSON Schema for the answer. With one, `result.data` comes back in that shape, and is partly filled on a partial run. Without one, the answer is the agent's own summary, and `result.suggestedSchema` may carry a schema derived from the task that you can pass next time.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
type Book = { title: string; price: string };
const result = await scout.run(
'On https://books.toscrape.com, open the first Poetry book and report its title and price.',
{
schema: {
type: 'object',
properties: {
title: { type: 'string' },
price: { type: 'string' },
},
required: ['title', 'price'],
},
},
);
```
The type parameter only types `result.data`; the schema is what shapes it.
## Cancelling
`signal` takes an `AbortSignal`. Aborting it stops the run, which ends with `stopReason: 'cancelled'` and returns what it found so far.
## Watching steps
`onStep` is called with each step as it is recorded. It must not throw. A step carries, among other fields:
* `index`: its position, from 0;
* `action.description`: what it did, in words, and `action.method`, for example `click` or `fill`;
* `reasoning`: the model's stated reason, when it gave one;
* `usage`: the tokens it used.
## The result
`run` resolves to a `ScoutResult`:
| Field | What it is |
| - | - |
| `id` | The run's id. |
| `status` | `completed`, `partial` (fell short, see `error`), `capped` (a cap you set was reached) or `failed`. |
| `data` | The answer: schema-shaped with a schema, the agent's summary without one; partly filled on a partial run. Absent when the task asked nothing. |
| `stopReason` | Why it stopped, for example `task-complete`, `schema-satisfied`, `max-steps`, `max-duration`, `max-budget` or `cancelled`. |
| `error` | On a partial or failed run, what went wrong or what is still missing. |
| `steps`, `stepsUsed` | Every step in order, and how many there were. |
| `goals` | The plan the agent made and how each part ended. Absent when it made none. |
| `findings` | Facts the run established on the way, including on a run that ended early. |
| `suggestedSchema` | A schema derived from the task, when you passed none. |
| `usage` | Tokens used across the run. |
| `costUsd` | Model spend in USD, as a decimal string. Absent when it is not known; see [cost](/docs/scout/models#cost). |
| `durationMs`, `pages` | Wall time, and pages loaded. |
| `replayUrl` | On Outcrawl's cloud, a link to watch the run. Absent on other browsers, or when the recording was declined. |
| `recordingDeclined` | On Outcrawl's cloud, `true` when you passed `record: false`. |
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
if (result.status === 'completed') {
console.log(result.data);
} else {
// what is still missing, and what it found anyway
console.log(result.status, result.stopReason, result.error);
console.log(result.data);
}
```
## Errors
`new Scout` and `run` throw only when a run cannot start. Both errors are exported from `@outcrawl/scout`, and each message says what to pass instead.
| Error | When |
| - | - |
| `ScoutConfigError` | A configuration Scout cannot run with: no `models.hands`, a bad option, an empty task, a cap it cannot read, `caps.budget` without `models.rates`, or `browser: 'outcrawl'` without an API key. |
| `ScoutBrowserError` | The browser could not be started, reached or connected to: no `playwright-core` or `puppeteer-core` to launch with, a CDP address that does not answer, or the cloud refusing the session. |
Everything after the start, including a model endpoint that refuses a request, ends the run with a `status` and an `error` rather than a throw.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Scout, ScoutBrowserError, ScoutConfigError } from '@outcrawl/scout';
try {
const result = await scout.run(task);
} catch (error) {
if (error instanceof ScoutConfigError) {
console.error('fix the options:', error.message);
} else if (error instanceof ScoutBrowserError) {
console.error('no browser:', error.message);
} else {
throw error;
}
}
```
## Logging
Pass `logger` to `new Scout` to hear what Scout is doing. Any structured logger fits: it needs `info(event, fields)` and `warn(event, fields)`.
```ts theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const scout = new Scout({ browser: 'outcrawl', models, logger: console });
```
Scout logs which browser a run opened (`scout.browser`), a model whose cost could not be known (`scout.cost_unknown`), and a cloud session that published no replay link (`scout.no_replay_url`).
# Overview
Source: https://outcrawl.ai/docs/scrape/overview
Fetch one URL, get clean structured content back.
Scrape fetches one URL and returns clean content in the requested formats. Use it for a single known page, not for following links (that is [crawl](/docs/crawl/overview)) and not for a task that needs decisions along the way (that is [agents](/docs/agents/overview)).
The rate is **1 credit per page**, the same rate whether you called scrape directly or a page was fetched as part of a crawl or a rendered search result.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const doc = await oc.scrape('https://example.com', { formats: ['markdown'] });
console.log(doc.markdown);
console.log(doc.usage);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
doc = await oc.scrape("https://example.com", formats=["markdown"])
print(doc.markdown)
print(doc.usage)
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/scrape \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"]
}'
```
`formats: ['markdown']` is a real, verified option. By default a scrape returns only the main content of a page; pass the CLI's `--whole-page` flag (`onlyMainContent: false` on the other clients) to get the whole document rather than the extracted article body.
The same fetch, walking a whole site from a seed URL.
Find pages first, then scrape the ones worth reading.
# SDKs
Source: https://outcrawl.ai/docs/sdks/overview
Official Outcrawl SDKs for TypeScript and Python.
Outcrawl provides two official SDKs. There is no third, everything either one does not cover is reachable the same way: plain HTTP.
`npm install @outcrawl/sdk`, zero runtime dependencies, the only client with live browser control.
`pip install outcrawl`, pure Python, one wheel for every OS, async and context managed.
## Which one should I use?
| | TypeScript | Python |
| - | - | - |
| Package | `@outcrawl/sdk` | `outcrawl` |
| Install | `npm install @outcrawl/sdk` | `pip install outcrawl` |
| Registry | npm | PyPI |
| Requirement | Node.js, zero runtime deps | Python 3.10+, one dependency (`httpx`) |
| Async model | Promises | `async`/`await`, context managed |
| Scrape, crawl, search | Yes | Yes |
| Agents (submit, watch, cancel, hand off to a person) | Yes | Yes |
| Live browser control (`oc.browser()`) | Yes | No, TypeScript SDK only |
| Monitors, profiles, secrets, rules, integrations | Yes | Yes |
| Usage and credits | Yes | Yes |
The one real asymmetry: live browser control needs a CDP connection held open in the caller's own process, so it exists only in TypeScript. Python reaches the same underlying page work through `agent` and `scrape`, which run the loop on the platform's own side rather than the caller's, a difference in kind rather than a missing feature.
## Authentication
Both SDKs read the same credential the same way: an API key, `ock_.`, from the `OUTCRAWL_API_KEY` environment variable, or passed directly at construction. Generate one from the console at **[https://outcrawl.ai/console/keys](https://outcrawl.ai/console/keys)**.
```
ock_YOUR_KEY_ID.YOUR_SECRET
```
# Python SDK
Source: https://outcrawl.ai/docs/sdks/python
Official Python SDK for Outcrawl. Pure Python, one wheel for every OS, async and context managed.
The official Python client. Pure Python: one `py3-none-any` wheel for every operating system, no compiler required on any of them, one runtime dependency (`httpx`, itself pure Python).
## Installation
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
pip install outcrawl
```
Requires Python 3.10 or newer.
## Quick start
```python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc: # reads OUTCRAWL_API_KEY
doc = await oc.scrape("https://example.com", formats=["markdown"])
print(doc.markdown) # or doc["markdown"], it is the wire object
print(doc.usage) # every call returns what it cost
job = oc.agent(
task="Report the title of the top story on Hacker News.",
caps={"budget": "16.00"}, # the one ceiling a submit must carry
)
async for record in job: # the event cursor, while it runs
print(record["kind"], record["value"])
run = await job # and the settled run, on the same job
print(run["status"], run.get("data")) # data is absent until there is an answer
asyncio.run(main())
```
Async and context managed throughout, a genuinely different shape from the TypeScript client. `await` and `async for` on the same job object are two separate requests against one durable row, watching a run and having its answer is not a choice between them.
## Configuration
| Variable | Meaning | Default |
| - | - | - |
| `OUTCRAWL_API_KEY` | Your key | Required |
| `OUTCRAWL_API_URL` | Base URL | `https://api.outcrawl.ai/v1` |
The same two names the TypeScript SDK and the CLI read. Neither SDK uses the CLI's stored `outcrawl login` session: a script needs `OUTCRAWL_API_KEY` or a key passed in. Pass `Outcrawl(api_key=..., base_url=...)` to override per client.
## Surface
Every one of the 46 capabilities is reachable except live browser control.
* `oc.scrape(url, **options)` / `oc.crawl(url, **options)` / `oc.search(query, **options)`. `await oc.crawl(url)` gives you a crawl you read with `async for`, one page at a time as each is fetched.
* `oc.crawl.job(id)` reaches a crawl already in progress: `status()`, `results()`, `cancel()`. Useful when a connection drops, hanging up on a streaming crawl cancels it, but the pages already delivered stay yours; these routes are how you read what was already paid for.
* `oc.agent(task=..., caps=...)`. `caps` is required at runtime and `caps.budget` inside it; `caps.steps` and `caps.duration` are optional. The job is awaitable, async-iterable, and carries `status()`, `results()`, `events()`, `cancel()`, `control()`, `answer()`, `add_file()`, `downloads()`. `schema` accepts a JSON Schema object and constrains `run["data"]` to it, the same typed submit the TypeScript SDK documents.
* `oc.agent.downloads(run_id, cursor=None, limit=None)` and `oc.sessions.downloads(session_id, cursor=None, limit=None)` list every file a run or a browser session downloaded, kept 7 days. `await save_download(download, directory)` writes one into a folder that already exists, never overwrites, and answers its `Path`. See [downloads](/docs/agents/downloads).
* `oc.profiles`, `oc.secrets`, `oc.rules`, `oc.integrations`, `oc.sessions`, `oc.monitors`.
* `oc.usage(**query)` and `oc.credits()`. Every credit figure is a decimal string, use `decimal.Decimal`, never `float()`.
## What is not here
`oc.browser()`. Live browser control needs a CDP connection held open in the caller's own process, which is why it exists only in the TypeScript SDK. The same underlying page capability is reachable through `agent` and `scrape`, which run the loop on the platform's own side, a difference in kind rather than an omission. `Session.take_control()` is absent for the identical reason, it opens a CDP connection to a running session's own page so a person can drive it directly.
## Errors
The same classes, same fields, same messages as the TypeScript SDK, kept in parity by a shared test fixture so neither client can drift from the other silently.
```python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl, ProfileInUseError, QuotaExceededError
async def main() -> None:
async with Outcrawl() as oc:
profile = await oc.profiles.create(label="acme-user-42")
try:
doc = await oc.scrape("https://example.com", profile=profile.id)
print(doc.markdown)
except ProfileInUseError as leased:
print("profile busy until", leased.held_until)
except QuotaExceededError as over:
await asyncio.sleep(over.retry_after_seconds)
asyncio.run(main())
```
Branch on `error.code`, never on `str(error)`. See [errors](/docs/api/errors) for the full table.
# TypeScript SDK
Source: https://outcrawl.ai/docs/sdks/typescript
Official TypeScript SDK for Outcrawl. Zero runtime dependencies, native fetch, and the only client with live browser control.
The official TypeScript client. Zero runtime dependencies: Playwright is an optional peer, reached only through a dynamic import, so a caller who never opens a browser never installs it.
## Installation
```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
npm install @outcrawl/sdk
```
## Quick start
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl(); // reads OUTCRAWL_API_KEY; pass { apiKey } to override
const doc = await oc.scrape('https://example.com', { formats: ['markdown'] });
console.log(doc.usage); // every call returns what it cost, on the result that caused it
```
## Configuration
| Variable | Meaning | Default |
| - | - | - |
| `OUTCRAWL_API_KEY` | Your key | Required unless `apiKey` is passed |
| `OUTCRAWL_API_URL` | Base URL | `https://api.outcrawl.ai/v1` |
## Scrape, crawl, search
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const doc = await oc.scrape('https://example.com', { formats: ['markdown'] });
for await (const page of await oc.crawl('https://example.com')) {
console.log(page.url, page.markdown);
}
const results = await oc.search('outcrawl');
```
`await oc.crawl(url)` gives you a crawl to iterate: it streams every page as it is fetched, rather than waiting for the whole site. `oc.crawl.job(id)` reaches a crawl by id if the stream drops.
## Live browsers, TypeScript only
The one thing that does not exist in the Python SDK. `oc.browser()` allocates a session on the fleet and connects to it over CDP, nothing launches locally, and every request already leaves from a residential exit. You bring no infrastructure of your own to run.
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const profile = await oc.profiles.create({ label: 'demo-user-42' }); // minted once, reused by id
const browser = await oc.browser({ profile: profile.id }); // its cookies and logins persist
const page = await browser.newPage();
await page.goto('https://practicetestautomation.com/practice-test-login/'); // a public demo login
await page.fill('#username', 'student'); // human input model, the default
await page.fill('#password', 'Password123');
await page.click('#submit');
await page.raw.waitForURL('**/logged-in-successfully/');
console.log(await page.title()); // "Logged In Successfully | Practice Test Automation"
await browser.close();
```
`BrowserOptions` is exactly `{ profile, proxy, record }`. See [browsers](/docs/browsers/overview) and [human input](/docs/browsers/human-input) for what `click`, `fill`, `type`, `hover`, `press` and `selectOption` actually do by default, and why it matters.
## Agents
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const run = oc.agent({
task: 'Report the title of the top story on Hacker News.',
caps: { budget: '16.00' },
});
const result = await run; // await for the settled result
console.log(result.data);
for await (const step of run) {} // or iterate the event log, both on one job is fine
```
`caps.budget` is required, both at the type level and at runtime, a hard ceiling on what the run is billed. `caps.steps` and `caps.duration` are optional. A budget at or below the price of one page (1 credit) is refused before the run starts.
Give `agent` a JSON Schema and `result.data` comes back shaped to it instead of free text:
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
const typed = await oc.agent({
task: 'Report the title of the top story on Hacker News.',
caps: { steps: 20, budget: '16.00', duration: '5m' },
schema: { type: 'object', properties: { title: { type: 'string' } }, required: ['title'] },
});
if (typed.status === 'completed') console.log(typed.data); // { title: '...' }
else console.log(typed.status, typed.stopReason); // capped, partial or failed: data may be absent
```
An `AgentJob` also carries `status()`, `results()`, `events()`, `cancel()`, `control('take' | 'release', who?)`, `answer(answer, answeredBy?)`, `addFile({ name, bytes, contentType? })` and `downloads({ cursor?, limit? })`. See [agents](/docs/agents/overview) for what each one is for.
## Downloads
```typescript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl, saveDownload } from '@outcrawl/sdk';
const oc = new Outcrawl();
for (const download of await oc.agent.downloads('')) {
console.log(await saveDownload(download, './files')); // the path it wrote
}
```
`oc.agent.downloads(runId, { cursor?, limit? })` and `job.downloads()` list the files a run downloaded. `oc.sessions.downloads(sessionId)`, `session.downloads()` and `browser.downloads()` do the same for a browser you drive. Each answers an array of `Download` with `cursor` set when there is another page. `saveDownload(download, dir)` writes one file into a folder that already exists, never overwrites, and answers the path. See [downloads](/docs/agents/downloads).
## Platform primitives
`oc.sessions`, `oc.profiles`, `oc.monitors`, `oc.secrets`, `oc.rules`, `oc.integrations` and `oc.credits()` all follow the shapes documented under [account](/docs/account/profiles) and [billing](/docs/billing/credits).
## Errors
Every error class `errorFromWire` can produce is exported: `OutcrawlError`, `BadRequestError`, `CapExceededError`, `CapabilityUnavailableError`, `ConcurrencyLimitError`, `NoCapacityError`, `NotFoundError`, `PlanRequiredError`, `ProfileInUseError`, `ProfileNotFoundError`, `QuotaExceededError`, `UnauthorizedError` and more, plus `isOutcrawlError` and `isKnownErrorCode`. Branch on `error.code`, never on `error.message`. See [errors](/docs/api/errors) for the full 31-code table.
## Testing an unmodified Playwright script
The SDK is ergonomics, never the only door. A plain Playwright script pointed at the CDP endpoint `oc.browser()` opens keeps working without it.
# Overview
Source: https://outcrawl.ai/docs/search/overview
Web search, optionally rendered through the scrape pipeline.
Search finds pages. Use it to discover URLs, not to read one you already have (that is [scrape](/docs/scrape/overview)). Every hit can optionally be rendered through the scrape pipeline in the same call, so you get clean content back instead of a link list to fetch separately.
The rate is **5 credits per 10 results**, pro rata on the number of results actually returned.
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import { Outcrawl } from '@outcrawl/sdk';
const oc = new Outcrawl();
const results = await oc.search('outcrawl web infrastructure', { limit: 10 });
console.log(results.usage);
```
```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
import asyncio
from outcrawl import Outcrawl
async def main() -> None:
async with Outcrawl() as oc:
results = await oc.search("outcrawl web infrastructure", limit=10)
print(results.usage)
asyncio.run(main())
```
```bash curl theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl https://api.outcrawl.ai/v1/search \
-H "Authorization: Bearer ock_YOUR_KEY_ID.YOUR_SECRET" \
-H "Content-Type: application/json" \
-d '{
"query": "outcrawl web infrastructure",
"limit": 10
}'
```
Already have a URL? Read it directly.
Know the site, not the exact page? Walk it from a seed.
# Home
Source: https://outcrawl.ai/docs/welcome/home
Web infrastructure for AI agents. Your agent gets the page, every time.
> ## For agents
>
> Fetch the complete documentation index at: [https://outcrawl.ai/docs/llms.txt](https://outcrawl.ai/docs/llms.txt)
Outcrawl is web infrastructure for AI agents: the layer an agent stands on when it touches the web. One account, one credit balance, five things: scrape, crawl, search, browsers, agents. What you get back is a request that returns the page.
Make your first API call in under five minutes.
Credits, jobs and the shape every capability shares.
TypeScript and Python, typed end to end.
Every route, generated from the capability registry.
## The Outcrawl platform
Five things on one credit balance, plus the monitors and platform primitives that support them, all under one account and one bill.
One URL, clean structured content instead of markup.
Walk a site from a seed and stream every page.
Results you can act on, optionally rendered.
A real browser your agent drives over plain CDP.
Hand over a task, it finishes in a real browser.
Watch a page on a schedule, notify on real change.
## Explore examples
Real jobs an agent does in a real browser, each a complete program in TypeScript and Python.
One agent per service walks the cancel flow and stops at the last button for your yes.
Twenty agents, each browsing from a different country, report what locals pay.
One tool, browse(task), so your own assistant can use live websites and cite them.
Agents sign up to your product from three countries and file an issue when a step breaks.
## Start building
The full platform, plus live browser control over CDP.
Async and context managed, one wheel for every OS.
29 tools your assistant can call directly.
A single static binary. Scrape, crawl, search, agents, monitors and billing, all from your terminal.