> ## Documentation Index
> Fetch the complete documentation index at: https://outcrawl.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Turn a website into an API

> A signed-in portal with no API, served as typed JSON from an endpoint of your own.

export const OutcrawlStyles = () => {
  if (typeof document !== 'undefined' && !document.getElementById('outcrawl-overrides')) {
    const s = document.createElement('style');
    s.id = 'outcrawl-overrides';
    s.textContent = `
      /* Match the product's radius grid (4 / 6 / 8 / 9999), not Mintlify's default. */
      .rounded-sm { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-md { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-lg { border-radius: 0.5rem !important; }   /* 8px, --r-panel */
      .rounded-l-sm { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-sm { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-md { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-md { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-lg { border-top-left-radius: 0.5rem !important; border-bottom-left-radius: 0.5rem !important; }
      .rounded-r-lg { border-top-right-radius: 0.5rem !important; border-bottom-right-radius: 0.5rem !important; }

      /* Cards: never pure white in light mode, matching the one-big-card surface tone. */
      .card { background-color: #fcfcfc !important; border-radius: 0.5rem !important; }
      html.dark .card { background-color: #141311 !important; }

      /* Docs hero box */
      .oc-hero { background-color: #fcfcfc; border: 1px solid #e5e2dc; }
      html.dark .oc-hero { background-color: #141311; border-color: #26241f; }
      html.dark .oc-hero h1 { color: #edeae4; }

      /* Thin, transparent-track, hide-until-hover scrollbar. */
      ::-webkit-scrollbar { width: 6px; height: 6px; background-color: transparent; }
      ::-webkit-scrollbar-track { background-color: transparent; }
      ::-webkit-scrollbar-thumb { background-color: rgba(155, 155, 155, 0.5); border-radius: 10px; transition: opacity 0.3s ease; opacity: 0; }
      ::-webkit-scrollbar-thumb:hover { background-color: rgba(155, 155, 155, 0.7); }
      *:hover::-webkit-scrollbar-thumb,
      *:focus::-webkit-scrollbar-thumb,
      *:active::-webkit-scrollbar-thumb { opacity: 1; }
      * { scrollbar-width: thin; scrollbar-color: rgba(155, 155, 155, 0.5) transparent; }
    `;
    document.head.appendChild(s);
  }
  return null;
};

<OutcrawlStyles />

Plenty of the systems a business runs on are portals with a login and no API: a supplier's dealer site, a carrier's account page, a government filing system. Someone on your team signs in and copies numbers out by hand. This program puts a small HTTP endpoint in front of one: `GET /orders/:po` runs an Outcrawl agent in a signed-in profile, and answers with typed JSON your other systems can call.

## What you need

* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* A [profile](/docs/account/profiles) signed in to the portal, its id in `PORTAL_PROFILE_ID`. The one-time script below makes it, with the password kept as a [secret](/docs/account/secrets).
* `npm install @outcrawl/sdk hono @hono/node-server` or `pip install outcrawl fastapi uvicorn`.

<Accordion title="Sign the profile in once">
  The agent types `{{secret:harbor_password}}` and the value is substituted below the model, so the password never appears in a prompt or a log. If the portal emails a sign-in code, connect an inbox [integration](/docs/account/integrations) and name it in `integrations` on this run.

  <CodeGroup>
    ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
    import { Outcrawl } from '@outcrawl/sdk';

    const oc = new Outcrawl();

    const profile = await oc.profiles.create({ label: 'harbor-supply' });
    await oc.secrets.create({ handle: 'harbor_password', kind: 'text', value: process.env.HARBOR_PASSWORD! });

    const run = await oc.agent({
      task:
        'Go to https://dealers.harbor-supply.example/login and sign in as purchasing@yourco.com ' +
        'with the password {{secret:harbor_password}}. Stop when the dashboard shows.',
      profile: profile.id,
      secrets: ['harbor_password'],
      caps: { budget: '20.00' },
    });

    console.log(run.status, `PORTAL_PROFILE_ID=${profile.id}`);
    ```

    ```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
    import asyncio
    import os

    from outcrawl import Outcrawl


    async def main() -> None:
        async with Outcrawl() as oc:
            profile = await oc.profiles.create(label="harbor-supply")
            await oc.secrets.create(handle="harbor_password", kind="text", value=os.environ["HARBOR_PASSWORD"])

            run = await oc.agent(
                task="Go to https://dealers.harbor-supply.example/login and sign in as purchasing@yourco.com "
                "with the password {{secret:harbor_password}}. Stop when the dashboard shows.",
                profile=profile["id"],
                secrets=["harbor_password"],
                caps={"budget": "20.00"},
            )
            print(run["status"], f"PORTAL_PROFILE_ID={profile['id']}")


    asyncio.run(main())
    ```
  </CodeGroup>
</Accordion>

## How it works

<Steps>
  <Step title="Keep the login in a profile">
    The [profile](/docs/account/profiles) holds the portal's cookies between runs, so each lookup starts signed in.
  </Step>

  <Step title="Answer each request with an agent run">
    The endpoint submits an [agent run](/docs/agents/overview) with a JSON schema for an order. `found: false` is part of the schema, so a missing order is a 404 and not a guess.
  </Step>

  <Step title="Keep it read-only">
    The task only reads, and the per-run [rule](/docs/account/rules) `Never place, change, cancel or reorder an order.` is enforced, so the run cannot place an order. The PO number is checked against a strict pattern before it goes anywhere near a task.
  </Step>

  <Step title="Cache, share and queue">
    Answers are cached for 10 minutes. Two callers asking for the same order share one run. A profile holds one session at a time, so lookups on it run one after another.
  </Step>
</Steps>

## The program

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import { serve } from '@hono/node-server';
  import { Hono } from 'hono';
  import { Outcrawl } from '@outcrawl/sdk';

  const oc = new Outcrawl();
  const PORTAL = 'https://dealers.harbor-supply.example';
  const PROFILE_ID = process.env.PORTAL_PROFILE_ID!; // the profile that is signed in to the portal
  const FRESH_FOR_MS = 10 * 60_000; // how long an answer is served from cache

  interface Order {
    found: boolean;
    po: string;
    status?: string;
    orderedOn?: string;
    shipBy?: string;
    carrier?: string;
    tracking?: string;
    lines?: { sku: string; description: string; quantity: number; unitPrice: string }[];
  }

  const orderSchema = {
    type: 'object',
    properties: {
      found: { type: 'boolean', description: 'false when the portal has no order with this number' },
      po: { type: 'string' },
      status: { type: 'string', description: 'the status exactly as the portal shows it' },
      orderedOn: { type: 'string', description: 'YYYY-MM-DD' },
      shipBy: { type: 'string', description: 'YYYY-MM-DD' },
      carrier: { type: 'string' },
      tracking: { type: 'string' },
      lines: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            sku: { type: 'string' },
            description: { type: 'string' },
            quantity: { type: 'integer' },
            unitPrice: { type: 'string', description: 'with currency, e.g. $14.20' },
          },
          required: ['sku', 'quantity'],
        },
      },
    },
    required: ['found', 'po'],
  };

  const cache = new Map<string, { at: number; order: Order }>();
  const inFlight = new Map<string, Promise<Order>>();
  // A profile holds one session at a time, so runs on it go one after another.
  let lane: Promise<unknown> = Promise.resolve();

  function lookUp(po: string): Promise<Order> {
    const next = lane.then(async () => {
      const run = await oc.agent({
        task:
          `On ${PORTAL}, open Orders, search for purchase order ${po} and open it. ` +
          'Report its status, order date, ship-by date, carrier, tracking number and every line item. ' +
          'If no order has that number, set found to false.',
        profile: PROFILE_ID,
        schema: orderSchema,
        caps: { budget: '30.00' },
        rules: ['Never place, change, cancel or reorder an order.'],
      });
      if (run.data === undefined) {
        throw new Error(`run ${run.id} ended ${run.status} (${run.stopReason ?? 'no reason given'})`);
      }
      return run.data as Order;
    });
    lane = next.catch(() => undefined);
    return next;
  }

  const app = new Hono();

  app.get('/orders/:po', async (c) => {
    const po = c.req.param('po').toUpperCase();
    if (!/^[A-Z0-9-]{4,20}$/.test(po)) {
      return c.json({ error: 'A purchase order number is 4 to 20 letters, digits or dashes.' }, 400);
    }

    const hit = cache.get(po);
    if (hit !== undefined && Date.now() - hit.at < FRESH_FOR_MS) {
      return c.json({ ...hit.order, cachedAt: new Date(hit.at).toISOString() }, hit.order.found ? 200 : 404);
    }

    // Two callers asking for the same order at once share one run.
    let pending = inFlight.get(po);
    if (pending === undefined) {
      pending = lookUp(po).finally(() => inFlight.delete(po));
      inFlight.set(po, pending);
    }

    try {
      const order = await pending;
      cache.set(po, { at: Date.now(), order });
      return c.json(order, order.found ? 200 : 404);
    } catch (err) {
      return c.json({ error: String(err) }, 502);
    }
  });

  serve({ fetch: app.fetch, port: 8787 });
  console.log('GET http://localhost:8787/orders/PO-20931');
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import asyncio
  import os
  import re
  import time

  from fastapi import FastAPI, HTTPException
  from outcrawl import Outcrawl

  PORTAL = "https://dealers.harbor-supply.example"
  PROFILE_ID = os.environ["PORTAL_PROFILE_ID"]  # the profile that is signed in to the portal
  FRESH_FOR = 10 * 60  # seconds an answer is served from cache

  ORDER_SCHEMA = {
      "type": "object",
      "properties": {
          "found": {"type": "boolean", "description": "false when the portal has no order with this number"},
          "po": {"type": "string"},
          "status": {"type": "string", "description": "the status exactly as the portal shows it"},
          "orderedOn": {"type": "string", "description": "YYYY-MM-DD"},
          "shipBy": {"type": "string", "description": "YYYY-MM-DD"},
          "carrier": {"type": "string"},
          "tracking": {"type": "string"},
          "lines": {
              "type": "array",
              "items": {
                  "type": "object",
                  "properties": {
                      "sku": {"type": "string"},
                      "description": {"type": "string"},
                      "quantity": {"type": "integer"},
                      "unitPrice": {"type": "string", "description": "with currency, e.g. $14.20"},
                  },
                  "required": ["sku", "quantity"],
              },
          },
      },
      "required": ["found", "po"],
  }

  oc = Outcrawl()  # one long-lived client for the server
  app = FastAPI()
  cache: dict[str, tuple[float, dict]] = {}
  in_flight: dict[str, asyncio.Task] = {}
  lane = asyncio.Lock()  # a profile holds one session at a time, so runs on it go one after another


  async def look_up(po: str) -> dict:
      async with lane:
          run = await oc.agent(
              task=f"On {PORTAL}, open Orders, search for purchase order {po} and open it. "
              "Report its status, order date, ship-by date, carrier, tracking number and every line item. "
              "If no order has that number, set found to false.",
              profile=PROFILE_ID,
              schema=ORDER_SCHEMA,
              caps={"budget": "30.00"},
              rules=["Never place, change, cancel or reorder an order."],
          )
      data = run.get("data")
      if data is None:
          raise RuntimeError(f"run {run['id']} ended {run['status']} ({run.get('stopReason', 'no reason given')})")
      return data


  @app.get("/orders/{po}")
  async def get_order(po: str) -> dict:
      po = po.upper()
      if not re.fullmatch(r"[A-Z0-9-]{4,20}", po):
          raise HTTPException(400, "A purchase order number is 4 to 20 letters, digits or dashes.")

      hit = cache.get(po)
      if hit is None or time.monotonic() - hit[0] >= FRESH_FOR:
          # Two callers asking for the same order at once share one run.
          task = in_flight.get(po)
          if task is None:
              task = asyncio.create_task(look_up(po))
              in_flight[po] = task
              task.add_done_callback(lambda _: in_flight.pop(po, None))
          try:
              order = await asyncio.shield(task)  # a caller hanging up does not cancel the shared run
          except RuntimeError as err:
              raise HTTPException(502, str(err)) from err
          hit = (time.monotonic(), order)
          cache[po] = hit

      order = hit[1]
      if not order["found"]:
          raise HTTPException(404, f"No order {po} on the portal.")
      return order


  # Run it: uvicorn orders:app --port 8787
  ```
</CodeGroup>

## What you get back

```bash theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
curl http://localhost:8787/orders/PO-20931
```

```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
{
  "found": true,
  "po": "PO-20931",
  "status": "Partially shipped",
  "orderedOn": "2026-09-14",
  "shipBy": "2026-09-29",
  "carrier": "Old Dominion",
  "tracking": "07731194625",
  "lines": [
    { "sku": "HS-4410-BLK", "description": "Deck screw 3in, black, box of 350", "quantity": 40, "unitPrice": "$38.90" },
    { "sku": "HS-2208", "description": "Joist hanger 2x8, galvanised", "quantity": 250, "unitPrice": "$1.42" },
    { "sku": "HS-9001-50", "description": "Construction adhesive 28oz", "quantity": 60, "unitPrice": "$7.15" }
  ]
}
```

An unknown order answers `404`, a malformed number `400`, and a run that ended without data `502` with its status and stop reason.

## What it costs

An agent run is billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in, proxy usage included. A lookup takes about 45 seconds (orders list, search, the order, its shipment tab): **about 11 credits per lookup**, with a `caps.budget` of 30 for each run. That is an estimate: the real figure depends on the task, and a heavy page costs more. A cache hit costs nothing, because it never reaches Outcrawl.

At 500 distinct lookups a month that is about 5,500 credits, inside a Builder plan's 10,000.

## Make it yours

* When the order page has a stable URL, swap the agent for `oc.scrape(url, { profile, formats: ['json'], schema })`: one page, 1 credit, a few seconds.
* For more throughput, sign three profiles in to the same account and give each its own queue.
* Add a second endpoint, `GET /invoices/:number`, with its own schema on the same profile and queue.
* Put the answer in a database instead of memory so the cache survives a restart.

## Related

<CardGroup cols={2}>
  <Card title="Profiles" icon="id" href="/docs/account/profiles">Logins that last between runs.</Card>
  <Card title="Secrets" icon="key" href="/docs/account/secrets">Passwords the model never sees.</Card>
  <Card title="Agents" icon="player-play" href="/docs/agents/overview">Schemas, caps and statuses.</Card>
  <Card title="Scrape" icon="file-text" href="/docs/scrape/overview">The `json` format for pages you can address directly.</Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.