> ## Documentation Index
> Fetch the complete documentation index at: https://outcrawl.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Write a cited research report

> Search, read every source in parallel, send an agent where a page needs using, and get a report where every claim has a URL.

export const OutcrawlStyles = () => {
  if (typeof document !== 'undefined' && !document.getElementById('outcrawl-overrides')) {
    const s = document.createElement('style');
    s.id = 'outcrawl-overrides';
    s.textContent = `
      /* Match the product's radius grid (4 / 6 / 8 / 9999), not Mintlify's default. */
      .rounded-sm { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-md { border-radius: 0.375rem !important; } /* 6px, --r-ctl */
      .rounded-lg { border-radius: 0.5rem !important; }   /* 8px, --r-panel */
      .rounded-l-sm { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-sm { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-md { border-top-left-radius: 0.375rem !important; border-bottom-left-radius: 0.375rem !important; }
      .rounded-r-md { border-top-right-radius: 0.375rem !important; border-bottom-right-radius: 0.375rem !important; }
      .rounded-l-lg { border-top-left-radius: 0.5rem !important; border-bottom-left-radius: 0.5rem !important; }
      .rounded-r-lg { border-top-right-radius: 0.5rem !important; border-bottom-right-radius: 0.5rem !important; }

      /* Cards: never pure white in light mode, matching the one-big-card surface tone. */
      .card { background-color: #fcfcfc !important; border-radius: 0.5rem !important; }
      html.dark .card { background-color: #141311 !important; }

      /* Docs hero box */
      .oc-hero { background-color: #fcfcfc; border: 1px solid #e5e2dc; }
      html.dark .oc-hero { background-color: #141311; border-color: #26241f; }
      html.dark .oc-hero h1 { color: #edeae4; }

      /* Thin, transparent-track, hide-until-hover scrollbar. */
      ::-webkit-scrollbar { width: 6px; height: 6px; background-color: transparent; }
      ::-webkit-scrollbar-track { background-color: transparent; }
      ::-webkit-scrollbar-thumb { background-color: rgba(155, 155, 155, 0.5); border-radius: 10px; transition: opacity 0.3s ease; opacity: 0; }
      ::-webkit-scrollbar-thumb:hover { background-color: rgba(155, 155, 155, 0.7); }
      *:hover::-webkit-scrollbar-thumb,
      *:focus::-webkit-scrollbar-thumb,
      *:active::-webkit-scrollbar-thumb { opacity: 1; }
      * { scrollbar-width: thin; scrollbar-color: rgba(155, 155, 155, 0.5) transparent; }
    `;
    document.head.appendChild(s);
  }
  return null;
};

<OutcrawlStyles />

Research by hand is a day of tabs: search a few angles, open twenty results, find that half the answers hide behind a country picker, and paste quotes into a document with their links. This program does the whole pass for one question. It searches each angle, reads every result in parallel with the claims extracted as it reads, sends an agent to the pages whose answer sits behind a control, and writes a markdown report in which every claim is numbered to its source.

## What you need

* An Outcrawl API key in `OUTCRAWL_API_KEY`.
* `npm install @outcrawl/sdk` or `pip install outcrawl`. No other service: extraction happens on the page read.

## How it works

<Steps>
  <Step title="Search each angle">
    One [search](/docs/search/overview) per topic, five results each. A URL two topics both found is read once.
  </Step>

  <Step title="Read every result in parallel">
    Each page is [scraped](/docs/scrape/overview) with the `json` format and a schema that asks for claims, each with the sentence it came from, and for two verdicts: is the page relevant, and is the answer behind a control on the page.
  </Step>

  <Step title="Send an agent where the page needs using">
    A page whose answer sits behind a destination picker or a plan selector goes to an [agent run](/docs/agents/overview) that uses the control and answers in the same schema. A per-run [rule](/docs/account/rules) tells it not to sign in, buy anything or enter personal details, and the buying part is enforced.
  </Step>

  <Step title="Stay inside your browser slots">
    Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren't served from cache each hold a slot. A small semaphore keeps the program below your plan's limit, waits 20 seconds and retries on `ConcurrencyLimitError`, and waits out a per-minute `QuotaExceededError` for the seconds it names.
  </Step>

  <Step title="Write the report">
    Claims go under their topic with a numbered citation, and the sources list says which ones an agent read. `claims.json` keeps every quote so a reviewer can check each claim against its page.
  </Step>
</Steps>

## The program

<CodeGroup>
  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import { writeFile } from 'node:fs/promises';
  import { setTimeout as sleep } from 'node:timers/promises';
  import { ConcurrencyLimitError, Outcrawl, QuotaExceededError } from '@outcrawl/sdk';

  const oc = new Outcrawl();

  const QUESTION = 'What does it cost a US traveller to use their phone in Japan for one week?';
  const TOPICS = [
    { heading: 'T-Mobile', query: 'T-Mobile international roaming Japan data price' },
    { heading: 'Verizon', query: 'Verizon TravelPass Japan daily price' },
    { heading: 'AT&T', query: 'AT&T International Day Pass Japan price' },
    { heading: 'Travel eSIMs', query: 'Japan eSIM 7 days data plan price' },
  ];

  interface Claim {
    claim: string;
    quote: string;
  }
  interface Extracted {
    relevant: boolean;
    needsInteraction: boolean;
    claims: Claim[];
  }

  const claimsSchema = {
    type: 'object',
    properties: {
      relevant: { type: 'boolean', description: `whether this page helps answer: ${QUESTION}` },
      needsInteraction: {
        type: 'boolean',
        description:
          'true when the answer sits behind a control on this page (a destination picker, a plan selector, a tab) ' +
          'and is not in the text as loaded',
      },
      claims: {
        type: 'array',
        description: `facts on this page that answer: ${QUESTION}`,
        items: {
          type: 'object',
          properties: {
            claim: { type: 'string', description: 'one fact, with its price, allowance and conditions' },
            quote: { type: 'string', description: 'the sentence on the page it comes from, verbatim' },
          },
          required: ['claim', 'quote'],
        },
      },
    },
    required: ['relevant', 'needsInteraction', 'claims'],
  };

  // Stay below your plan's concurrent browsers (Builder: 5). Every search, uncached scrape and run holds one.
  const SLOTS = 4;
  let free = SLOTS;
  const waiting: (() => void)[] = [];

  async function withSlot<T>(work: () => PromiseLike<T>): Promise<T> {
    if (free > 0) free -= 1;
    else await new Promise<void>((wake) => waiting.push(wake));
    try {
      for (let attempt = 1; ; attempt += 1) {
        try {
          return await work();
        } catch (err) {
          // Another job on the account holds a slot: wait for one of yours to end.
          if (err instanceof ConcurrencyLimitError && attempt < 6) {
            await sleep(20_000);
            continue;
          }
          // Over the requests-per-minute ceiling: the error says how long to wait. Out of monthly credits: stop.
          if (err instanceof QuotaExceededError && err.ceiling === 'requests_per_minute' && attempt < 6) {
            await sleep(err.retryAfterSeconds * 1000);
            continue;
          }
          throw err;
        }
      }
    } finally {
      const next = waiting.shift();
      if (next !== undefined) next();
      else free += 1;
    }
  }

  // 1. Search each topic. One URL is read once, under the first topic that found it.
  const sources: { topic: string; url: string; title: string }[] = [];
  const hitsPerTopic = await Promise.all(TOPICS.map((t) => withSlot(() => oc.search(t.query, { limit: 5 }))));
  TOPICS.forEach((topic, i) => {
    for (const hit of hitsPerTopic[i]!) {
      if (!sources.some((s) => s.url === hit.url)) sources.push({ topic: topic.heading, url: hit.url, title: hit.title });
    }
  });

  // 2. Read every page in parallel, extracting claims with the quote each one rests on.
  const reads = await Promise.allSettled(
    sources.map((s) => withSlot(() => oc.scrape(s.url, { formats: ['json'], schema: claimsSchema }))),
  );

  // 3. Pages whose answer is behind a picker or a tab go to an agent that uses the page.
  const found: { topic: string; url: string; title: string; via: string; claims: Claim[] }[] = [];
  await Promise.all(
    sources.map(async (source, i) => {
      const read = reads[i]!;
      const page = read.status === 'fulfilled' ? (read.value.json as Extracted | undefined) : undefined;
      if (page === undefined || !page.relevant) return;
      if (!page.needsInteraction) {
        found.push({ ...source, via: 'read', claims: page.claims });
        return;
      }
      const run = await withSlot(() =>
        oc.agent({
          task:
            `Open ${source.url}. The answer to "${QUESTION}" for ${source.topic} is behind a control on this page, ` +
            'such as a destination picker or a plan selector. Choose Japan and a 7-day trip, then report every fact ' +
            'that answers the question, each with the sentence on the page it comes from.',
          schema: claimsSchema,
          caps: { budget: '40.00' },
          rules: ['Do not sign in, buy anything, or enter personal details.'],
        }),
      );
      const data = run.data as Extracted | undefined;
      if (data !== undefined) found.push({ ...source, via: 'agent', claims: data.claims });
    }),
  );

  // 4. Write the report: claims under each topic, numbered sources at the end.
  const cited = sources.filter((s) => found.some((f) => f.url === s.url && f.claims.length > 0));
  let report = `# ${QUESTION}\n\nResearched ${new Date().toISOString().slice(0, 10)}.\n`;
  for (const topic of TOPICS) {
    report += `\n## ${topic.heading}\n\n`;
    for (const f of found.filter((x) => x.topic === topic.heading)) {
      const n = cited.findIndex((s) => s.url === f.url) + 1;
      for (const c of f.claims) report += `- ${c.claim} [${n}]\n`;
    }
  }
  report += '\n## Sources\n\n';
  cited.forEach((s, i) => {
    const via = found.find((f) => f.url === s.url)?.via === 'agent' ? ', read by an agent' : '';
    report += `${i + 1}. [${s.title}](${s.url})${via}\n`;
  });

  await writeFile('report.md', report);
  await writeFile('claims.json', JSON.stringify(found, null, 2));
  console.log(`${cited.length} sources, ${found.reduce((n, f) => n + f.claims.length, 0)} claims: report.md`);
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
  import asyncio
  import datetime
  import json
  from pathlib import Path

  from outcrawl import ConcurrencyLimitError, Outcrawl, QuotaExceededError

  QUESTION = "What does it cost a US traveller to use their phone in Japan for one week?"
  TOPICS = [
      {"heading": "T-Mobile", "query": "T-Mobile international roaming Japan data price"},
      {"heading": "Verizon", "query": "Verizon TravelPass Japan daily price"},
      {"heading": "AT&T", "query": "AT&T International Day Pass Japan price"},
      {"heading": "Travel eSIMs", "query": "Japan eSIM 7 days data plan price"},
  ]

  CLAIMS_SCHEMA = {
      "type": "object",
      "properties": {
          "relevant": {"type": "boolean", "description": f"whether this page helps answer: {QUESTION}"},
          "needsInteraction": {
              "type": "boolean",
              "description": "true when the answer sits behind a control on this page (a destination picker, "
              "a plan selector, a tab) and is not in the text as loaded",
          },
          "claims": {
              "type": "array",
              "description": f"facts on this page that answer: {QUESTION}",
              "items": {
                  "type": "object",
                  "properties": {
                      "claim": {"type": "string", "description": "one fact, with its price, allowance and conditions"},
                      "quote": {"type": "string", "description": "the sentence on the page it comes from, verbatim"},
                  },
                  "required": ["claim", "quote"],
              },
          },
      },
      "required": ["relevant", "needsInteraction", "claims"],
  }

  # Stay below your plan's concurrent browsers (Builder: 5). Every search, uncached scrape and run holds one.
  slots = asyncio.Semaphore(4)


  async def with_slot(work):
      async with slots:
          for attempt in range(1, 7):
              try:
                  return await work()
              except ConcurrencyLimitError:
                  # Another job on the account holds a slot: wait for one of yours to end.
                  if attempt == 6:
                      raise
                  await asyncio.sleep(20)
              except QuotaExceededError as err:
                  # Over the requests-per-minute ceiling: the error says how long to wait. Out of monthly credits: stop.
                  if attempt == 6 or err.ceiling != "requests_per_minute":
                      raise
                  await asyncio.sleep(err.retry_after_seconds)


  async def main() -> None:
      async with Outcrawl() as oc:
          # 1. Search each topic. One URL is read once, under the first topic that found it.
          hits_per_topic = await asyncio.gather(
              *(with_slot(lambda t=t: oc.search(t["query"], limit=5)) for t in TOPICS)
          )
          sources: list[dict] = []
          for topic, hits in zip(TOPICS, hits_per_topic):
              for hit in hits:
                  if all(s["url"] != hit["url"] for s in sources):
                      sources.append({"topic": topic["heading"], "url": hit["url"], "title": hit["title"]})

          # 2. Read every page in parallel, extracting claims with the quote each one rests on.
          reads = await asyncio.gather(
              *(with_slot(lambda s=s: oc.scrape(s["url"], formats=["json"], schema=CLAIMS_SCHEMA)) for s in sources),
              return_exceptions=True,
          )

          # 3. Pages whose answer is behind a picker or a tab go to an agent that uses the page.
          found: list[dict] = []

          async def finish(source: dict, read) -> None:
              page = None if isinstance(read, BaseException) else read.get("json")
              if not page or not page["relevant"]:
                  return
              if not page["needsInteraction"]:
                  found.append({**source, "via": "read", "claims": page["claims"]})
                  return
              run = await with_slot(
                  lambda: oc.agent(
                      task=f'Open {source["url"]}. The answer to "{QUESTION}" for {source["topic"]} is behind a control '
                      "on this page, such as a destination picker or a plan selector. Choose Japan and a 7-day trip, "
                      "then report every fact that answers the question, each with the sentence on the page it comes from.",
                      schema=CLAIMS_SCHEMA,
                      caps={"budget": "40.00"},
                      rules=["Do not sign in, buy anything, or enter personal details."],
                  )
              )
              data = run.get("data")
              if data:
                  found.append({**source, "via": "agent", "claims": data["claims"]})

          await asyncio.gather(*(finish(s, r) for s, r in zip(sources, reads)))

          # 4. Write the report: claims under each topic, numbered sources at the end.
          cited = [s for s in sources if any(f["url"] == s["url"] and f["claims"] for f in found)]
          number = {s["url"]: i + 1 for i, s in enumerate(cited)}
          report = f"# {QUESTION}\n\nResearched {datetime.date.today().isoformat()}.\n"
          for topic in TOPICS:
              report += f"\n## {topic['heading']}\n\n"
              for f in (f for f in found if f["topic"] == topic["heading"]):
                  for c in f["claims"]:
                      report += f"- {c['claim']} [{number[f['url']]}]\n"
          report += "\n## Sources\n\n"
          via = {f["url"]: f["via"] for f in found}
          for s in cited:
              by_agent = ", read by an agent" if via[s["url"]] == "agent" else ""
              report += f"{number[s['url']]}. [{s['title']}]({s['url']}){by_agent}\n"

          Path("report.md").write_text(report)
          Path("claims.json").write_text(json.dumps(found, indent=2))
          print(f"{len(cited)} sources, {sum(len(f['claims']) for f in found)} claims: report.md")


  asyncio.run(main())
  ```
</CodeGroup>

## What you get back

`report.md` (an illustrative sample; your run reports what the pages say on the day):

```markdown theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
# What does it cost a US traveller to use their phone in Japan for one week?

Researched 2026-09-25.

## T-Mobile

- Go5G Plus and Go5G Next include unlimited data and texting in Japan at no extra cost, with 15 GB of high-speed data, then up to 256 kbps. [1]
- Calls from Japan cost $0.25 a minute on those plans. [1]

## Verizon

- TravelPass for Japan is $12 a day, charged only on days the phone is used abroad. [2]
- A monthly international plan covers 30 days for $100. [3]

## AT&T

- International Day Pass is $12 a day for the first device, with a monthly cap on charges per billing cycle. [4]

## Travel eSIMs

- 7-day Japan eSIM data plans range from $9 for 3 GB to $27 for unlimited data. [5] [6]

## Sources

1. [International Roaming with T-Mobile](https://www.t-mobile.com/travel-abroad-with-simple-global)
2. [TravelPass](https://www.verizon.com/plans/international/travelpass/), read by an agent
3. [International monthly plans](https://www.verizon.com/plans/international/)
4. [International Day Pass](https://www.att.com/international/international-day-pass/), read by an agent
5. [Japan eSIM plans](https://www.airalo.com/japan-esim)
6. [Japan eSIM 7 days](https://www.holafly.com/esim-japan/)
```

`claims.json` holds the same claims with their verbatim quotes:

```json theme={"theme":{"light":"github-light","dark":"catppuccin-mocha"}}
[
  {
    "topic": "AT&T",
    "url": "https://www.att.com/international/international-day-pass/",
    "title": "International Day Pass",
    "via": "agent",
    "claims": [
      {
        "claim": "International Day Pass is $12 a day for the first device, with a monthly cap on charges per billing cycle.",
        "quote": "Japan: $12/day for the first device on your account"
      }
    ]
  }
]
```

## What it costs

* Search: 5 credits per 10 results. Four searches of 5 results is 20 results: **10 credits**.
* Reads: 1 credit per page. About 16 distinct pages: **16 credits**.
* Agents: billed for the model's usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in. About 3 pages need an agent, each about a minute: 3 × 15 = **about 45 credits**, with a `caps.budget` of 40 for each run.

That is **about 71 credits per report**. The agent figure is an estimate that includes proxy usage; the real cost depends on the task, and a heavy page costs more. A Builder plan's 10,000 monthly credits covers about 140 reports of this size.

## Make it yours

* Let your own model write the topics and queries from the question, and write prose from `claims.json`, keeping the `[n]` markers.
* Add a second pass: search again for any topic that ended with no claims.
* Raise `limit` on each search to 10 for a wider report; the search line doubles and the reads grow with it.
* Set the semaphore to your plan's browser count less one, so other jobs on the account still have a slot.

## Related

<CardGroup cols={2}>
  <Card title="Search" icon="search" href="/docs/search/overview">Queries, limits and hits.</Card>
  <Card title="Scrape" icon="file-text" href="/docs/scrape/overview">The `json` format and schemas.</Card>
  <Card title="Agents" icon="player-play" href="/docs/agents/overview">Schemas, caps and statuses.</Card>
  <Card title="Errors" icon="alert-triangle" href="/docs/api/errors">`ConcurrencyLimitError` and `QuotaExceededError`.</Card>
  <Card title="Pricing" icon="tag" href="/docs/billing/pricing">Every rate on this page.</Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.