Write a cited research report

Research by hand is a day of tabs: search a few angles, open twenty results, find that half the answers hide behind a country picker, and paste quotes into a document with their links. This program does the whole pass for one question. It searches each angle, reads every result in parallel with the claims extracted as it reads, sends an agent to the pages whose answer sits behind a control, and writes a markdown report in which every claim is numbered to its source.

What you need

An Outcrawl API key in OUTCRAWL_API_KEY.
npm install @outcrawl/sdk or pip install outcrawl. No other service: extraction happens on the page read.

How it works

Search each angle

One search per topic, five results each. A URL two topics both found is read once.

Read every result in parallel

Each page is scraped with the json format and a schema that asks for claims, each with the sentence it came from, and for two verdicts: is the page relevant, and is the answer behind a control on the page.

Send an agent where the page needs using

A page whose answer sits behind a destination picker or a plan selector goes to an agent run that uses the control and answers in the same schema. A per-run rule tells it not to sign in, buy anything or enter personal details, and the buying part is enforced.

Stay inside your browser slots

Every key on your account shares one browser slot count. Browsers, agent runs, crawls, searches and scrapes that aren’t served from cache each hold a slot. A small semaphore keeps the program below your plan’s limit, waits 20 seconds and retries on ConcurrencyLimitError, and waits out a per-minute QuotaExceededError for the seconds it names.

Write the report

Claims go under their topic with a numbered citation, and the sources list says which ones an agent read. claims.json keeps every quote so a reviewer can check each claim against its page.

The program

import { writeFile } from 'node:fs/promises';
import { setTimeout as sleep } from 'node:timers/promises';
import { ConcurrencyLimitError, Outcrawl, QuotaExceededError } from '@outcrawl/sdk';

const oc = new Outcrawl();

const QUESTION = 'What does it cost a US traveller to use their phone in Japan for one week?';
const TOPICS = [
  { heading: 'T-Mobile', query: 'T-Mobile international roaming Japan data price' },
  { heading: 'Verizon', query: 'Verizon TravelPass Japan daily price' },
  { heading: 'AT&T', query: 'AT&T International Day Pass Japan price' },
  { heading: 'Travel eSIMs', query: 'Japan eSIM 7 days data plan price' },
];

interface Claim {
  claim: string;
  quote: string;
}
interface Extracted {
  relevant: boolean;
  needsInteraction: boolean;
  claims: Claim[];
}

const claimsSchema = {
  type: 'object',
  properties: {
    relevant: { type: 'boolean', description: `whether this page helps answer: ${QUESTION}` },
    needsInteraction: {
      type: 'boolean',
      description:
        'true when the answer sits behind a control on this page (a destination picker, a plan selector, a tab) ' +
        'and is not in the text as loaded',
    },
    claims: {
      type: 'array',
      description: `facts on this page that answer: ${QUESTION}`,
      items: {
        type: 'object',
        properties: {
          claim: { type: 'string', description: 'one fact, with its price, allowance and conditions' },
          quote: { type: 'string', description: 'the sentence on the page it comes from, verbatim' },
        },
        required: ['claim', 'quote'],
      },
    },
  },
  required: ['relevant', 'needsInteraction', 'claims'],
};

// Stay below your plan's concurrent browsers (Builder: 5). Every search, uncached scrape and run holds one.
const SLOTS = 4;
let free = SLOTS;
const waiting: (() => void)[] = [];

async function withSlot<T>(work: () => PromiseLike<T>): Promise<T> {
  if (free > 0) free -= 1;
  else await new Promise<void>((wake) => waiting.push(wake));
  try {
    for (let attempt = 1; ; attempt += 1) {
      try {
        return await work();
      } catch (err) {
        // Another job on the account holds a slot: wait for one of yours to end.
        if (err instanceof ConcurrencyLimitError && attempt < 6) {
          await sleep(20_000);
          continue;
        }
        // Over the requests-per-minute ceiling: the error says how long to wait. Out of monthly credits: stop.
        if (err instanceof QuotaExceededError && err.ceiling === 'requests_per_minute' && attempt < 6) {
          await sleep(err.retryAfterSeconds * 1000);
          continue;
        }
        throw err;
      }
    }
  } finally {
    const next = waiting.shift();
    if (next !== undefined) next();
    else free += 1;
  }
}

// 1. Search each topic. One URL is read once, under the first topic that found it.
const sources: { topic: string; url: string; title: string }[] = [];
const hitsPerTopic = await Promise.all(TOPICS.map((t) => withSlot(() => oc.search(t.query, { limit: 5 }))));
TOPICS.forEach((topic, i) => {
  for (const hit of hitsPerTopic[i]!) {
    if (!sources.some((s) => s.url === hit.url)) sources.push({ topic: topic.heading, url: hit.url, title: hit.title });
  }
});

// 2. Read every page in parallel, extracting claims with the quote each one rests on.
const reads = await Promise.allSettled(
  sources.map((s) => withSlot(() => oc.scrape(s.url, { formats: ['json'], schema: claimsSchema }))),
);

// 3. Pages whose answer is behind a picker or a tab go to an agent that uses the page.
const found: { topic: string; url: string; title: string; via: string; claims: Claim[] }[] = [];
await Promise.all(
  sources.map(async (source, i) => {
    const read = reads[i]!;
    const page = read.status === 'fulfilled' ? (read.value.json as Extracted | undefined) : undefined;
    if (page === undefined || !page.relevant) return;
    if (!page.needsInteraction) {
      found.push({ ...source, via: 'read', claims: page.claims });
      return;
    }
    const run = await withSlot(() =>
      oc.agent({
        task:
          `Open ${source.url}. The answer to "${QUESTION}" for ${source.topic} is behind a control on this page, ` +
          'such as a destination picker or a plan selector. Choose Japan and a 7-day trip, then report every fact ' +
          'that answers the question, each with the sentence on the page it comes from.',
        schema: claimsSchema,
        caps: { budget: '40.00' },
        rules: ['Do not sign in, buy anything, or enter personal details.'],
      }),
    );
    const data = run.data as Extracted | undefined;
    if (data !== undefined) found.push({ ...source, via: 'agent', claims: data.claims });
  }),
);

// 4. Write the report: claims under each topic, numbered sources at the end.
const cited = sources.filter((s) => found.some((f) => f.url === s.url && f.claims.length > 0));
let report = `# ${QUESTION}\n\nResearched ${new Date().toISOString().slice(0, 10)}.\n`;
for (const topic of TOPICS) {
  report += `\n## ${topic.heading}\n\n`;
  for (const f of found.filter((x) => x.topic === topic.heading)) {
    const n = cited.findIndex((s) => s.url === f.url) + 1;
    for (const c of f.claims) report += `- ${c.claim} [${n}]\n`;
  }
}
report += '\n## Sources\n\n';
cited.forEach((s, i) => {
  const via = found.find((f) => f.url === s.url)?.via === 'agent' ? ', read by an agent' : '';
  report += `${i + 1}. [${s.title}](${s.url})${via}\n`;
});

await writeFile('report.md', report);
await writeFile('claims.json', JSON.stringify(found, null, 2));
console.log(`${cited.length} sources, ${found.reduce((n, f) => n + f.claims.length, 0)} claims: report.md`);

import asyncio
import datetime
import json
from pathlib import Path

from outcrawl import ConcurrencyLimitError, Outcrawl, QuotaExceededError

QUESTION = "What does it cost a US traveller to use their phone in Japan for one week?"
TOPICS = [
    {"heading": "T-Mobile", "query": "T-Mobile international roaming Japan data price"},
    {"heading": "Verizon", "query": "Verizon TravelPass Japan daily price"},
    {"heading": "AT&T", "query": "AT&T International Day Pass Japan price"},
    {"heading": "Travel eSIMs", "query": "Japan eSIM 7 days data plan price"},
]

CLAIMS_SCHEMA = {
    "type": "object",
    "properties": {
        "relevant": {"type": "boolean", "description": f"whether this page helps answer: {QUESTION}"},
        "needsInteraction": {
            "type": "boolean",
            "description": "true when the answer sits behind a control on this page (a destination picker, "
            "a plan selector, a tab) and is not in the text as loaded",
        },
        "claims": {
            "type": "array",
            "description": f"facts on this page that answer: {QUESTION}",
            "items": {
                "type": "object",
                "properties": {
                    "claim": {"type": "string", "description": "one fact, with its price, allowance and conditions"},
                    "quote": {"type": "string", "description": "the sentence on the page it comes from, verbatim"},
                },
                "required": ["claim", "quote"],
            },
        },
    },
    "required": ["relevant", "needsInteraction", "claims"],
}

# Stay below your plan's concurrent browsers (Builder: 5). Every search, uncached scrape and run holds one.
slots = asyncio.Semaphore(4)


async def with_slot(work):
    async with slots:
        for attempt in range(1, 7):
            try:
                return await work()
            except ConcurrencyLimitError:
                # Another job on the account holds a slot: wait for one of yours to end.
                if attempt == 6:
                    raise
                await asyncio.sleep(20)
            except QuotaExceededError as err:
                # Over the requests-per-minute ceiling: the error says how long to wait. Out of monthly credits: stop.
                if attempt == 6 or err.ceiling != "requests_per_minute":
                    raise
                await asyncio.sleep(err.retry_after_seconds)


async def main() -> None:
    async with Outcrawl() as oc:
        # 1. Search each topic. One URL is read once, under the first topic that found it.
        hits_per_topic = await asyncio.gather(
            *(with_slot(lambda t=t: oc.search(t["query"], limit=5)) for t in TOPICS)
        )
        sources: list[dict] = []
        for topic, hits in zip(TOPICS, hits_per_topic):
            for hit in hits:
                if all(s["url"] != hit["url"] for s in sources):
                    sources.append({"topic": topic["heading"], "url": hit["url"], "title": hit["title"]})

        # 2. Read every page in parallel, extracting claims with the quote each one rests on.
        reads = await asyncio.gather(
            *(with_slot(lambda s=s: oc.scrape(s["url"], formats=["json"], schema=CLAIMS_SCHEMA)) for s in sources),
            return_exceptions=True,
        )

        # 3. Pages whose answer is behind a picker or a tab go to an agent that uses the page.
        found: list[dict] = []

        async def finish(source: dict, read) -> None:
            page = None if isinstance(read, BaseException) else read.get("json")
            if not page or not page["relevant"]:
                return
            if not page["needsInteraction"]:
                found.append({**source, "via": "read", "claims": page["claims"]})
                return
            run = await with_slot(
                lambda: oc.agent(
                    task=f'Open {source["url"]}. The answer to "{QUESTION}" for {source["topic"]} is behind a control '
                    "on this page, such as a destination picker or a plan selector. Choose Japan and a 7-day trip, "
                    "then report every fact that answers the question, each with the sentence on the page it comes from.",
                    schema=CLAIMS_SCHEMA,
                    caps={"budget": "40.00"},
                    rules=["Do not sign in, buy anything, or enter personal details."],
                )
            )
            data = run.get("data")
            if data:
                found.append({**source, "via": "agent", "claims": data["claims"]})

        await asyncio.gather(*(finish(s, r) for s, r in zip(sources, reads)))

        # 4. Write the report: claims under each topic, numbered sources at the end.
        cited = [s for s in sources if any(f["url"] == s["url"] and f["claims"] for f in found)]
        number = {s["url"]: i + 1 for i, s in enumerate(cited)}
        report = f"# {QUESTION}\n\nResearched {datetime.date.today().isoformat()}.\n"
        for topic in TOPICS:
            report += f"\n## {topic['heading']}\n\n"
            for f in (f for f in found if f["topic"] == topic["heading"]):
                for c in f["claims"]:
                    report += f"- {c['claim']} [{number[f['url']]}]\n"
        report += "\n## Sources\n\n"
        via = {f["url"]: f["via"] for f in found}
        for s in cited:
            by_agent = ", read by an agent" if via[s["url"]] == "agent" else ""
            report += f"{number[s['url']]}. [{s['title']}]({s['url']}){by_agent}\n"

        Path("report.md").write_text(report)
        Path("claims.json").write_text(json.dumps(found, indent=2))
        print(f"{len(cited)} sources, {sum(len(f['claims']) for f in found)} claims: report.md")


asyncio.run(main())

What you get back

report.md (an illustrative sample; your run reports what the pages say on the day):

# What does it cost a US traveller to use their phone in Japan for one week?

Researched 2026-09-25.

## T-Mobile

- Go5G Plus and Go5G Next include unlimited data and texting in Japan at no extra cost, with 15 GB of high-speed data, then up to 256 kbps. [1]
- Calls from Japan cost $0.25 a minute on those plans. [1]

## Verizon

- TravelPass for Japan is $12 a day, charged only on days the phone is used abroad. [2]
- A monthly international plan covers 30 days for $100. [3]

## AT&T

- International Day Pass is $12 a day for the first device, with a monthly cap on charges per billing cycle. [4]

## Travel eSIMs

- 7-day Japan eSIM data plans range from $9 for 3 GB to $27 for unlimited data. [5] [6]

## Sources

1. [International Roaming with T-Mobile](https://www.t-mobile.com/travel-abroad-with-simple-global)
2. [TravelPass](https://www.verizon.com/plans/international/travelpass/), read by an agent
3. [International monthly plans](https://www.verizon.com/plans/international/)
4. [International Day Pass](https://www.att.com/international/international-day-pass/), read by an agent
5. [Japan eSIM plans](https://www.airalo.com/japan-esim)
6. [Japan eSIM 7 days](https://www.holafly.com/esim-japan/)

claims.json holds the same claims with their verbatim quotes:

[
  {
    "topic": "AT&T",
    "url": "https://www.att.com/international/international-day-pass/",
    "title": "International Day Pass",
    "via": "agent",
    "claims": [
      {
        "claim": "International Day Pass is $12 a day for the first device, with a monthly cap on charges per billing cycle.",
        "quote": "Japan: $12/day for the first device on your account"
      }
    ]
  }
]

What it costs

Search: 5 credits per 10 results. Four searches of 5 results is 20 results: 10 credits.
Reads: 1 credit per page. About 16 distinct pages: 16 credits.
Agents: billed for the model’s usage, browser time and proxy usage, and a typical run costs about 15 credits a minute all in. About 3 pages need an agent, each about a minute: 3 × 15 = about 45 credits, with a caps.budget of 40 for each run.

That is about 71 credits per report. The agent figure is an estimate that includes proxy usage; the real cost depends on the task, and a heavy page costs more. A Builder plan’s 10,000 monthly credits covers about 140 reports of this size.

Make it yours

Let your own model write the topics and queries from the question, and write prose from claims.json, keeping the [n] markers.
Add a second pass: search again for any topic that ended with no claims.
Raise limit on each search to 10 for a wider report; the search line doubles and the reads grow with it.
Set the semaphore to your plan’s browser count less one, so other jobs on the account still have a slot.

Search

Queries, limits and hits.

Scrape

The json format and schemas.

Agents

Schemas, caps and statuses.

Errors

ConcurrencyLimitError and QuotaExceededError.

Pricing

Every rate on this page.

Agent Examples

Scale Examples

Scheduled Examples

Build Examples

Write a cited research report

What you need

How it works

The program

What you get back

What it costs

Make it yours

Search

Scrape

Agents

Errors

Pricing

​What you need

​How it works

​The program

​What you get back

​What it costs

​Make it yours

​Related

Search

Scrape

Agents

Errors

Pricing

What you need

How it works

The program

What you get back

What it costs

Make it yours

Related