Working with large APIs

Real business APIs return thousands of rows across dozens of pages. A model paging by hand is slow, expensive and silently incomplete: it fetches page one, sees twenty results, and reports “I found twenty”. This page covers the platform’s answer — the single most under-used capability in the product, and the fix for most “the agent only found six of them” incidents.

Two layers

Every tool an agent can call — API, MCP, connected app — gets a size guard unconditionally. A response that would not fit the conversation is cut at an item boundary, the full payload is kept aside for a few minutes, and the agent receives a note explaining what it is looking at and how to get the rest. On top of that, looping is opt-in per tool. Set loop_responses on the tool’s entry in an agent’s tool list and the tool gains a small set of reserved arguments that let the agent walk every page, fan out over a list of ids, and project only the fields it needs — inside one tool call.

{
  "agentTools": [
    { "tool_id": "TOOL_ID", "provider_id": "PROVIDER_ID", "loop_responses": true }
  ]
}

In the Studio the same switch is Allow response looping for this tool, on the tool card and in the agent’s tool list. Turning it on for any tool also attaches two retrieval tools, queryToolResult and fetchToolResult, that read a stored result without calling upstream again.

The flag lives on the agent’s binding, not on the tool. Two agents can hold the same tool with looping on for one and off for the other.

The reserved arguments

Looping adds these arguments to the tool’s input schema. They are stripped before the real request is built, so none of them ever reaches your API.

Argument	What it does
`_fields`	Project every item down to the listed dot-paths. Typically a 10–50× reduction in what enters the conversation.
`_all_pages`	Walk every page inside one step and return the merged list with a completeness verdict.
`_for_each`	Call once per value of a parameter and merge the rows, each tagged with the value it came from.
`_filter`	Filter the merged rows server-side before they reach the model.
`_refresh`	Bypass a cached result and fetch again.

`_fields`

{ "status": "open", "_fields": ["id", "customer.name", "total"] }

Paths map over arrays along the way, and the projection is always applied when set — not only when the result would overflow. The full, unprojected payload is still kept aside under a _resultId, so the agent can come back for a field it left out. Wrong paths don’t fail quietly. A path that resolves to nothing comes back as null, and a path that matched nothing on any item is reported in _unmatchedFields together with the items’ real top-level keys and a sample item. The agent corrects itself in one step instead of publishing twelve empty rows.

`_all_pages`

Offered only when the tool’s own schema declares a page-size parameter and a page-number or offset parameter. Recognised spellings:

Role	Parameter names
Page size	`itemsPerPage`, `items_per_page`, `perPage`, `per_page`, `pageSize`, `page_size`, `maxResults`, `max_results`, `limit`, `$top`
Page number	`page`, `pageNumber`, `page_number`, `pageIndex`, `page_index`
Offset	`offset`, `skip`, `$skip`, `startIndex`, `start_index`

The parameters may sit at the top level or one level down in a request body. There is nothing to configure: detection reads the schema you imported.

Cursor-based pagination (nextPageToken, Link headers) is not walked. A cursor-style endpoint still gets the size guard and _fields, but not _all_pages.

Page one is fetched alone, then pages go four at a time, in order; the walk stops at the first short page (pages requested past it in the same batch are discarded, and still count against the request budget), and on a bounded budget:

Bound	Default
Pages per collection	20
Requests per tool call	60
Items	5,000
Raw upstream bytes	24 MB
Wall-clock	60 s
Values per `_for_each`	50

A walk that hits a bound says so. The result carries _pagingComplete: false, _likelyMoreRows: true and a note naming the reason, so the agent reports a partial set as partial rather than passing it off as complete. Totals, counts and averages are only presented as sound when every page was fetched.

`_for_each`

{ "_for_each": { "key": "property_id", "values": ["p_1", "p_2", "p_3"] } }

One call per value, merged into a single list. Each row is tagged with the value it came from (an existing field of the same name is never overwritten — the API’s own answer wins). Values are de-duplicated and coerced to the parameter’s declared type. _for_each and _all_pages combine — every page of every value — under one shared budget. The key must be a parameter of this tool. Naming one that isn’t returns an invalid_for_each_key error that lists the tool’s real parameters. Per-value failures are reported in _forEachErrors alongside the rows that succeeded; only if every value fails does the call fail.

`_filter`

{ "_filter": [{ "field": "status", "op": "eq", "value": "open" }] }

Operators: eq, neq, gt, gte, lt, lte, contains, in, exists. Clauses are ANDed. _filter applies to the merged result of the current call — after the walk, after the fan-out — so the model never sees the rows it did not ask for. It is honoured but not advertised to the model on the source tool, because next to your API’s real filter parameters a model will reach for the reserved one and fetch everything to filter it afterwards. Use your API’s own filters first; _filter is for what they cannot express. To filter rows that were already fetched, with no upstream call at all, the agent uses queryToolResult on the stored _resultId — see below.

Stored results

When a response is cut, or a walk completes, the full payload is kept for about 15 minutes, scoped to the organization and the thread. The agent receives _resultId and two tools:

queryToolResult — fields, an optional filter, offset and limit over the stored items. Filtering runs on the full set before paging, so a clause may name a field that was not projected.
fetchToolResult — raw character slices for results that have no usable item boundary.

If a step ends with a truncated result and the agent has not yet queried it, the next step is restricted to the retrieval tools. That is what stops an agent from answering off a preview.

Caching

A complete walk can be cached so that the next identical call — same tool, same arguments, same fan-out values — is served without touching upstream. Cache entries are scoped by whose credentials fetched them: organization-level provider credentials share across the org, per-user runtime credentials are private to that user (or to a tenant-wide cache_scope your backend asserts at mint time). Defaults are 24 hours for org credentials and 1 hour for runtime credentials, with a per-binding override, dataset_ttl_hours. A cache hit is labelled: _cache: { hit: true, cachedAt, expiresAt } plus a note telling the agent that the user can get the very latest data by re-calling with _refresh: true. Re-calling with different _fields or _filter is still a hit — the stored set is unprojected and unfiltered.

Fixed arguments hit the cache every run. A workflow’s arguments are baked into its definition, so its cache key is identical on every scheduled run — exactly the runs that most need fresh data, with no human present to pass _refresh. In workflows the cache is therefore off unless the step opts in with a TTL, and it is worth opting in only where re-fetching is expensive and staleness up to the TTL is genuinely acceptable.

Caching is enabled per deployment; in shadow mode it records walks without serving them. Experiment runs never read from the cache.

Live progress

During a long walk the stream carries data-tool-loop-progress chunks, at most one every 750 ms, with the page and request counters (page, pagesMax, items, requests, requestsMax, elapsedMs) and, for a fan-out, the value currently being fetched. The same chunks flow through streamed workflow runs. Use them to render a progress line under the tool call; nothing in them is needed for correctness.

Tool discovery

A model chooses worse from a long list, and every tool schema it does not need still costs context on every step. Tool discovery keeps the visible list short: part of the agent’s roster is withheld from the prompt and reached through two meta-tools instead.

search_tools takes a query and returns the best matches with short descriptions.
load_tool takes a toolName (or toolNames) and makes those tools callable for the rest of the conversation.

An agent has one search surface, whatever is behind it: API imports, MCP server tools and connected-app toolkits all share the same index and the same pair of meta-tools. Which tools are loaded is read back from the conversation itself — the load_tool results in the thread — not from server memory. A loaded tool survives a restart and a new server instance, and a load that has scrolled out of the context window is forgotten by the model and the platform alike.

What is searchable

Tool search is on by default for any agent with more than 40 tools of its own. The per-agent option, saved with the draft and versioned with the rest of the configuration, tunes it or turns it off:

{
  "tool_search": {
    "threshold": 40,
    "providers": ["PROVIDER_ID", "gmail"],
    "always_loaded": ["list_orders"],
    "top_k": 8,
    "show_catalog": false
  }
}

Key	What it does
`enabled`	`false` turns search off for this agent. Omit it (or send `null` for the whole object) for the platform default, which is on.
`threshold`	Search applies only when the agent has more than this many tools of its own, connected-app tools included (default 40). Platform tools such as the workflow builder are not counted.
`providers`	Scope. A list of provider ids (API and MCP tools) and connected-app toolkit slugs; only those providers’ tools are searchable, everything else stays in the prompt. Leave it out to search the whole roster.
`always_loaded`	Tool names (or ids) that stay in the prompt on every step whatever the scope says — the two or three endpoints every conversation starts with.
`top_k`	Results per search (1–20, default 8).
`show_catalog`	Also list every searchable tool’s name and a one-line description in the prompt. One fewer round-trip per tool, at the cost of tokens on every step: worth it for a few dozen tools, not for hundreds.

Whatever the configuration says, these stay loaded:

the rendering components (showTable, showChart, …) the agent was given, so the model always knows it can render — they never count against the threshold either;
queryToolResult and fetchToolResult, so a stored _resultId can always be opened;
the workflow builder;
every tool with require_approval, because an approved call resumes against the tools the agent holds statically;
list_connections and connect_toolkit;
a connected-app entry with discovery: false.

Connected-app toolkits under the threshold

An agent under its threshold — or with search turned off — still gets connected-app toolkits pooled automatically, so a small agent never carries more schemas than it does today. Above a budget of 15 directly injected tools, toolkits are kept direct whole, smallest first, while they fit; the first toolkit that would not fit moves entirely behind search. That is how a three-tool WhatsApp toolkit stays directly callable next to a sixty-tool Gmail one. Override per entry with discovery: true (always pooled) or discovery: false (always direct, and counted against the budget). Your own API and MCP tools are never pooled by this budget rule; only the threshold above puts them behind search.

What the agent sees

The agent is told upfront that part of its roster is behind search, which integrations are pooled, and to search before concluding that it cannot do something. Tools the conversation loads are the agent’s own bindings — the size guard, looping arguments, pinned configuration and approval pauses apply exactly as if the tool had been in the prompt from the start. Tools supplied per request with the caller’s own credentials (see Tool auth) are never behind search: they stay directly callable, and the matching catalogue entry is hidden for that request so the caller’s credentials are the ones that apply.

search_tools is a keyword search over tool names and descriptions. The model finds what your descriptions say, so a catalogue imported with good summaries searches well and one with bare operation ids does not. Fix the description on the tool, not in the agent’s prompt.

In workflows

A tool step takes the same capabilities as a loop block, declared once rather than chosen per call:

{
  "id": "fetch",
  "type": "tool",
  "tool": "list_orders",
  "input": { "status": "open" },
  "loop": {
    "all_pages": true,
    "fields": ["id", "total"],
    "for_each": { "key": "property_id", "values": { "ref": "$.steps.properties.ids" } },
    "filter": [{ "field": "total", "op": "gt", "value": 100 }],
    "cache": { "ttl_minutes": 60 }
  }
}

With cache, "refresh": true makes the step ignore a stored set, walk upstream and store the fresh walk for the next reader: what a dashboard refresh sets on its sources, so the refresh stays live while the views that reuse the same walk read it back within the TTL. Pages are fetched one, then four at a time, so a long walk takes about a quarter of the sequential time. The step returns the tool’s own payload shape with the verdict under a single _loop key — complete, requests, itemCount, stoppedBy, note — so a branch can gate on $.steps.fetch._loop.complete. There is no _resultId in workflows: nothing downstream of a step could redeem one, so the walk’s byte budget is the only limit. See Building a workflow.

Guidance the agent already has

The agent learns these rules where it meets them. Each looping tool’s own schema describes _fields, _all_pages and _for_each before the first call, and every trimmed or paged result explains itself in its _note or _pagingNote: what was cut, whether the set is complete, and how to get the rest. On top of that, an agent with guarded tools carries a short block in its system prompt: read those notes, state totals only from a complete set, fetch the scope that was asked with the API’s own parameters, and describe coverage to the user in their terms rather than in parameter names. You do not need to repeat any of it in your own prompt. What is worth writing is which collections matter: “orders are paginated; always fetch every page before summarising” is a better instruction than an explanation of _all_pages.

Getting started

Build

Connect

Control

Automate

Operate

Working with large APIs

Two layers

The reserved arguments

`_fields`

`_all_pages`

`_for_each`

`_filter`

Stored results

Caching

Live progress

Tool discovery

What is searchable

Connected-app toolkits under the threshold

What the agent sees

In workflows

Guidance the agent already has

​Two layers

​The reserved arguments

​_fields

​_all_pages

​_for_each

​_filter

​Stored results

​Caching

​Live progress

​Tool discovery

​What is searchable

​Connected-app toolkits under the threshold

​What the agent sees

​In workflows

​Guidance the agent already has

Two layers

The reserved arguments

`_fields`

`_all_pages`

`_for_each`

`_filter`

Stored results

Caching

Live progress

Tool discovery

What is searchable

Connected-app toolkits under the threshold

What the agent sees

In workflows

Guidance the agent already has