create_dataset, add_dataset_items,
run_experiment, get_experiment, …), so an assistant connected to your
organization can build a dataset and run it in one conversation.
What an experiment runs against
The real agent: its published or draft instructions, its tools with their real credentials, its model or routing policy. By default every tool call an item triggers is a real call — a dataset item that asks the agent to send an email will send one. To keep a run off the outside world, give the item tool mocks: canned results the run serves in place of the real tool, with an optional policy that fails any item calling a tool it has no mock for. An experiment targets one agent version. Pass either the agent’s id (its active version runs) or a specific version id, including the draft’s. A draft is rebuilt fresh on every run, so the experiment reflects what you just saved. Networks are not a target: to test routing, run a golden set throughPOST /v1/copilots/{id}/run instead — see Designing a network.
Or a workflow. Send workflowId instead of agentId and every item runs
the workflow — the active version by default, workflowVersion: "draft" or a
number for another — with the item’s input as the run’s input, the object
the version’s input_schema describes. Items are checked against that schema
before anything runs; one that would fail is reported with its id (400)
unless strictInput: false. A workflow has no scorers attached, so the request
names them in scorers: catalogue types, each { "scorerType", "model"?, "options"? } or just the type. They grade the version’s output_text
rendered for that run — the text a scheduled run would post — against the
item’s groundTruth when the scorer wants one. Tool steps use the
organization’s credentials, or the runtimeAuth you send, exactly as
POST /v1/workflows/{id}/run does.
Datasets
A dataset is a list of items:
Add items in the Studio one at a time with the form, or in bulk from a CSV
with
input, groundTruth, requestContext and metadata columns
(JSON-valued cells are stored as JSON, everything else as text). A template
CSV is downloadable from the same dialog.
Or create the dataset from the API, items included:
agentId only records which agent the dataset was written for; the
experiment names its agent explicitly. Add more items later with
POST /v1/datasets/{id}/items, edit or remove one with PATCH / DELETE
on /v1/datasets/{id}/items/{itemId}. Every change bumps the dataset’s
version, and an experiment can pin one, so a run stays reproducible while
the dataset keeps growing — GET /v1/datasets/{id}/items?version=N shows
exactly what a pinned run saw.
Isolation
All items of a run write into one run-scoped thread whose memory is write-only: nothing is recalled between items, so an item can never see another item’s answer, and items run concurrently (five at a time by default). The thread exists so that every answer is replayable afterwards. To test a multi-turn flow deliberately, give consecutive items the samerequestContext.threadId. Those items share memory in order; run with
maxConcurrency: 1 so they do.
Tool mocks
An agent that sends emails, creates tickets or posts to a channel cannot be evaluated by running it for real on every dataset item. A tool mock on the item declares what a tool call should return, and the run serves that instead of executing the tool — the agent sees the result exactly as if the tool had run, and nothing leaves the platform.
Several mocks for the same tool are consumed in call order, one per call, so
an item that expects two lookups declares two mocks. A mock the agent never
calls is reported, not failed.
Failures are deterministic and not retried. A mocked tool called with
arguments no mock matches fails the item with
error.code
TOOL_MOCK_MISMATCH; a tool called more often than it has mocks fails with
TOOL_MOCK_EXHAUSTED. The item stops at the failing call, so the model can
make no further live calls.
Undeclared tools. By default a tool with no mock runs for real, which is
right when only the side-effecting tools are mocked and lookups may hit their
real backends. To forbid it, set unmockedToolPolicy to deny — on the item,
or for the whole run on POST /v1/experiments (the item’s value wins). Under
deny, the first call to a tool without a mock fails the item with
TOOL_MOCK_NOT_DECLARED before the tool executes.
Every result carries a toolMockReport:
liveCalls lists the unmocked tools that ran for real — non-empty means the
item was not fully deterministic. A failure entry (code, toolName,
args) appears only when a mocked tool was mis-called and the item failed. Tool calls on an item with mocks run one at a
time so consumption order is stable. A served mock does not execute the tool,
so it leaves no tool span in the trace and does not appear among the item’s
participants; the report is where to look.
Scorers
An experiment is scored by the scorers attached to the agent — the same scorer types, judge models and options that score its live traffic — applied to every item rather than sampled. There is nothing to choose at run time: to score differently, change the scorers on the agent (in the Studio, or withPATCH /v1/agents/{id}/draft) and run again. An agent with no scorers still
runs and records its answers, which is enough to read them side by side.
The catalogue an agent can pick from has two kinds:
- Code scorers run locally and need no model:
keyword-coverage,completeness,content-similarity,textual-difference,tone. - LLM judges ask a model to grade the answer:
answer-relevancy,answer-similarity,faithfulness,hallucination,bias,toxicity,prompt-alignment,noise-sensitivity,llm-tool-call-accuracy.
groundTruth on the item; the catalogue says which.
Judge spend is billed to a dedicated LLM Judges group with its own
monthly cap, so evaluation cost never lands on a customer’s bill or drains a
customer’s limit. See Traces, cost and billing groups.
Sampling and eligibility
Live traffic is sampled: each scorer on an agent has asampling_rate
between 0 and 1 (the share of runs it scores, default 1). Experiments ignore
it and score every item.
A scorer can also carry a filter — a predicate over the run, evaluated
before sampling, so the rate applies only to the runs the filter admits.
This is how one scorer judges production traffic for one group without a
separate agent per audience, and how judge spend stays proportional to what
is worth judging. The predicate reads requestContext.* (groupId,
environment, copilotId, source, userId…), entity.*, threadId and
resourceId; it cannot read the answer itself. A path that resolves to
nothing fails closed on the comparison operators (the run is not scored) and
passes on notIn / notExists.
eq / ne / lt / lte / gt / gte compare left and
right, each a { "path" } or a { "literal" }; in / notIn test a
value against a set; exists / notExists take a path; truthy /
falsy take a value; and / or take args; not takes arg.
PATCH /v1/agents/{id}/draft answers 400 for a filter that does not parse
or reads outside those roots, so a typo cannot save as a scorer that never
fires.
Running from the API
A workflow run, scored by two catalogue scorers:
skippedScorers appears when a scorer configured on the agent could not be
built — an LLM judge with no judge key, say — and the run went ahead with the
rest.
The X-User-Id header names who ran it, for score provenance. Usage limits
apply before any item starts: an organization or group over its cap gets a
429 and nothing runs.
A/B by variables
variables applies to every item; an item’s own requestContext.variables
is merged over it leaf by leaf. Running the same dataset twice with
different values — two tones, two escalation thresholds — is the cleanest
way to compare prompts, because nothing else changes.
Unlike chat, an experiment refuses to start when a placeholder with no
default has no value for some item, because otherwise it would bake a
literal {{userName}} into every affected item and bill you for the run.
The response reports it:
unused is usually a typo in a key and never blocks. Pass
"strictVariables": false to run anyway. See
Variables in experiments.
Reading results
The Studio’s results panel shows one row per item — input, output, ground truth, each scorer’s score and reason, cost, latency, and the participants (the tools and sub-agents that ran) — with run-level averages, total cost and the models that actually served. Every row has a thread badge that opens the item’s conversation read-only, exactly as the agent produced it, which is where you go when a score surprises you. Export produces a CSV with a column per output field, per scorer score and reason, plus cost, latency, status, trace id and thread id, named after the experiment. From the API,GET /v1/experiments/{experimentId} returns the same thing:
experiment.status moves from pending through running to completed
(or failed) — poll it after an async run. results holds one entry per
item with the input, the agent’s answer, the trace id of that run, the scores
it got and its toolMockReport (null when the item had no mocks), and
scores averages every score in the run per scorer:
GET /v1/datasets/{id}/experiments lists every run against a dataset, so two
runs — before and after a prompt change, or with different variables — can
be compared on their scores.
Comparing runs
Two runs of the same dataset — two agent versions, or one version before and after a prompt change — read side by side withPOST /v1/datasets/{id}/compare: for every item, the input, the ground
truth, and per run the answer and its scores per scorer. baselineId names
the run the others are compared against (default: the first id); up to ten
runs at once, all of that dataset.
DELETE /v1/experiments/{experimentId} removes a run and everything it
produced — its results, its scores and the traces of its items — and leaves
the dataset alone. Use it for runs made while tuning a prompt, so the
dataset’s history only keeps the runs worth comparing.
Scoring live traffic
The same scorers apply to real conversations after the fact. Select traces in the Studio and choose Score traces, or call the endpoint:scorerId that will run; read the scores back per trace
with GET /v1/traces/{traceId}/scores, narrowing on that scorerId (or on a
spanId) when the trace has been scored more than once:
source tells live scores (a scorer attached to the agent, scored as the
answer was produced) apart from on-demand ones. Combined with a filter on a
group or an agent, this is how you audit a week of production answers without
building a dataset first — and the traces that score badly are the ones worth
turning into dataset items.