create_dataset, add_dataset_items,
run_experiment, get_experiment, …), so an assistant connected to your
organization can build a dataset and run it in one conversation.
What an experiment runs against
The real agent: its published or draft instructions, its tools with their real credentials, its model or routing policy. Nothing is simulated. Every tool call an item triggers is a real call — a dataset item that asks the agent to send an email will send one — so point write tools at safe targets or leave them off the version under test. An experiment targets one agent version. Pass either the agent’s id (its active version runs) or a specific version id, including the draft’s. A draft is rebuilt fresh on every run, so the experiment reflects what you just saved. Networks are not a target: to test routing, run a golden set throughPOST /v1/copilots/{id}/run instead — see Designing a network.
Datasets
A dataset is a list of items:
Add items in the Studio one at a time with the form, or in bulk from a CSV
with
input, groundTruth, requestContext and metadata columns
(JSON-valued cells are stored as JSON, everything else as text). A template
CSV is downloadable from the same dialog.
Or create the dataset from the API, items included:
agentId only records which agent the dataset was written for; the
experiment names its agent explicitly. Add more items later with
POST /v1/datasets/{id}/items, edit or remove one with PATCH / DELETE
on /v1/datasets/{id}/items/{itemId}. Every change bumps the dataset’s
version, and an experiment can pin one, so a run stays reproducible while
the dataset keeps growing — GET /v1/datasets/{id}/items?version=N shows
exactly what a pinned run saw.
Isolation
All items of a run write into one run-scoped thread whose memory is write-only: nothing is recalled between items, so an item can never see another item’s answer, and items run concurrently (five at a time by default). The thread exists so that every answer is replayable afterwards. To test a multi-turn flow deliberately, give consecutive items the samerequestContext.threadId. Those items share memory in order; run with
maxConcurrency: 1 so they do.
Scorers
An experiment is scored by the scorers attached to the agent — the same scorer types, judge models and options that score its live traffic — applied to every item rather than sampled. There is nothing to choose at run time: to score differently, change the scorers on the agent (in the Studio, or withPATCH /v1/agents/{id}/draft) and run again. An agent with no scorers still
runs and records its answers, which is enough to read them side by side.
The catalogue an agent can pick from has two kinds:
- Code scorers run locally and need no model:
keyword-coverage,completeness,content-similarity,textual-difference,tone. - LLM judges ask a model to grade the answer:
answer-relevancy,answer-similarity,faithfulness,hallucination,bias,toxicity,prompt-alignment,noise-sensitivity,llm-tool-call-accuracy.
groundTruth on the item; the catalogue says which.
Judge spend is billed to a dedicated LLM Judges group with its own
monthly cap, so evaluation cost never lands on a customer’s bill or drains a
customer’s limit. See Traces, cost and billing groups.
Running from the API
The response says which of the agent’s scorers ran:
skippedScorers appears when a scorer configured on the agent could not be
built — an LLM judge with no judge key, say — and the run went ahead with the
rest.
The X-User-Id header names who ran it, for score provenance. Usage limits
apply before any item starts: an organization or group over its cap gets a
429 and nothing runs.
A/B by variables
variables applies to every item; an item’s own requestContext.variables
is merged over it leaf by leaf. Running the same dataset twice with
different values — two tones, two escalation thresholds — is the cleanest
way to compare prompts, because nothing else changes.
Unlike chat, an experiment refuses to start when a placeholder with no
default has no value for some item, because otherwise it would bake a
literal {{userName}} into every affected item and bill you for the run.
The response reports it:
unused is usually a typo in a key and never blocks. Pass
"strictVariables": false to run anyway. See
Variables in experiments.
Reading results
The Studio’s results panel shows one row per item — input, output, ground truth, each scorer’s score and reason, cost, latency, and the participants (the tools and sub-agents that ran) — with run-level averages, total cost and the models that actually served. Every row has a thread badge that opens the item’s conversation read-only, exactly as the agent produced it, which is where you go when a score surprises you. Export produces a CSV with a column per output field, per scorer score and reason, plus cost, latency, status, trace id and thread id, named after the experiment. From the API,GET /v1/experiments/{experimentId} returns the same thing:
experiment.status moves from pending through running to completed
(or failed) — poll it after an async run. results holds one entry per
item with the input, the agent’s answer, the trace id of that run and the
scores it got, and scores averages every score in the run per scorer:
GET /v1/datasets/{id}/experiments lists every run against a dataset, so two
runs — before and after a prompt change, or with different variables — can
be compared on their scores.