> ## Documentation Index > Fetch the complete documentation index at: https://sidenet.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Experiments > Run test cases against a real agent, score them with the agent's own scorers, and compare two configurations. An experiment runs every item of a dataset through an agent and scores the answers. It is how you find out whether a prompt change helped before it reaches users, and how you keep an agent from regressing as its tools and blocks move underneath it. Everything on this page is available over the REST API and as tools on the [MCP server](/docs/mcp-server) (`create_dataset`, `add_dataset_items`, `run_experiment`, `get_experiment`, …), so an assistant connected to your organization can build a dataset and run it in one conversation. ## What an experiment runs against The **real agent**: its published or draft instructions, its tools with their real credentials, its model or routing policy. Nothing is simulated. Every tool call an item triggers is a real call — a dataset item that asks the agent to send an email will send one — so point write tools at safe targets or leave them off the version under test. An experiment targets **one agent version**. Pass either the agent's id (its active version runs) or a specific version id, including the draft's. A draft is rebuilt fresh on every run, so the experiment reflects what you just saved. Networks are not a target: to test routing, run a golden set through `POST /v1/copilots/{id}/run` instead — see [Designing a network](/docs/designing-a-network#testing-routing). ## Datasets A dataset is a list of items: | Field | Purpose | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input` | The prompt. Required. A string is one user message; an array of `{ role, content }` messages sets up a multi-turn prompt whose last message is the turn the agent answers. | | `groundTruth` | The expected answer, for scorers that compare against one. | | `requestContext` | Per-item run settings — `variables` overrides, or a `threadId` to chain items. Nothing else is accepted there. | | `metadata` | Anything else you want to carry through to the results. | Add items in the Studio one at a time with the form, or in bulk from a CSV with `input`, `groundTruth`, `requestContext` and `metadata` columns (JSON-valued cells are stored as JSON, everything else as text). A template CSV is downloadable from the same dialog. Or create the dataset from the API, items included: ```bash theme={null} curl https://api.sidenet.ai/v1/datasets \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Refund questions", "agentId": "AGENT_ID", "items": [ { "input": "Can I get a refund on an order I placed yesterday?", "groundTruth": "Yes — orders can be refunded within 14 days. Ask for the order number." }, { "input": [ { "role": "user", "content": "I placed an order yesterday" }, { "role": "assistant", "content": "Got it — what would you like to do with it?" }, { "role": "user", "content": "Cancel it and refund me" } ], "groundTruth": "Confirms the cancellation window and asks for the order number." } ] }' ``` `agentId` only records which agent the dataset was written for; the experiment names its agent explicitly. Add more items later with `POST /v1/datasets/{id}/items`, edit or remove one with `PATCH` / `DELETE` on `/v1/datasets/{id}/items/{itemId}`. Every change bumps the dataset's `version`, and an experiment can pin one, so a run stays reproducible while the dataset keeps growing — `GET /v1/datasets/{id}/items?version=N` shows exactly what a pinned run saw. ## Isolation All items of a run write into **one run-scoped thread** whose memory is write-only: nothing is recalled between items, so an item can never see another item's answer, and items run concurrently (five at a time by default). The thread exists so that every answer is replayable afterwards. To test a multi-turn flow deliberately, give consecutive items the same `requestContext.threadId`. Those items share memory in order; run with `maxConcurrency: 1` so they do. ## Scorers An experiment is scored by **the scorers attached to the agent** — the same scorer types, judge models and options that score its live traffic — applied to every item rather than sampled. There is nothing to choose at run time: to score differently, change the scorers on the agent (in the Studio, or with `PATCH /v1/agents/{id}/draft`) and run again. An agent with no scorers still runs and records its answers, which is enough to read them side by side. The catalogue an agent can pick from has two kinds: * **Code scorers** run locally and need no model: `keyword-coverage`, `completeness`, `content-similarity`, `textual-difference`, `tone`. * **LLM judges** ask a model to grade the answer: `answer-relevancy`, `answer-similarity`, `faithfulness`, `hallucination`, `bias`, `toxicity`, `prompt-alignment`, `noise-sensitivity`, `llm-tool-call-accuracy`. Some scorers require `groundTruth` on the item; the catalogue says which. Judge spend is billed to a dedicated **LLM Judges** group with its own monthly cap, so evaluation cost never lands on a customer's bill or drains a customer's limit. See [Traces, cost and billing groups](/docs/observability#billing-groups). ## Running from the API ```bash theme={null} curl -X POST https://api.sidenet.ai/v1/experiments \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "X-User-Id: you@example.com" \ -H "Content-Type: application/json" \ -d '{ "agentId": "AGENT_VERSION_ID", "datasetId": "DATASET_ID", "async": true, "variables": { "userName": "Maya", "account": { "tier": "pro" } } }' ``` | Field | Notes | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `agentId` | Agent id (active version) or agent version id (any version, including the draft). | | `datasetId`, `version` | The dataset and, optionally, a dataset version. | | `async` | `true` returns `{ status: "started", experimentId }` immediately; `false` (default) waits and returns the full `summary`. | | `maxConcurrency`, `itemTimeout`, `maxRetries` | Run tuning. Defaults: 5, per-item timeout, 2 retries. | | `variables`, `strictVariables` | Prompt variables for every item; see below. | The response says which of the agent's scorers ran: ```json theme={null} { "status": "pending", "experimentId": "exp_7f31a9c2", "totalItems": 12, "scorers": [ { "name": "tone", "scorerType": "tone", "model": null }, { "name": "faithfulness", "scorerType": "faithfulness", "model": "openai/gpt-4o-mini" } ], "variables": { "supplied": ["userName", "account.tier"], "required": ["userName"], "missing": [], "dropped": [], "unused": ["account.tier"] } } ``` `skippedScorers` appears when a scorer configured on the agent could not be built — an LLM judge with no judge key, say — and the run went ahead with the rest. The `X-User-Id` header names who ran it, for score provenance. Usage limits apply before any item starts: an organization or group over its cap gets a `429` and nothing runs. ## A/B by variables `variables` applies to every item; an item's own `requestContext.variables` is merged over it leaf by leaf. Running the same dataset twice with different values — two tones, two escalation thresholds — is the cleanest way to compare prompts, because nothing else changes. Unlike chat, an experiment **refuses to start** when a placeholder with no default has no value for some item, because otherwise it would bake a literal `{{userName}}` into every affected item and bill you for the run. The response reports it: ```json theme={null} { "variables": { "supplied": ["account.tier"], "required": ["userName"], "missing": ["userName"], "dropped": [], "unused": [], "itemsAffected": 2 } } ``` `unused` is usually a typo in a key and never blocks. Pass `"strictVariables": false` to run anyway. See [Variables in experiments](/docs/prompt-blocks#variables-in-experiments). ## Reading results The Studio's results panel shows one row per item — input, output, ground truth, each scorer's score and reason, cost, latency, and the participants (the tools and sub-agents that ran) — with run-level averages, total cost and the models that actually served. Every row has a thread badge that opens the item's conversation **read-only**, exactly as the agent produced it, which is where you go when a score surprises you. **Export** produces a CSV with a column per output field, per scorer score and reason, plus cost, latency, status, trace id and thread id, named after the experiment. From the API, `GET /v1/experiments/{experimentId}` returns the same thing: ```bash theme={null} curl https://api.sidenet.ai/v1/experiments/exp_7f31a9c2 \ -H "Authorization: Bearer YOUR_API_KEY" ``` `experiment.status` moves from `pending` through `running` to `completed` (or `failed`) — poll it after an async run. `results` holds one entry per item with the input, the agent's answer, the trace id of that run and the scores it got, and `scores` averages every score in the run per scorer: ```json theme={null} { "experiment": { "id": "exp_7f31a9c2", "status": "completed", "totalItems": 12, "succeededCount": 12 }, "results": [ { "itemId": "dsi_2c7e91b4", "input": "Can I get a refund on an order I placed yesterday?", "output": { "text": "Yes — you can. Orders can be refunded within 14 days; what is the order number?" }, "traceId": "7a1c3e5d9b2f4a6c8e0d1b3f5a7c9e2d", "scores": [{ "scorerId": "tone-scorer", "score": 0.92, "reason": null }] } ], "scores": { "tone-scorer": { "mean": 0.88, "count": 12 } } } ``` `GET /v1/datasets/{id}/experiments` lists every run against a dataset, so two runs — before and after a prompt change, or with different `variables` — can be compared on their `scores`. ## Scoring live traffic The same scorers apply to real conversations after the fact. Select traces in the Studio and choose **Score traces**, or call the endpoint: ```bash theme={null} curl -X POST https://api.sidenet.ai/v1/score-trace \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "scorerType": "hallucination", "targets": [{ "traceId": "TRACE_ID" }] }' ``` Scoring runs in the background and the scores appear on the trace. Combined with a filter on a group or an agent, this is how you audit a week of production answers without building a dataset first — and the traces that score badly are the ones worth turning into dataset items.