Skip to main content
An evaluation is a judge reading a conversation and saying whether the agent did the thing you wanted. Three objects do the work. A judge scores a conversation. A test case is a simulated caller that produces conversations to score. A run pairs them and gives you a result. A test case carries no judges of its own; you pick both when you start a run.

​
Prerequisites

  • An agent you have built, from Agent builder
  • An owner, admin, operator or prompter role to make an evaluation a required check

​
Where evaluations live

Inside an agent, Evaluations is under Validate, with three tabs: Judges, Test cases and Runs.
Searching for evals? That is this page. The word is still in the URL, which reads /judges rather than /evaluations.
At All agents there is a shorter version of the same page: Judges only. Test cases and runs belong to an agent, so those two tabs do not exist at organization scope.

​
Write a judge

New judge opens a form with five sections. Only Judge task is required, and it is the one that does the work: it is the question you are asking about the conversation. The button reads Create judge the first time and Update afterwards. There is no Save.
Save as draft keeps your work without making the judge live. A draft is not used by any run, so a judge you drafted and forgot will not quietly start scoring things.

​
Write a test case

New test case starts either from scratch or from your own traffic. Generate based on production calls reads up to 50 real calls and writes one test case per call, which is the faster way to get a suite that resembles what actually happens to you. Then set a Title, a Prompt describing who the simulated caller is and what they want, and optionally Labels, Max turns (20 by default), a Start node and Request data. Save finishes it.

​
Run them

A run pairs judges with test cases. Pick both, choose how many conversations each test case should produce, and start it.
A run you start here is stored in your browser, not on Bland.Your teammates cannot see it, and neither can you from another machine or another browser. Only the runs that happen as part of a publish are kept on the server.
The version picker in the new run dialog does not change which version runs.Every conversation runs the agent’s current version. The picker records your choice on the run for reference, and the tooltip says so.

​
Read a run

A conversation passes only if every judge passed it. A judge that could not tell counts against it here. This is worth holding onto, because the promotion gate scores the same run differently: there, each judge is scored on its own, across conversations. So a run can show failed conversations in the results and still clear the gate.

​
Make it a required check

An evaluation becomes a gate from the agent’s Environments tab rather than from this page. The Evaluations card sits between Development and Staging, and Add evaluations opens the picker. Attach the judges you want, up to five test cases, and choose 1, 3 or 5 conversations per test case.
Attaching a judge makes it required by default. Clear Req’d on the row to make it advisory instead.An advisory judge reports its verdict and blocks nothing.
Only staging has a configuration, and it gates both hops: the publish into staging and the promotion out of it. See Environments for what each of those does when it is blocked.

​
How a judge passes

A judge does not pass or fail per conversation. It takes the share of conversations that matched the verdict you were hoping for, and passes when that share is half or more. Conversations where the judge could not reach a verdict are left out of the share entirely, rather than counted against it.
You cannot change the half. It is fixed, so there is no target-rate setting to look for.What you choose is how many conversations each test case produces, and the options are odd numbers for a reason: with 3 or 5, a tie cannot happen.

​
Simulation

Simulation, in the builder under Testing, runs one conversation against a test case so you can watch the agent handle it. It is the same machinery, used for looking rather than scoring. Those conversations are not kept and never appear in Runs.

​
Common failures

Manual runs are stored in the browser that started them. A different machine, a different browser, or cleared site data all lose them. Runs attached to a publish are on the server and survive.
The two screens score differently. Results fail a conversation if any judge failed it; the gate scores each judge across conversations and passes at half or more. Both numbers are right, and they answer different questions.
It is still a draft, or it is not attached. Save as draft keeps a judge out of every run, and attaching happens on the Environments tab rather than when you write it.
Req’d is cleared on that row, so the judge is advisory. Advisory judges report and never block.
A required evaluation failed. The button dims rather than disabling, and the reason is on hover.
It ran the current one. The picker records a version on the run and does not select what executes.

​
Next steps

Environments

Where an evaluation becomes a condition of shipping.

Agent builder

Change the agent the judges keep failing.

Conversations

Read the real calls a test case was generated from.

Testbed

Try an agent by hand before you score it.
Docs for agents: llms.txt