Evals & Testing

Write test cases, run automated evaluations, and gate publishing on pass rate.

What evals are

Evals are automated tests for your agent's behavior. You write test cases describing a scenario and the expected outcome. The system simulates each case as a real conversation against your agent's current prompt and model, then an LLM grades the response on a 0–100 scale. A case passes if it scores 70 or above.

The point is to catch regressions — when you edit a prompt or swap a model, you can run evals before publishing to confirm the agent still behaves correctly.

Test suites

Test cases are grouped into suites. A suite is a named collection of related cases (e.g. "Booking Flow Tests" or "Edge Cases"). Suites are shared across your agents — you can run the same suite against multiple agents to compare behavior.

Note · Suites are created at the business level, not per-agent. Any agent can run any suite.

Creating a suite and adding cases

Open an agent → Testing tab.
Click New Suite, give it a name, and choose the channel (Call, Website, Text, Workflow Call).
The new suite appears expanded. Click Add Case to write a case manually.
Fill in the Test Name (a short label), the Scenario (what happens — e.g. "Caller asks for a refund at 11pm"), and the Expected Outcome (what the agent should do — e.g. "Apologize for being closed, offer to call back tomorrow").
Click Add Test Case. Repeat for each scenario you want to cover.

Tip · You can also generate test cases with AI — click Generate with AI inside any suite, describe what edge cases to focus on, and choose how many cases to create. AI generation uses credits; adding manually is free.

Running a suite

Once a suite has at least one test case, click Run. The system sends each scenario to your agent, collects the response, and has an LLM grade it. Results appear inline: pass (green), fail (red), or error (amber).

Each result shows the score (0–100), latency, and a one-line explanation from the grader. The suite header shows the overall pass rate as a colored badge.

Badge color	Pass rate
Green	80% or above
Amber	50 – 79%
Red	Below 50%

Run Evals & Publish

In the agent builder header, click the ⋯ menu → Run Evals & Publish. This runs all test suites linked to the agent and, if the pass rate meets the threshold in Security → Publish Gate (default 70%), publishes the agent atomically.

If the pass rate is too low, the deploy is blocked. You see the exact pass rate and can expand the suite to see which cases failed before trying again.

If the agent has no test suites with cases, it publishes immediately (no gate).

Note · Plain Publish in the header always goes live immediately and skips evals entirely. Use Run Evals & Publish when you want the safety gate.

Customizing the pass rate threshold

The threshold defaults to 70% but is configurable per agent. Go to Security tab → Publish Gate → Min pass rate. Set it to 0 to disable the gate entirely, or raise it to 90–100% for high-stakes production agents.