Evaluation Lab: benchmark your agents

You tuned a prompt — did it actually get better? The Evaluation Lab answers with numbers: build a benchmark, run your agent against it, and compare runs instead of guessing.

Create a benchmark

Build and run a benchmark

1. Open the Evaluation Lab

   Open the Lab by adding /eval to your Wingman Chat address — its sidebar menu entry is still rolling out, so direct navigation is the reliable door today. The page lists your benchmarks — each one a named suite of test cases for measuring an agent on a task that matters to you.

2. Create a benchmark

   Create a benchmark and name it for the behavior it measures — "Voyager support answers", "PR summary quality". A tight scope beats a broad one: one benchmark per behavior keeps results readable.

3. Build or import a dataset

   Open the benchmark detail and add a dataset: build it by hand, case by case, or import existing cases in bulk. Real examples from real conversations make the best test cases — the questions your agent actually gets asked.

4. Edit the test cases

   Each test case pairs an input with what a good answer looks like. Open any case in the editor to refine it — sharpen the input, tighten the expectation. Weak cases produce noisy scores; this editing pass is where benchmark quality is made.

5. Launch runs

   Launch a run against the agent you want to measure. The run executes every case and lands in the benchmark detail with its results. Now change one thing — the prompt, the model, a tool — and run again.

6. Read the results

   Read results in the benchmark detail: per-case outcomes for finding exactly where the agent fails, and run-level results for the trend. Two runs on the same dataset are an honest before-and-after for any change you made.

Run a benchmark and read the results

Start with a baseline