Automating Domino's local competitor pricing research.Read the case study

Primitives

Benchmarks

Reusable test suites for your agents. Change the instructions, tools, skills, model or runtime, run the benchmark, and see what improved and what regressed before it ships.

v1v2v3Draft

Versioned cases

Fixed cases saved as immutable versions, so a result today means the same as one next month.

  • Output
  • Behavior
  • Files
  • Commands
  • Model judge

Checks for every outcome

Grade the output, the agent's behaviour, the files it changed and the commands it ran, or have a model judge.

Claude Opus 5.592%
GPT-6 Sol86%
Kimi K378%
Qwen3.671%

Compare models side by side

Run up to ten model configurations on the same cases and see pass rate and cost for each.

BuildBenchmarkDeploy

Gate your deploys

Everything is in the API, so a Benchmark can stand between a change and production.

Related primitives