Skip to main content
pipecat eval run tests scenarios against an agent you started yourself. A suite goes one step further: you list agents and scenarios in a manifest, and pipecat eval suite spawns each agent with its eval transport on its own port, runs its scenarios, tears it down, and aggregates the results, several runs at a time. Suites are the right tool when you have more than one agent, more than a handful of scenarios, or want a single command for CI. Pipecat’s own release evals are a manifest with 100+ example agents plus this command.

​
The manifest

manifest.yaml
Paths in the manifest (bots_dir, scenarios_dir, runs_dir, the bot: entries) resolve relative to the manifest file, so a manifest is portable: check it into your repo and run it from anywhere. Scenarios are reusable across agents. One greeting scenario can cover every agent in the suite.
An optional runner_body: points at a JSON file passed to the agent as --runner-body. It supplies session data the agent would normally receive in a /start request body (for example, a vision agent’s image path).

​
Running a suite

In a terminal, a live dashboard shows each run’s status, a running tally, and total time. When piped (in CI, or driven by a coding assistant), it streams one plain result line per run instead. The command exits 0 only if every run passes. Useful flags:
Everything except the suite: list can live in the manifest or be passed on the command line (the command line wins), so a manifest can be as minimal as a suite: list.

​
Run output

Each invocation writes to <runs_dir>/<name>/ (a timestamp when -n is omitted):
results.jsonl carries one line per run — its bot, scenario, attempt, outcome, duration, failures, per-turn results, and paths to its artifacts. Lines are appended as each run finishes, so an interrupted sweep keeps everything already done. Runs that didn’t pass also carry events_seen.
results.jsonl is written by pipecat eval suite. pipecat eval run doesn’t produce one.
When a run fails, start with the .eval.log decision trace: it’s a timestamped record of every event the harness saw, what it matched, what the judge said, and why an assertion failed. The agent’s own log sits next to it.

​
Testing one agent with many scenarios

If you just want to run a batch of scenarios against an agent you already have running, you don’t need a manifest. pipecat eval run accepts multiple scenario files and shares the suite’s dashboard and tally:
A directory expands to its .yaml and .yml files in filename order, non-recursively, and files and directories can be mixed in one invocation. A directory holding no scenarios is an error rather than an empty run. By default the agent is left running afterward so it can serve more evals; pass --stop-bot to shut it down when the batch finishes.

​
Repeating a run

A behavior with a race in it passes sometimes. repeat: in the manifest, or --repeat / -r on the command line, runs every (bot, scenario) pair N times and reports a pass rate per pair instead of a single verdict:
Attempts interleave across bots rather than running back to back, and artifact filenames gain an attempt suffix (__001) only when repeat is above 1, so a single pass keeps the filenames it always had.
A repeated sweep always exits 0. A pass rate isn’t a pass or a fail, so the threshold is yours to choose — read results.jsonl and decide. Leave repeat unset for the CI gate below.

​
Suites in CI

The exit code makes suites CI-ready with no extra glue:
For deterministic, key-free CI runs, prefer text-mode scenarios and an OpenAI-compatible judge endpoint you control. Audio-mode scenarios work in CI too, but need the harness’s TTS and STT services available (local models by default, which also need more CPU).