Tutorial: benchmark models on your card

A pipeline that runs is not the same as a pipeline you can ship. Measure FPS, latency, and throughput on your card, then compare model variants and pick with numbers instead of hunches.

Benchmark on Metis

Run a benchmark

Benchmarking in the Voyager SDK is inference.py with performance flags — no separate tool. You fix the frame count, turn off the display, and read the stats. Ask Agent Dev to run the whole comparison for you, or drive it yourself below.

  1. Warm up the environment

    Activate the SDK environment and confirm the card answers. Benchmarks on a card that is throttling or half-detected measure the problem, not the model.

    bash

    source axelera-env/bin/activate
axdevice

  2. Run the first benchmark

    Run a fixed number of frames with the display off and stats on. Use a real video for end-to-end numbers, or the synthetic fakevideo source to take file I/O out of the measurement.

    bash

    ./inference.py yolov8n-coco media/traffic1_1080p.mp4 \
  --no-display --frames 500 --show-stats

./inference.py yolov8n-coco fakevideo:640x480@30 \
  --no-display --frames 1000 --show-stats

  3. Read the numbers

    Four numbers matter: system throughput (end-to-end FPS through the whole pipeline), device throughput (what the AIPU alone delivers), latency per frame, and host CPU utilization. A big gap between system and device FPS means the bottleneck is pre- or post-processing on the host, not the card.

  4. Compare model variants

    Save tracer data to CSV, then run the same benchmark for each candidate model. A + prefix appends to an existing file so one CSV holds the whole comparison.

    bash

    ./inference.py yolov8n-coco video.mp4 --no-display --frames 500 \
  --show-stats --save-tracers bench.csv
./inference.py yolov8s-coco video.mp4 --no-display --frames 500 \
  --show-stats --save-tracers +bench.csv
./inference.py yolov8m-coco video.mp4 --no-display --frames 500 \
  --show-stats --save-tracers +bench.csv

  5. Push multi-stream and accuracy

    Feed several sources into one run to find the card’s stream capacity — and measure accuracy alongside performance by benchmarking against the dataset source, so a faster variant cannot quietly cost you precision.

    bash

    ./inference.py yolov8n-coco usb:0 usb:1 video.mp4 \
  --no-display --show-stats --save-tracers capacity.csv

./inference.py yolov8n-coco dataset \
  --no-display --show-stats --save-tracers acc_perf.csv

Metrics and what they mean

MetricWhat it measuresWatch for
System throughput (FPS)End-to-end frames per second through the whole pipeline.This is the number your application actually gets.
Device throughput (FPS)What the AIPU alone delivers.Much higher than system FPS? The host is the bottleneck.
Latency (ms)Time per frame.The metric that matters for live camera reactions.
CPU utilization (%)Host CPU spent on pre/post-processing.High CPU with idle AIPU means work is in the wrong place.

Let the agent run the comparison

What benchmarking gives you

  • Same tool, fixed conditions: Benchmarks are inference.py runs with --no-display, --frames, and --show-stats — nothing new to install.
  • Two FPS numbers: System FPS and device FPS are different numbers, and the gap between them tells you where the bottleneck lives.
  • Decide with data: CSV tracers plus the dataset source let you compare speed and accuracy in one table before you commit.