BenchmarkΒΆ

Kernel-level benchmark. Each run executes the same set of cases for every backend listed in the YAML config and writes a CSV, a Markdown report and a log. Results only hold for the stated hardware, software versions, config and measurement method; public performance claims must also pass the correctness check.

Install HALOps first (Build Guide), then the benchmark dependencies. Reference backends (FlashAttention, FlashInfer, DeepGEMM, SGLang, cuDNN) are not installed by this file; install only what the YAML backends list uses; the SGLang backend reads its local source tree from SGLANG_PATH. Run all commands from benchmark/.

python -m pip install -r requirements.txt

# All backends listed in the YAML
python benchmark.py --config=attention/prefill_chunked_c16_gqa14-1.yaml --gpu=0

# A single backend (must be in the YAML `backends` list; one --config only)
python benchmark.py --config=gemm/dense_w8a8.yaml --backend=halops --gpu=0

  • --config takes a path relative to configs/ (recommended), a file name unique across configs/<op>/, or an absolute path. Bundled configs live in configs/attention/ and configs/gemm/.

  • Common options: --gpu (default 0, sets CUDA_VISIBLE_DEVICES), --timing-method (profiler or cupti, default profiler), --dry-run-iters (default 10), --repeat-iters (default 50), --refcheck-reference. See python benchmark.py --help.

  • Outputs share one timestamp {ts}: results/{ts}_{config}.csv, reports/{ts}_{scenario}_{case_tag}.md, logs/{ts}_{op}_{config}.log.

  • Speedup is other_backend_median_us / halops_median_us; above 1.00x means HALOps is faster.