bench: Benchmark Testing

1. bench latency: Offline Latency Test

Parameters

Parameter	Description	Default
--input-len	Input sequence length (tokens)	32
--output-len	Output sequence length (tokens)	128
--batch-size	Batch size	8
--n	Number of sequences generated per prompt	1
--use-beam-search	Whether to use beam search	False
--num-iters-warmup	Number of warmup iterations	10
--num-iters	Number of actual test iterations	30
--profile	Whether to enable performance profiling	False
--output-json	Path to save latency results as a JSON file	None
--disable-detokenize	Whether to disable detokenization	False

Example

# Run latency benchmark on the inference engine
fastdeploy bench latency --model baidu/ERNIE-4.5-0.3B-Paddle

2. bench serve: Online Latency and Throughput Test

Parameters

Parameter	Description	Default
--backend	Backend type	"openai-chat"
--base-url	Base URL of the server or API	None
--host	Host address	"127.0.0.1"
--port	Port	8000
--endpoint	API endpoint path	"/v1/chat/completions"
--model	Model name	Required
--dataset-name	Dataset name	"sharegpt"
--dataset-path	Path to dataset	None
--num-prompts	Number of prompts to process	1000
--request-rate	Requests per second	inf
--max-concurrency	Maximum concurrency	None
--top-p	Sampling top-p (OpenAI backend)	None
--top-k	Sampling top-k (OpenAI backend)	None
--temperature	Sampling temperature (OpenAI backend)	None

Example

# Run online performance test
fastdeploy bench serve --backend openai-chat \
  --model baidu/ERNIE-4.5-0.3B-Paddle \
  --endpoint /v1/chat/completions \
  --host 0.0.0.0 \
  --port 8891 \
  --dataset-name EBChat \
  --dataset-path /datasets/filtered_sharedgpt_2000_input_1136_output_200.json \
  --percentile-metrics ttft,tpot,itl,e2el,s_ttft,s_itl,s_e2el,s_decode,input_len,s_input_len,output_len \
  --metric-percentiles 80,95,99,99.9,99.95,99.99 \
  --num-prompts 1 \
  --max-concurrency 1 \
  --save-result

3. bench throughput: Throughput Test

Parameters

Parameter	Description	Default
--backend	Inference backend	"fastdeploy"
--dataset-name	Dataset name	"random"
--model	Model name	Required
--input-len	Input sequence length	None
--output-len	Output sequence length	None
--prefix-len	Prefix length	0
--n	Number of sequences generated per prompt	1
--num-prompts	Number of prompts	50
--output-json	Path to save results as a JSON file	None
--disable-detokenize	Whether to disable detokenization	False
--lora-path	Path to LoRA adapter	None

Example

# Run throughput benchmark on the inference engine
fastdeploy bench throughput --model baidu/ERNIE-4.5-0.3B-Paddle \
--backend fastdeploy-chat \
--dataset-name EBChat \
--dataset-path /datasets/filtered_sharedgpt_2000_input_1136_output_200.json \
--max-model-len 32768

4. bench eval: Online Task Evaluation

Parameters

Parameter	Description	Default
--model, -m	Model name	"hf"
--tasks, -t	List of evaluation tasks	None
--model_args, -a	Model arguments	""
--num_fewshot, -f	Number of few-shot examples	None
--samples, -E	Number of samples	None
--batch_size, -b	Batch size	1
--device	Device	None
--output_path, -o	Output file path	None
--write_out, -w	Whether to write output results	False

Example

# Run task evaluation on an online service
fastdeploy bench eval --model local-completions \
  --model_args pretrained=./baidu/ERNIE-4.5-0.3B-Paddle,base_url=http://0.0.0.0:8490/v1/completions
  --write_out \
  --tasks ceval-valid_accountant