Skip to content

Supported Models

FastDeploy currently supports the following models, which can be downloaded automatically during FastDeploy deployment.Specify the model parameter as the model name in the table below to automatically download model weights (all supports resumable downloads). The following three download sources are supported:

When using automatic download, the default download source is AIStudio. Users can modify the default download source by setting the FD_MODEL_SOURCE environment variable, which can be set to “AISTUDIO”, ‘MODELSCOPE’ or “HUGGINGFACE”. The default download path is ~/ (i.e., the user's home directory). Users can modify the default download path by setting the FD_MODEL_CACHE environment variable, e.g.:

export FD_MODEL_SOURCE=AISTUDIO # "AISTUDIO", "MODELSCOPE" or "HUGGINGFACE"
export FD_MODEL_CACHE=/ssd1/download_models

⭐ Note: Models marked with an asterisk can directly use HuggingFace Torch weights and support FP8/WINT8/WINT4 as well as BF16. When running inference, you need to enable --load_choices "default_v1".

Example launch Command using baidu/ERNIE-4.5-21B-A3B-PT:

python -m fastdeploy.entrypoints.openai.api_server \
       --model baidu/ERNIE-4.5-0.3B-PT \
       --port 8180 \
       --metrics-port 8181 \
       --engine-worker-queue-port 8182 \
       --max-model-len 32768 \
       --max-num-seqs 32 \
       --load_choices "default_v1"

Large Language Models

These models accept text input.

Models DataType Example HF Model
⭐ERNIE BF16\WINT4\WINT8\W4A8C8\WINT2\FP8 baidu/ERNIE-4.5-VL-424B-A47B-Paddle;
baidu/ERNIE-4.5-300B-A47B-Paddle
 quick start   best practice;
baidu/ERNIE-4.5-300B-A47B-2Bits-Paddle;
baidu/ERNIE-4.5-300B-A47B-W4A8C8-TP4-Paddle;
baidu/ERNIE-4.5-300B-A47B-FP8-Paddle;
baidu/ERNIE-4.5-300B-A47B-Base-Paddle;
baidu/ERNIE-4.5-21B-A3B-Paddle;
baidu/ERNIE-4.5-21B-A3B-Base-Paddle;
baidu/ERNIE-4.5-21B-A3B-Thinking;
baidu/ERNIE-4.5-0.3B-Paddle
 quick start   best practice;
baidu/ERNIE-4.5-0.3B-Base-Paddle, etc.
⭐QWEN3-MOE BF16/WINT4/WINT8/FP8 Qwen/Qwen3-235B-A22B;
Qwen/Qwen3-30B-A3B, etc.
⭐QWEN3 BF16/WINT8/FP8 Qwen/qwen3-32B;
Qwen/qwen3-14B;
Qwen/qwen3-8B;
Qwen/qwen3-4B;
Qwen/qwen3-1.7B;
Qwen/qwen3-0.6B, etc.
⭐QWEN2.5 BF16/WINT8/FP8 Qwen/qwen2.5-72B;
Qwen/qwen2.5-32B;
Qwen/qwen2.5-14B;
Qwen/qwen2.5-7B;
Qwen/qwen2.5-3B;
Qwen/qwen2.5-1.5B;
Qwen/qwen2.5-0.5B, etc.
⭐QWEN2 BF16/WINT8/FP8 Qwen/Qwen/qwen2-72B;
Qwen/Qwen/qwen2-7B;
Qwen/qwen2-1.5B;
Qwen/qwen2-0.5B;
Qwen/QwQ-32, etc.
⭐DEEPSEEK BF16/WINT4 unsloth/DeepSeek-V3.1-BF16;
unsloth/DeepSeek-V3-0324-BF16;
unsloth/DeepSeek-R1-BF16, etc.

Multimodal Language Models

These models accept multi-modal inputs (e.g., images and text).

Models DataType Example HF Model
ERNIE-VL BF16/WINT4/WINT8 baidu/ERNIE-4.5-VL-424B-A47B-Paddle
 quick start   best practice ;
baidu/ERNIE-4.5-VL-28B-A3B-Paddle
 quick start   best practice ;
QWEN-VL BF16/WINT4/FP8 Qwen/Qwen2.5-VL-72B-Instruct;
Qwen/Qwen2.5-VL-32B-Instruct;
Qwen/Qwen2.5-VL-7B-Instruct;
Qwen/Qwen2.5-VL-3B-Instruct

More models are being supported. You can submit requests for new model support via Github Issues.