vLLM recipes (pick a path)

WIP: Stack picks and tok/s figures are community-reported (#vllm-rdna, Sep 2026). See Verification status. Prefer this page when you want “which image / which model / how many cards” — not a kernel deep-dive.

Discord keeps asking the same questions: which Docker stack, which Hugging Face weights, and whether 1× V620 is enough. This page consolidates those answers. Details for env vars and CUDA graphs live on Configuration; fork history on vLLM forks.

Which stack?

#vllm-rdna (Sep 2026): the recipe repo is treated as a parts pile (compose/env/patches against pristine vLLM 0.27.1). New Flash-Next work lives on opengfx1030/vllm-rdna rdna_extras (Sep 14–15 cherry-picks); vllm-rdna2-qwen is the older published container. Official kernel development is the same org extras repo — Hub -extras tags may lag HEAD until bake retargets.

Do not mix host ROCm userspace into the recipe container — the image carries its own stack. Mounting host ROCm into it is a common break (recipe TROUBLESHOOTING.md).

Pick by card count

Also see What fits well on V620.

#vllm-rdna (Oct 8), for llama.cpp migrants: --pipeline-parallel-size N is the vLLM analogue of llama.cpp layer split (-sm layer); --tensor-parallel-size N is the analogue of tensor split (-sm row / -ts). vLLM can combine them (example: TP=2 + PP=2 on four cards). Prefer power-of-2 TP. Flash-Next on four cards still wants TP=4, not PP=4 (PP=4 can look great on long prefill then tank decode — table above).

Gemma 4 note

Community reports Gemma 4 ~26B still fails or is unfinished on current gfx1030 vLLM paths (#vllm-rdna, Sep 13 2026). Prefer Qwen / Ornith until someone posts a working recipe.

Model cheat sheet

Small-VRAM experiment: Ornith 9B EXL3 (~6.8 GB) vs AWQ (~9 GB) — experimental, not in published -extras tags yet. See Quantization.

Path A — Hub -extras (recommended default)

docker pull docker.io/blivioniag/vllm-rdna:v0.31.0-extras

docker run -it --rm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  docker.io/blivioniag/vllm-rdna:v0.31.0-extras \
  vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
    --dtype float16 \
    --max-model-len 8192 \
    --language-model-only --skip-mm-profiling --trust-remote-code

Short V620 walkthrough: Onboard on V620.

• Add --tensor-parallel-size N for multi-GPU. Enable PCIe P2P when possible.
• Prefer --dtype float16. Mount Triton / torch-compile caches for faster restarts — Configuration.
• Force the native W4A16 path when needed: VLLM_DISABLED_KERNELS=ExllamaLinearKernel,TritonW4A16LinearKernel — confirm Using RDNA2W4A16LinearKernel in logs.

Full env block and Compose (GPTQ + MTP): Configuration.

Hub -extras TP4 Qwen3.8-27B AWQ

Sanitized from a #vllm-rdna (Aug 31 2026) bench recipe on 4× V620 with working P2P / custom all-reduce. Drop the custom-AR block if P2P is broken on your board — use VLLM_DISABLE_CUSTOM_ALL_REDUCE=1 instead (Configuration).

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export VLLM_RDNA_FORCE_FP16=1
export VLLM_USE_RDNA2_FA=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_ENABLED=1
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_BATCH_INVARIANT=0
export GPU_MAX_HW_QUEUES=2

# Only when GPU↔GPU P2P works:
export VLLM_FORCE_CUSTOM_ALL_REDUCE=1
export NCCL_P2P_LEVEL=pix
export RCCL_P2P_NET_DISABLE=1
export RCCL_P2P_BATCH_ENABLE=1
export NCCL_PROTO=Simple
export RCCL_MSCCL_ENABLE=0

cd /tmp   # avoid sys.path collisions with a local vllm checkout
vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 20480 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.88 \
  --dtype float16 \
  --language-model-only --skip-mm-profiling --trust-remote-code \
  --enable-prefix-caching --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Host-venv TP=2 Qwen3.8-27B AWQ (community, Sep 26)

#vllm-rdna (Sep 26): one host cloned opengfx1030/vllm-rdna and built from source (not a Hub tag), then served cyankiwi AWQ-INT4 at TP=2. Same short test on both models; answers coherent, no NaNs. Community / Needs verify.

Day-to-day 27B on 4 cards still uses the TP4 block or a recipe preset. This table is only the two-card / source-build snapshot.

#vllm-rdna (Oct 6): a different 2× host ran unquantized fp16 Qwen3.8-27B at TP=2 with MTP and reported ~40–48 t/s single-stream. A shorter-reasoning Swift 1.5 27B (not the Flash-Next Swift packs) on the same card count was ~48 t/s single-stream and ~70 t/s across two instances. Community / Needs verify. Concurrency notes for wvSplitK vs stacked LLMM1: troubleshooting.

Path B — Recipe container (27B / 122B presets)

Fastest path when you want the recipe book’s measured knobs without installing ROCm on the host:

IMG=ghcr.io/leapdragon/vllm-rdna2-recipe:0.27.1-rocm7.2.3-gfx1030
docker pull "$IMG"
docker run --rm "$IMG" list-presets

docker run -d --name vllm-rdna2 --network=host \
  --device /dev/kfd --device /dev/dri \
  --group-add "$(getent group render | cut -d: -f3)" \
  --group-add "$(getent group video | cut -d: -f3)" \
  --ipc=host --ulimit memlock=-1 --security-opt seccomp=unconfined \
  -e ROCR_VISIBLE_DEVICES=0,1 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMG" preset:qwen38-27b-gptq

First boot cold-compiles for ~10–15 minutes; mount /compile-cache and Triton caches for ~3 min warm boots — see the recipe containers/README.md. Community ballpark on 2× V620: ~40–49 tok/s decode when TunableOp rows are seeded; ~27 t/s flat if they are missing (recipe troubleshooting).

122B builds need the repo wrapper / one-time weight prep — not a bare preset: one-liner.

Path C — Flash-Next

Flash-Next (Qwen3.8, PLE sidecar, 3×/4× V620 serve lines, AutoRound, DeepSeek-V4) lives on Flash-Next. This page stays the picker for Hub -extras and the recipe-container presets.

GLM-5.3-Flash (community, Needs verify)

#vllm-rdna (Sep 30): a community 8× V620 host served public wtdcode/GLM-5.3-Flash-AWQ-W4A16 (AWQ W4A16 of zai-org/GLM-5.3-Flash; routed experts INT4, attention / vision / MTP left BF16) on a local vLLM 0.30 fork. TP=8, MTP-2, P2P off (PLX), 150 W caps, dual-socket Broadwell-EP class. This is not Hub -extras and not rdna_extras HEAD.

Org draft opengfx1030/vllm-rdna#2 is a runtime-unverified Glm5Next load path on a side branch (later/glm53-flash-awq); HIP gates default off. #vllm-rdna (Sep 30): at that time day-to-day extras were still 0.28 and the maintainer paused the 0.30 rebase, inviting a PR of the community 0.30 work. #announcements (11 Oct 2026): day-to-day is now Hub v0.31.0-extras — re-check GLM on that line before treating the Sep 30 pause as current. V2 model runner can work on the local-fork line.

Community snapshot (one host — Needs verify):

gfx1030-relevant lessons (do not treat the unpublished kernel list as a Hub recipe):

• Serve --dtype float16, not bf16 — no bf16 hardware. Same quality; community +68% decode on this host.
• Vision: MIOpen compiled the downsample conv per image size (3–4 min; killed the engine once). Community used conv-as-matmul (158 s → 4 ms). Separate from the Flash-Next max_pixels OOM — troubleshooting.
• Indexer / tiled-fp8 / fused mHC / skinny-int4 MoE work lived on the local fork.

Not a drop-in for 4× Flash-Next hosts. #general (25 Oct): 1× V620 + host-RAM offload is also a poor fit. Community Maya snapshot ~200 PP / ~13 TG on one host (Needs verify, not recently rechecked). Public 18B active implies a theoretical decode ceiling around 30 t/s before MTP if experts stayed on-GPU; real offload will not get close.

Related

• Serve scripts — in-tree serve_rdna.sh after a host-venv build
• Running (Docker) — image matrix and minimal docker run
• Flash-Next — Path C serve lines
• Configuration — env vars, CUDA graphs, Compose
• Quantization — GPTQ/AWQ, KV, MTP
• vLLM forks — rdna_extras vs Flash-Next vs recipe ports
• Useful resources