vLLM recipes (pick a path)

WIP: Stack picks and tok/s figures are community-reported (#vllm-rdna, Sep 2026). See Verification status. Prefer this page when you want “which image / which model / how many cards” — not a kernel deep-dive.

Discord keeps asking the same questions: which Docker stack, which Hugging Face weights, and whether 1× V620 is enough. This page consolidates those answers. Details for env vars and CUDA graphs live on Configuration; fork history on vLLM forks.

Which stack?

#vllm-rdna (Sep 2026): the recipe repo is treated as a parts pile (compose/env/patches against pristine vLLM 0.27.1). New Flash-Next work lives in vllm-rdna2-qwen. Official kernel development is opengfx1030/vllm-rdna rdna_extras — Hub -extras tags may lag HEAD until bake retargets.

Do not mix host ROCm userspace into the recipe container — the image carries its own stack. Mounting host ROCm into it is a common break (recipe TROUBLESHOOTING.md).

Pick by card count

Also see What fits well on V620.

Gemma 4 note

Community reports Gemma 4 ~26B still fails or is unfinished on current gfx1030 vLLM paths (#vllm-rdna, Sep 13 2026). Prefer Qwen / Ornith until someone posts a working recipe.

Model cheat sheet

Small-VRAM experiment: Ornith 9B EXL3 (~6.8 GB) vs AWQ (~9 GB) — experimental, not in published -extras tags yet. See Quantization.

Path A — Hub -extras (recommended default)

docker pull docker.io/blivioniag/vllm-rdna:v0.27.1-extras

docker run -it --rm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  docker.io/blivioniag/vllm-rdna:v0.27.1-extras \
  vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
    --dtype float16 \
    --max-model-len 8192 \
    --language-model-only --skip-mm-profiling --trust-remote-code

• Add --tensor-parallel-size N for multi-GPU. Enable PCIe P2P when possible.
• Prefer --dtype float16. Mount Triton / torch-compile caches for faster restarts — Configuration.
• Force the native W4A16 path when needed: VLLM_DISABLED_KERNELS=ExllamaLinearKernel,TritonW4A16LinearKernel — confirm Using RDNA2W4A16LinearKernel in logs.

Full env block and Compose (GPTQ + MTP): Configuration.

Hub -extras TP4 Qwen3.8-27B AWQ

Sanitized from a #vllm-rdna (Aug 31 2026) bench recipe on 4× V620 with working P2P / custom all-reduce. Drop the custom-AR block if P2P is broken on your board — use VLLM_DISABLE_CUSTOM_ALL_REDUCE=1 instead (Configuration).

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export VLLM_RDNA_FORCE_FP16=1
export VLLM_USE_RDNA2_FA=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_ENABLED=1
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_BATCH_INVARIANT=0
export GPU_MAX_HW_QUEUES=2

# Only when GPU↔GPU P2P works:
export VLLM_FORCE_CUSTOM_ALL_REDUCE=1
export NCCL_P2P_LEVEL=pix
export RCCL_P2P_NET_DISABLE=1
export RCCL_P2P_BATCH_ENABLE=1
export NCCL_PROTO=Simple
export RCCL_MSCCL_ENABLE=0

cd /tmp   # avoid sys.path collisions with a local vllm checkout
vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 20480 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.88 \
  --dtype float16 \
  --language-model-only --skip-mm-profiling --trust-remote-code \
  --enable-prefix-caching --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Path B — Recipe container (27B / 122B presets)

Fastest path when you want the recipe book’s measured knobs without installing ROCm on the host:

IMG=ghcr.io/leapdragon/vllm-rdna2-recipe:0.27.1-rocm7.2.3-gfx1030
docker pull "$IMG"
docker run --rm "$IMG" list-presets

docker run -d --name vllm-rdna2 --network=host \
  --device /dev/kfd --device /dev/dri \
  --group-add "$(getent group render | cut -d: -f3)" \
  --group-add "$(getent group video | cut -d: -f3)" \
  --ipc=host --ulimit memlock=-1 --security-opt seccomp=unconfined \
  -e ROCR_VISIBLE_DEVICES=0,1 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMG" preset:qwen38-27b-gptq

First boot cold-compiles for ~10–15 minutes; mount /compile-cache and Triton caches for ~3 min warm boots — see the recipe containers/README.md. Community ballpark on 2× V620: ~40–49 tok/s decode when TunableOp rows are seeded; ~27 t/s flat if they are missing (recipe troubleshooting).

122B builds need the repo wrapper / one-time weight prep — not a bare preset: one-liner.

Path C — Flash-Next (vllm-rdna2-qwen)

For agentic / long-context Flash-Next on 4× V620, use the dedicated fork and its docs — not Hub -extras and not the 27B recipe presets:

• Repo: leapdragon/vllm-rdna2-qwen
• Docs: docs/rdna2/
• Weights: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16
• PLE sidecar: primitive-ai/Qwen3.8-Flash-Next-PLE-quant

Expect large host DRAM for the n-gram / PLE store (~64 GB class called out in-channel). Long-prompt stalls: try VLLM_USE_V2_MODEL_RUNNER=0 — troubleshooting. Throughput snapshot: overview.

Related

• Running (Docker) — image matrix and minimal docker run
• Configuration — env vars, CUDA graphs, Compose
• Quantization — GPTQ/AWQ, KV, MTP
• vLLM forks — rdna_extras vs Flash-Next vs recipe ports
• Useful resources — external links