Running vLLM (Docker)

WIP: See Verification status.

Image matrix

Base images — blivioniag/rocm-rdna

These are a general-purpose ROCm PyTorch base for RDNA — useful on their own if you just want a working torch on a Radeon card.

vLLM images — blivioniag/vllm-rdna

The -extras tags use the rdna_extras fork lineage (historical bake still clones blivioniag/vllm rdna2_extras), which adds hand-written RDNA2 HIP kernels (FlashAttention, quantized GEMM, MoE, GDN, …). Check Docker Hub for the current tag list. Tags are refreshed in place — docker pull before debugging.

Multi-GPU: pick a 7.2.0 or 7.14.0 tag, not a host ROCm in the 7.2.1–7.13 gap. RCCL on those in-between releases is reported broken with more than one card — see Installing ROCm.

Every image bakes these PYTORCH_ROCM_ARCH targets: gfx1030;gfx1100;gfx1101;gfx1150;gfx1151;gfx1200;gfx1201.

Run it

docker run -it --rm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  docker.io/blivioniag/vllm-rdna:v0.27.1-extras \
  vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
    --dtype float16 \
    --max-model-len 8192 \
    --language-model-only --skip-mm-profiling --trust-remote-code

• Give the container the GPU with --device /dev/kfd --device /dev/dri and the video and render groups. Missing render is a common #vllm-rdna cause of Failed to infer device type.
• Prefer --dtype float16. RDNA2 has weak/emulated BF16; letting vLLM pick bf16 from a model’s config.json can trigger slow float32 fallbacks.
• The 27B AWQ example is the current #vllm-rdna day-to-day dense pick (Sep 2026). For a smaller smoke test, swap in any instruct model that fits VRAM. Model / card-count matrix: Recipes.
• For a non-Navi-21 RDNA2 card (gfx1031/1032/…), add -e HSA_OVERRIDE_GFX_VERSION=10.3.0. See HSA_OVERRIDE.
• Multi-GPU: add --tensor-parallel-size N; enabling PCIe P2P helps a lot here.

Query the OpenAI-compatible endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"cyankiwi/Qwen3.8-27B-AWQ-INT4","messages":[{"role":"user","content":"Hi from gfx1030!"}]}'

Next steps

• Choose Hub vs recipe container vs Flash-Next: Recipes
• Tune env vars and CUDA graphs: Configuration
• Pick a quant format: Quantization
• Kernel details: rdna_extras fork