Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

vLLM troubleshooting

CUDA graph capture crashes

Symptom: crash at GDN _output_projection all-reduce / _SimpleCData.__new__, or OOM during graph capture.

Root cause (fixed in current extras): TP comm wrappers were not allow_in_graph. Fork fix: fix(distributed): allow TP comm ops in torch.compile graph capture.

First, pull the latest image:

docker pull blivioniag/vllm-rdna:v0.27.1-extras-rocm7.14.0

Then try CUDA graphs (fast path):

--compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Mount cache volumes — see Configuration.

Fallback: --enforce-eager

On multi-GPU AOT cache issues: VLLM_USE_AOT_COMPILE=0 VLLM_DISABLE_COMPILE_CACHE=1

Low throughput (~4–5 t/s on 27B)? Check for older image where AWQ still used Triton — see Quantization.

First boot is extremely slow

Triton and torch.compile JIT on first run. Typical cache: ~/.triton/cache (~3 GB), ~/.cache/vllm/torch_compile_cache (~700 MB).

GPTQ/AWQ not using RDNA2 kernels

Check logs for Using RDNA2W4A16LinearKernel. If you see Triton/Exllama instead:

export VLLM_DISABLED_KERNELS=ExllamaLinearKernel,TritonW4A16LinearKernel

Confirm -extras image from current extras. See Fork kernel dispatch.

vLLM picks the wrong platform / doesn’t see my Radeon

Use published blivioniag/vllm-rdna images with patches/*rocm-platform* fixes rather than stock upstream builds.

Failed to infer device type / AMDSMI_STATUS_NOT_INIT

Symptom: vLLM logs ROCm platform is not available because no GPU is found and AMDSMI_STATUS_NOT_INIT - Device not initialized (often right after CUDA/NVML is also missing — that part is expected on AMD).

Check the Docker device/group block exactly — #vllm-rdna hits this when render is missing:

devices: [/dev/kfd, /dev/dri]
group_add: [video, render]    # both — video alone is not enough on many hosts
ipc: host
security_opt: [label=disable]  # or seccomp=unconfined on docker run

Host user still needs render/video as in General troubleshooting. Debug with VLLM_LOGGING_LEVEL=DEBUG. If the same compose worked on an older tag, docker pull a known-good image — a bad rebuild can also fail AMDSMI init.

ROCM_ATTN hangs for hours (Triton compile)

AMD Triton flash-attention compile on gfx1030 can sit there for hours (RCCL vs Triton). On -extras, use --attention-backend RDNA_ATTN and/or VLLM_USE_RDNA2_FA=1 instead of ROCM_ATTN. See Configuration.

FA_RDNA2 / RDNA_ATTN may not appear in the backend list on older -extras images or some GPTQ models (logs only show Triton / ROCM / TurboQuant). Pull the latest -extras tag and confirm Using RDNA2W4A16LinearKernel / native FA in startup logs. Qwen3.8-27B AWQ needs head size 256 on the fork — see Quantization.

shm_broadcast / one GPU + one CPU pegged

Symptom (#vllm-rdna Sep 2026, often Flash-Next / multi-GPU): serve looks wedged; logs repeat something like:

No available shared memory broadcast block found in 60 seconds.
This typically happens when some processes are hanging or doing some time-consuming work
(e.g. compilation, weight/kv cache quantization).

Community diagnosis: classic Triton JIT compile fighting RCCL — not necessarily a dead process. Guidance:

  1. Wait — first boots can sit like this a long time; keep the Triton / compile cache volumes mounted (Configuration).
  2. Stay on ROCm 7.2.0 or 7.14.x — avoid mid-7.2.x (same pin as multi-GPU RCCL).
  3. Prefer HIP / RDNA_ATTN paths over AMD Triton FA where the fork offers them — less Triton means fewer of these stalls.
  4. Next cold start with a warm cache should be much shorter; if it never recovers after hours, A/B VLLM_USE_V2_MODEL_RUNNER=0 and the long-prompt notes.

Multi-GPU RCCL hangs or cards drop offline

If TP works on one image and dies after a host ROCm bump, check the ROCm version before the model. 7.2.1 through ~7.13 are reported to have a multi-card RCCL bug. Stay on 7.2.0 or 7.14.0 — see Installing ROCm.

Flash-Next long-prompt stalls

Symptom (Flash-Next fork / recipe containers, #vllm-rdna Sep 2026): short prompts decode fine, but large prompts (tens of k tokens — agentic coding, session resume) take many minutes, timeout, or appear wedged. Temps and power caps look healthy.

Community fix that unblocked one 4× V620 host:

export VLLM_USE_V2_MODEL_RUNNER=0

Reporter then saw stable ~68 tok/s with dense INT8 + custom all-reduce, including large prompts. The Flash-Next fork author added this to their docs / troubleshooting. Official-extras authors note separate Dense-on-V2 fixes in progress on the org rebase — A/B both values on your image.

Also rule out thermal / power first (Power tuning), and measure expected prefill time (~1k tok/s class ⇒ ~40 s for 40k tokens, not minutes). Prefill campaign numbers: vLLM overview.

Flash-Next 128k prefill cliff

Symptom (#vllm-rdna Sep 2026, Intel AutoRound / draft opengfx1030/vllm-rdna#5): prefill holds ~950 tok/s through 64k, then falls to ~375 tok/s at 128k while decode stays flat (~48–55 tok/s). Looks like a scheduler / chunking misconfig more than a kernel cliff.

Community fix on that host: raise the batch cap:

--max-num-batched-tokens 4096

The suspected combination was enable_chunked_prefill=True with 2048 scheduled / batched tokens. After 4096, 128k prose/code PP returned to the ~950 tok/s class (same 4× V620, FP16 KV, CPU PLE offload). See overview.

If 128k is still slow after 4096, A/B V2 runner and confirm you are not on quantized PLE (known-good is the embedded BF16 n-gram table).

Prefill blocks decode / MTP stalls under concurrency

Symptom: with speculative decode (MTP) and multiple in-flight requests, generation stalls while prefill runs; or graph + MTP3 reaches “Application startup complete” then hangs on PLE lookup / sample_tokens timeout.

Community notes (#vllm-rdna):

  • Prefer GPTQ + RDNA2 W4A16 (or AWQ HIP) paths over GGUF-in-vLLM for these cards.
  • Concurrent MTP / prefill-vs-decode fixes land in community recipes first — see open PRs on leapdragon/vllm-rdna2-recipe.
  • MTP=0 vs MTP=3 are different bug surfaces; a commit that “works” at MTP=3 can still emit spurious tokens at MTP=0. A/B and pin a known-good recipe commit.
  • Slow or broken P2P + custom all-reduce can look like MTP latency bugs — A/B the disable vs PIX custom AR stacks.

ROCR idle CPU spin (TheRock 7.14)

Symptom: after starting a multi-GPU vLLM serve on TheRock / ROCm 7.14 (ROCR 1.21), the host holds several CPU cores at high utilization even when the GPUs are idle (~one core per HIP process, plus more once RCCL initializes).

Root cause: AsyncEventsLoop / signal-wait paths busy-spin without backoff (ROCm/TheRock#7051, ROCm/ROCm#6522). Stock env knobs (HSA_ENABLE_INTERRUPT, etc.) do not fix multi-GPU cases.

Fix: rebuild only libhsa-runtime64.so with the poll-backoff patch and LD_PRELOAD it. Step-by-step for host builds (and note that recipe containers already bake the patch): docs/rdna2/ROCR-CPU-FIX.md in leapdragon/vllm-rdna2-qwen. Pull latest recipe / container before re-debugging idle CPU.