vLLM troubleshooting

CUDA graph capture crashes

Symptom: crash at GDN _output_projection all-reduce / _SimpleCData.__new__, or OOM during graph capture.

Root cause (fixed in current extras): TP comm wrappers were not allow_in_graph. Fork fix: fix(distributed): allow TP comm ops in torch.compile graph capture.

First, pull the latest image:

docker pull blivioniag/vllm-rdna:v0.31.0-extras

Then try CUDA graphs (fast path):

--compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Mount cache volumes — see Configuration.

Fallback: --enforce-eager

On multi-GPU AOT cache issues: VLLM_USE_AOT_COMPILE=0 VLLM_DISABLE_COMPILE_CACHE=1

TORCHINDUCTOR_CACHE_DIR survives VLLM_DISABLE_COMPILE_CACHE

rdna_extras (1 Oct, EXL3 27B mul1 TP=4): piecewise assert_size_stride kept failing after a fake-kernel stride fix because Inductor reused a stale compiled graph. VLLM_DISABLE_COMPILE_CACHE does not cover TORCHINDUCTOR_CACHE_DIR — clear that directory (or unset and delete the default cache) after kernel / fake-tensor changes, then recapture. See Quantization.

Low throughput (~4–5 t/s on 27B)? Check for older image where AWQ still used Triton — see Quantization.

GLM-5.3 vision: MIOpen compiles every image size

#vllm-rdna (Sep 30), community 8× GLM-5.3-Flash AWQ on a local 0.30 fork (not Hub): MIOpen compiled the vision downsample conv per image size (3–4 min; one report that it killed the engine). Community workaround: implement that conv as a matmul (158 s → 4 ms). This is not the Flash-Next max_pixels startup OOM (Flash-Next vision). Needs verify on org extras — recipes.

First boot is extremely slow

Triton and torch.compile JIT on first run. Typical cache: ~/.triton/cache (~3 GB), ~/.cache/vllm/torch_compile_cache (~700 MB).

GPTQ/AWQ not using RDNA2 kernels

Check logs for Using RDNA2W4A16LinearKernel. If you see Triton/Exllama instead:

export VLLM_DISABLED_KERNELS=ExllamaLinearKernel,TritonW4A16LinearKernel

Confirm -extras image from current extras. See Fork kernel dispatch.

TunableOp aborts or is ~20% slow after a ROCm / rocBLAS bump

#vllm-rdna (Sep 23–24) + in-tree tunableop/: solver IDs are locked to one librocblas.so.5 build. Reusing them across a different rocBLAS, even when the version string matches, is unsafe.

Current registry (profiles.json on rdna_extras):

tools/rdna/serve_rdna.sh auto-selects the profile whose hash matches the loaded library. A ROCm 10 venv on rocBLAS 5.6 (c27e2252cc7a) already picks rocm10-rocblas5.6 — no launcher change. Untuned shapes stay on the heuristic until that host runs a capture campaign and registers the new rows. TUNABLEOP_PROFILE=<name> forces a registered name and aborts on mismatch. TUNABLEOP_ALLOW_MISMATCH=1 is only for experiments. An unmatched sha uses rocblas-<sha12>/ or $HOME/.cache/tunableop/, not the other profile. Details: Serve scripts.

Symptoms when the table does not match the library:

• First GEMM aborts / TunableOp turns itself off.
• Prefill sits ~25% below the #17 published table (community: ~1496 vs ~1958 tok/s at 16k) until a matching table is loaded. ROCm 10 vs 7.14 was not the gap on that host.

Fix: stop the serve, then regenerate and qualify for this library — see Regenerate for another rocBLAS build. Lookup stays on and tuning stays off (PYTORCH_TUNABLEOP_ENABLED=1, PYTORCH_TUNABLEOP_TUNING=0). The helper refuses a mismatched named profile instead of loading it. A legacy tunableop/rocblas-<hash>/ folder is still honoured when no registered profile matches.

Do not commit a host-specific CSV into the wiki. Do not copy another machine’s library hash.

vLLM picks the wrong platform / doesn’t see my Radeon

Use published blivioniag/vllm-rdna images with patches/*rocm-platform* fixes rather than stock upstream builds.

Failed to infer device type / AMDSMI_STATUS_NOT_INIT

Symptom: vLLM logs ROCm platform is not available because no GPU is found and AMDSMI_STATUS_NOT_INIT - Device not initialized (often right after CUDA/NVML is also missing — that part is expected on AMD).

Check the Docker device/group block exactly — #vllm-rdna hits this when render is missing:

devices: [/dev/kfd, /dev/dri]
group_add: [video, render]    # both — video alone is not enough on many hosts
ipc: host
security_opt: [label=disable]  # or seccomp=unconfined on docker run

Host user still needs render/video as in General troubleshooting. Debug with VLLM_LOGGING_LEVEL=DEBUG. If the same compose worked on an older tag, docker pull a known-good image — a bad rebuild can also fail AMDSMI init.

ROCM_ATTN hangs for hours (Triton compile)

AMD Triton flash-attention compile on gfx1030 can sit there for hours (RCCL vs Triton). On -extras, use --attention-backend RDNA_ATTN and/or VLLM_USE_RDNA2_FA=1 instead of ROCM_ATTN. See Configuration.

FA_RDNA2 / RDNA_ATTN may not appear in the backend list on older -extras images or some GPTQ models (logs only show Triton / ROCM / TurboQuant). Pull the latest -extras tag and confirm Using RDNA2W4A16LinearKernel / native FA in startup logs. Qwen3.8-27B AWQ needs head size 256 on the fork — see Quantization.

shm_broadcast / one GPU + one CPU pegged

Symptom (#vllm-rdna Sep 2026, often Flash-Next / multi-GPU): serve looks wedged; logs repeat something like:

No available shared memory broadcast block found in 60 seconds.
This typically happens when some processes are hanging or doing some time-consuming work
(e.g. compilation, weight/kv cache quantization).

Community diagnosis: classic Triton JIT compile fighting RCCL — not necessarily a dead process. Guidance:

1. Wait — first boots can sit like this a long time; keep the Triton / compile cache volumes mounted (Configuration).
2. Stay on ROCm 7.2.0 or 7.14.x — avoid mid-7.2.x (same pin as multi-GPU RCCL).
3. Prefer HIP / RDNA_ATTN paths over AMD Triton FA where the fork offers them — less Triton means fewer of these stalls.
4. Next cold start with a warm cache should be much shorter; if it never recovers after hours, A/B VLLM_USE_V2_MODEL_RUNNER=0 and the long-prompt notes.

Multi-GPU RCCL hangs or cards drop offline

If TP works on one image and dies after a host ROCm bump, check the ROCm version before the model. 7.2.1 through ~7.13 are reported to have a multi-card RCCL bug. Stay on 7.2.0 or 7.14.0 — see Installing ROCm.

Silent decode hang / RPC call … timed out (rdna_ar)

Symptom (#vllm-rdna / fork #128, 18–19 Oct): all ranks at ~99% GPU util and 0% memory activity, host idle, then RPC call … timed out. No rdna_ar wedged line. Seen on rdna_extras without --monokernel (MonoKernel does those all-reduces inside its FFN kernel).

Merged #128 (19 Oct): the one-shot now honors its sticky abort, fails the engine as WEDGED, and the next boot falls back to RCCL. Hub v0.31.0-extras lags — clone HEAD. If you still see the hang on a bake older than #128, A/B VLLM_RDNA_AR=0 (RCCL) or P2P=0.

A different non-mono hang (workers silent, same RPC timeout, #128 never reported WEDGED) was the CUDA-graph shared-experts aux-stream — next section.

Non-mono engine hang after ~minutes (CUDA-graph aux stream)

Symptom: serve looks healthy, then workers stop answering (~300 s RPC timeout) without --monokernel. #vllm-rdna (19 Oct) + merged #131: gfx10x SharedExperts forked to a second HIP queue inside captured FULL/PIECEWISE graphs; ROCm 7.14 could decrement a barrier signal twice and wait forever. Eager steps keep the overlap.

Fix is on HEAD (#131). Cost in the PR matrix was ~0 at c=1; MTP-0 c=8 up to ~1.5% slower. If you cannot pull HEAD, --monokernel avoided the hang in ~40 author-host serves (far fewer captured fork/joins per step) — not a substitute for the patch.

Prefill blocks decode / MTP stalls under concurrency

Symptom: with speculative decode (MTP) and multiple in-flight requests, generation stalls while prefill runs; or graph + MTP3 reaches “Application startup complete” then hangs on PLE lookup / sample_tokens timeout.

Community notes (#vllm-rdna):

• Prefer GPTQ + RDNA2 W4A16 (or AWQ HIP) paths over GGUF-in-vLLM for these cards. #vllm-rdna (Sep 15): RDNA2W4A16MoEExperts was reported to raise usable concurrency vs the older MoE path — confirm the kernel name in logs; A/B if your image is older.
• One prefill stream can fully block other decode streams (community Sep 15). When you bench, alternate prefill+decode in flight — concurrency-only or decode-only numbers miss this.
• Concurrent MTP / prefill-vs-decode fixes land in community recipes first — see open PRs on leapdragon/vllm-rdna2-recipe.
• MTP=0 vs MTP=3 are different bug surfaces; a commit that “works” at MTP=3 can still emit spurious tokens at MTP=0. A/B and pin a known-good recipe commit.
• Slow or broken P2P + custom all-reduce can look like MTP latency bugs — A/B the disable vs PIX custom AR stacks.
• If MTP never starts (drafter dies on ranks 1–3 / 60 s NCCL watchdog during profile_run), pull rdna_extras 700753d9 or later (amdsmi shutdown masking + local-argmax draft). #vllm-rdna (Sep 25): MTP-2 then boots but stays slower than MTP-0 on the author host.
• 30632b2 (27 Sep) captures the draft decode graph at max_num_reqs so MTP-2 c=8 is not eager every step. Author-host: +18% at 8×1k / +5% at 8×16k. Still ~0.87× vs MTP-0 at 16k — #vllm-rdna: “better MTP, still not great.”
• #vllm-rdna (Sep 27–29): decode-only concurrency is usable (two streams often ~70–80 t/s; three starts to sag). Mixed prefill + decode is the cliff — decode drops to ~1–6 t/s, prefill below 500 tok/s, MTP accept collapses. A local scheduler experiment first showed as VLLM_PREFILL_DEFER_INTERVAL=4 + --long-prefill-token-threshold 256, then as CLI --prefill-schedule-interval 2 or 4 (higher = more decode time between prefills) with --long-prefill-token-threshold 256 / 512 / 1024. Community: interval 4 + threshold 256 recovered ~20–30 t/s decode with a parallel prefill but halved single-stream PP. Interval 2 was called a milder prefill penalty. One host’s live 4-agent run at 1024 / 8 held mixed ~1030–1050 tok/s PP / ~33 t/s gen (decode-only ~24 t/s with ~94% prefix hit; single-stream PP sometimes ~1750). Synthetic benches were called untrustworthy vs live agentic. --max-num-batched-tokens 2048 slightly helped ITL on that host.
• Merged opengfx1030/vllm-rdna#36 (5 Oct 2026) is the public landing of that work: opt-in VLLM_RDNA_DYNAMIC_PREFILL=1 self-tunes the same two knobs (chunk cap + step cadence) from live traffic. Default off — the static CLI path is unchanged. #vllm-rdna (Oct 4–5): the earlier static pair still paused decode while a prefill ran; the dynamic controller kept decode moving (~25 t/s burst with pauses at 3 decode + 1 prefill). Author follow-up (Oct 5): recovered about half of the mixed-prefill cliff (below 500 → ~1100 tok/s PP) and auto-adjusts with load. Author: synthetic benches will not show much (slight tok/s drop, better inter-token latency). Still Needs verify on a second host; not in Hub -extras. When chasing max single-stream PP, leave the dynamic flag off and unset the static interval / threshold — they trade PP for mixed decode. Same Sep 29 thread: a ~2k → ~1450 PP drop on latest extras was resident MoE off (VLLM_RDNA_MOE_RESIDENT=1 recovered ~2k), not the scheduler flags.
• Concurrent MTP 9–16 token rows (three chats × MTP-2) used to fall back to tile Triton. Merged #26 opts in VLLM_ROCM_MOE_SKINNY_MAX_M=16 (default stays 8). PR regular-text c=3: ~47 → ~52 t/s aggregate (+11%). No single-chat gain. Coding-suite quality still failed dominant_repeated_token — do not treat as a quality fix.
• #vllm-rdna (Oct 9–10): on the v0.31 test line, mixed prefill/decode still stalls (per-request, then hangs). Maintainer: the fix looks like widening CUDA-graph capture range; c=8 also looked wrong on that host. DeepSeek-V4 is expected to work after that lands. Needs verify — no public flag / endpoint list yet; do not invent compile_ranges_endpoints. Day-to-day stays on rdna_extras HEAD.

wvSplitK vs stacked LLMM1 (concurrency)

#vllm-rdna (Oct 6): this is not the Flash-Next post-profile GPU fault (VLLM_RDNA_DENSE_GEMV=1). On a 4× W6800X host (same gfx1030 as V620) serving Qwen3.8-27B fp16, no quant, community said skinny-GEMM stacked LLMM1 is a poor decode path and that selecting rocBLAS wvSplitK (“opening the gate”) read weights better. Decode-only concurrency (no MTP) at 1k context:

Same host at 240k context: c=1 ~20.2, c=2 ~36.8, c=3 ~44.2 (c=4 not posted). With MTP that host reported ~36 t/s. Needs verify on V620; do not unset VLLM_RDNA_DENSE_GEMV=1 on hosts that already fault wvSplitK_hf_sml_* after profile.

#vllm-rdna (Oct 6) also posted 2× V620, Qwen3.8-27B fp16, TP=2, MTP, no quant: ~40–48 t/s single-stream. A shorter-reasoning Swift 1.5 27B tune on the same 2× / TP=2 / MTP layout: ~48 t/s single-stream and ~70 t/s across two instances (community: scales better than stock 3.8 27B). Not the Flash-Next Swift packs — two-card Flash-Next fit is still unconfirmed. Community / Needs verify. See recipes.

W6800X Duo ALU 0.9.4 (community, 0.30 MTP-2)

#benchmarks (Oct 10): same gfx1030 class as V620 — 4× W6800X Duo, vLLM 0.30.0 TP=4, Qwen3.8-27B fp16 (no quant), ALU 0.9.4, 512 generated tokens. Community, not wiki-tested, not a V620 serve recipe. Contrast the Sep 24 0.29.0 / MTP-0 row (~29 → ~19 TG, ~1394 → ~560 PP) in Verification.

Concurrency 1, APC off, MTP-2:

APC on, MTP-2 (per-stream / aggregate decode):

Prefill on the c=1 table stays in the ~1400–550 band of the older 0.29 no-MTP run; MTP-2 is the decode jump. Needs verify on V620.

Native vLLM KV RAM offload

#vllm-rdna (Sep 15): stock / pre-fix CPU → SSD KV cache offload (not LMCache) failed to bring blocks back to VRAM usefully. Community: decode fell to the ~1 t/s class. Mamba / SSM state models were called out as especially unreliable on that path (including on Hopper-class hosts in-channel). Treat that as the broken baseline, not the current fork experiment.

#vllm-rdna (Sep 26–27) + opengfx1030/vllm-rdna#24 (merged 27 Sep 2026 into rdna_extras): built-in RAM-tier KV offload — private pinned CPU buffers per ROCm rank, hybrid-group index corrections, MTP/EAGLE boundary blocks, and scratch-group filters. Community: ~190k tokens RAM→VRAM in ~0.4 s; three concurrent sessions of ~150k tokens each while VRAM held only ~270k; tool-call turns did not re-prefill the parked session. Slower than fitting everything in VRAM, especially when several sessions prefill at once. #24 is in Hub v0.31.0-extras; commits after the bake pin still want clone + host venv.

PR-recorded live swap (author host, 48 GiB RAM tier, swap off, SSD tier not enabled):

#vllm-rdna (Sep 27): a ~8 GiB RAM tier was ballparked at ~100k tokens. That is Needs verify and will scale with model / KV layout — do not treat 8 GiB as a production overflow budget. vLLM can also attach a second (storage) tier; #24 did not enable or test SSD.

#vllm-rdna (Oct 7): the user-facing CLI on current extras / 0.30+ trees is --kv-offloading-size <GiB> (optional --kv-offloading-backend native). Community parked unused sessions with --kv-offloading-size 64 (64 GiB total across TP ranks, not per GPU — see upstream cache config). RAM-tier only; storage offload still untested on gfx1030. Hybrid Qwen may still need --disable-hybrid-kv-cache-manager if the connector refuses HMA — that is a compatibility trade, not a speed win.

Needs verify on a second host. #24 / --kv-offloading-size are in Hub v0.31.0-extras; commits after the bake pin still want host venv. See fork landscape and Flash-Next KV.

#vllm-rdna (25–26 Oct): a community isolation that already left the #127 dynamic MTP schedule unset still token-repeated with MTP-2 and then with MTP off. Strongest suspect was CPU-memory / disk KV cache restoration. After KV offloading disabled (no --kv-offloading-size, no disk KV tier), the same host ran 4 h+ without a loop (VLLM_FA_RDNA2_GQA_DECODE=1 stayed on). #vllm-rdna (26 Oct follow-up): prefix cache + CPU-memory KV offload stayed loop-free for 6 h+ when filesystem / disk KV offload was off. Treat disk KV restore as the first A/B; RAM-tier parking is a weaker suspect on that host — Needs verify. Details: dynamic MTP.

#vllm-rdna (30 Oct): filesystem / disk KV at huge context can write terabytes per day to NVMe (one 8× V620 host with ~1M-token disk offload). Community: do not use consumer NVMe as a KV spill unless you accept that wear, or use Optane. Optane guidance from the same thread: M.2 M10 class is slow; prefer U.2 / PCIe / DIMM Optane. LMCache still has no working NVMe path in-channel. Draft #148 adds experimental native hybrid KV paging — the PR itself says live paging stalls; that is not a recipe. Prefer RAM-tier --kv-offloading-size or no offload while isolating loops.

Repeating !!!! tokens (clocks / undervolt)

#vllm-rdna (27 Oct): a different token-repeat (!!!! / !!!!!!) showed up while hunting the KV-restore loop. One host could only reproduce it with undervolt larger than ~20 mV or core above ~2479 MHz. Another host at 2350 MHz core / 1050 MHz memory, no undervolt, stock 250 W did not see it. A third reported 2400 / 1075 MHz at 160 W. Hardware check suggested in-channel: GpuZelenograd/memtest_vulkan plus dmesg for ECC errors. Community / Needs verify — do not treat those clocks as a wiki default. Power caps: Power tuning.

#vllm-rdna (Sep 16): public RAM-offload Flash-Next weight packs (example: Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound) are a different topic. Recipes that need --no-enable-prefix-caching remain a non-starter on gfx1030 vLLM — parking non-linear state in host RAM fights prefix cache.

#vllm-rdna (Sep 20): upstream vllm#57160 (mainline ROCm, not v0.29) uses private pinned tensors after cudaHostRegister failures on large TP. #24 is the gfx1030-facing follow-through — do not assume mainline #57160 alone fixes rdna_extras without the fork PR.

#vllm-rdna / #general (Sep 18–19): official rocm/pytorch images are the wrong place to compile an LMCache connector (missing HIP / developer packages). Intended shape:

1. Run lmcache/standalone in CPU mode — see the standalone starter.
2. Add the LMCache connector to a bare-metal / venv rdna_extras serve, or patch it onto blivioniag/vllm-rdna / blivioniag/rocm-rdna.
3. Do not use lmcache/vllm-openai on gfx1030 — that image is the CUDA path.

NVMe-as-KV is the usual motive (low host RAM). Until someone posts a working gfx1030 compose, treat this as Needs verify.

#vllm-rdna (Sep 22) + #lmcache (Sep 23–24): still no working gfx1030 compose. Extra facts:

• Do not remake standalone — use lmcache/standalone.
• A community 0.5.6.dev wheel built against gfx1030; that is the library, not a proven connector. Official LMCache kernels are CDNA-only; generic fallback still has Mamba align problems (last known real blocker on 0.5.4).
• Hybrid Qwen (MambaSpec + QSA FullAttentionSpec) creates multiple KV groups and needs HMA. LMCacheConnectorV1 is not HMA-capable (SupportsHMA missing; vendored copy still asserts a single group) → startup ValueError: Failed to promote local KV cache specs to one unified type.
• A community LDS ~143 KB vs 64 KB claim is still Needs verify (LLM-sourced, no compile log).

Keep the standalone-CPU + connector shape; expect fork patches before this is a recipe.

Leftover vLLM / PLE workers after a restart

#vllm-rdna (Sep 18): killing the OpenAI API process is not enough. VLLM::Worker, VLLM::EngineCore, and PleOffloadWorker can stay up and hold VRAM / the PLE sidecar. Before relaunch, stop those leftover processes (match the process names, not a broad pattern) and wait a few seconds. The 4× Flash-Next recipe already cds to /tmp so GPU coredumps do not land in a checkout.

ROCR idle CPU spin (TheRock 7.14)

Symptom: after starting a multi-GPU vLLM serve on TheRock / ROCm 7.14 (ROCR 1.21), the host holds several CPU cores at high utilization even when the GPUs are idle (~one core per HIP process, plus more once RCCL initializes).

Root cause: AsyncEventsLoop / signal-wait paths busy-spin without backoff (ROCm/TheRock#7051, ROCm/ROCm#6522). Stock env knobs (HSA_ENABLE_INTERRUPT, etc.) do not fix multi-GPU cases.

Fix: rebuild only libhsa-runtime64.so with the poll-backoff patch and LD_PRELOAD it. Step-by-step for host builds (and note that recipe containers already bake the patch): docs/rdna2/ROCR-CPU-FIX.md in leapdragon/vllm-rdna2-qwen. Pull latest recipe / container before re-debugging idle CPU.

rocprofiler register idle CPU (#108)

Symptom (#vllm-rdna 16 Oct 2026): a multi-GPU vLLM serve holds ≥1 CPU core per GPU even when the GPUs are idle. This is not the TheRock ROCR AsyncEventsLoop busy-spin (ROCR idle CPU).

Root cause: HIP/HSA rocprofiler client registration on every serve process. Merged opengfx1030/vllm-rdna#108 (16 Oct): tools/rdna/rdna_launcher_common.sh defaults ROCPROFILER_REGISTER_ENABLED=0.

Fix: stay on current rdna_extras HEAD (or export ROCPROFILER_REGISTER_ENABLED=0 yourself). Hub v0.31.0-extras (11 Oct bake) lags this default. Profiling runs set =1.

Flash-Next stalls, graph corruption, KV accounting, PP3, and PLE issues are on Flash-Next troubleshooting.