llama.cpp troubleshooting

KV checkpoint crash on tensor split

Symptom: fatal error in ggml-backend-meta.cpp during warmup with tensor split.

Fix: --ctx-checkpoints 0. Known on stock llama.cpp and the RDNA2 fork. See RDNA2 serving limits.

FlashAttention abort: max_blocks_per_sm > 0

Symptom: server aborts with something like:

fattn-common.cuh:…: GGML_ASSERT(max_blocks_per_sm > 0) failed
  launch_fattn<256, …>

#llamacpp / community reports this on gfx1030 when the HIP occupancy query returns 0 for the head-size 256 FA tile kernel (common on Qwen3.8 dense and some MoE paths).

Constraints that make this painful:

Mitigations to try (in order):

1. Prefer the RDNA2 fork build script (./scripts/build-rdna2-portable.sh) so FA paths match the fork’s gfx1030 profile.
2. Keep the simplified env stack — especially GGML_HIP_SAFE_STATE_IO=1 (known ROCm FA crash workaround). Do not pile on every GGML_HIP_GFX1030_* flag; that set can clash with the RCCL autotune path. See Serving.
3. If FA still aborts on head-256 models under TP: fall back to f16 KV + FA on only after a fork update / local occupancy patch, or temporarily use layer split for that model until FA occupancy is fixed upstream/fork-side.

DAX-backed mmap oopses amdgpu SVM

Symptom: loading a GGUF from an Optane / pmem dax=always mount with default mmap → kernel fault in svm_range_dma_map_dev, process becomes unkillable, VRAM leaked on all cards, reboot required.

Fix: --no-mmap and/or --load-mode none whenever the model file lives on a DAX mount. Storage only affects load time (community: Optane DAX ~4 GB/s vs NVMe hundreds of MB/s for a ~30 GB GGUF); once weights are in VRAM, inference is unchanged. Great for swap-testing models; useless for a single long-lived production load.

RCCL all-reduce fails (HIP “operation cannot be performed”)

Symptom: ggml_backend_cuda_comm_allreduce_nccl crash, NCCL WARN HIP failure.

Try in order:

1. Confirm PCIe P2P is working.
2. Set NCCL_P2P_LEVEL=PHB or NCCL_P2P_DISABLE=1.
3. On the RDNA2 fork: GGML_HIP_GFX1030_P2P_ALLREDUCE=off or GGML_CUDA_ALLREDUCE=none.
4. Check ACS — CPU root-port ACS can block GPU-direct P2P.

P2P enabled but slower inference

Bandwidth tests can pass while inference regresses on gen3 x4 or ACS-blocked topologies. A/B with NCCL_P2P_DISABLE=1. See When P2P hurts.

PSU dies the moment tensor-split prefill starts

Symptom: layer split is stable; --split-mode tensor kills power (no HIP error in logs). #llamacpp traced this to PSU transients, not the kernels — especially old miner PSUs and Lenovo P620 proprietary GPU cables (that chassis PSU often only feeds two cards).

1. Cap at 160 W or 140 W (Power Tuning).
2. A/B card pairs — one slot pair can trip protection while others do not.
3. Prefer 2 or 4 GPUs; TP3 has caused driver crashes after a “successful” run.
4. Split GPU power off the motherboard PSU if the board only has two GPU power ports.

Dual-socket / NUMA is slow

Pin llama.cpp to one socket (numactl --cpunodebind=0 --membind=0) and keep all TP GPUs on that socket. Crossing NUMA for tensor split is a known prefill killer — see Host topology.

Vulkan RADV hard-crashes; AMDVLK is slow

#llamacpp: Mesa RADV can hard-reboot or crash the host on V620 llama.cpp; switching the ICD to AMDVLK (VK_ICD_FILENAMES=/etc/vulkan/icd.d/amd_icd64.json) can get inference running but is much slower. For multi-GPU --split-mode tensor, the community path is ROCm / HIP, not Vulkan — tensor parallel needs RCCL. See Building llama.cpp.

If a new V620 hard-reboots a box that was stable with a 3080, read Slot power and PSU transients before chasing Vulkan ICDs.

DFlash2 / sidecar crashes

Symptoms (#llamacpp Sep 2026):

• HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION in a DFlash gemv_mmvq2_* kernel, then [dflash-sidecar] an illegal memory access was encountered and ROCm abort.
• Sidecar probe: qwen35-mtp target mismatch: target GGUF model identity differs → falls back off sidecar.

See Sidecar / DFlash gotchas: match publisher families for target/draft GGUFs, keep --spec-draft-p-min 0, pull latest fork, A/B MTP.

MTP slot wedge: LCP / prompt-cache position desync

Symptom (#llamacpp, Sep 2026): llama-server stays up and the HTTP API returns 200, but no tokens are generated (clients such as Hermes show 0 tok/s). Common on long agentic sessions (>~40k tokens; community failures clustered around 32–89k). Restarting the process is not required — a later new-context request often recovers the wedged slot.

Typical stderr fingerprint (M-RoPE / draft context):

W find_slot: non-consecutive token position … after … for sequence …
E init: the tokens of sequence … have inconsistent sequence positions:
  - the last position stored in the memory module … is X = …
  - the tokens … have a starting position of Y = …
  for M-RoPE, it is required that the position satisfies: X < Y
E spec process: llama_decode(ctx_dft) head=0 failed rc=-1
E srv decode: failed to process speculative batch

X = Y and X > Y both show up. The main decode can accept the batch; the MTP draft context (ctx_dft) is what rejects it.

This is not the same as the KV checkpoint crash. Reporters already had --ctx-checkpoints 0. The trigger is the LCP / prompt-cache slot-reuse path (selected slot by LCP similarity) plus --spec-type draft-mtp: a reused prefix is re-spanned at a stale position while the draft context holds another. It also is not specific to quantized KV — the same wedge was seen after switching KV from q8_0 back to f16.

Community setup that reproduced it daily: RDNA2 fork, 3× V620 tensor-split, Qwen3.8-27B Quark-AWQ-MXFP4, --parallel 2, Flash Attention on, --spec-type draft-mtp --spec-draft-n-max 4, --ctx-checkpoints 0.

Mitigations to try (in order):

1. Turn MTP off for that slot / session (or gate it when logs show non-consecutive token position).
2. Avoid prefix-cache reuse: new session / --parallel 1, or force a fresh slot after a 200-with-no-tokens event.
3. Prefer Q6+ / Q8 long-session quants over Quark-AWQ-MXFP4 — see Sidecar / DFlash gotchas.
4. Pull latest edwinbrowwn/llama.cpp-rdna2 and report the three-step fingerprint on #llamacpp if it still wedges.

If the hang is a failed tool call or a premature stop token (raw payload looks complete), that is a different class — inspect the harness payload before blaming MTP.

Concurrent decode collapses

Symptom: single-stream is ~35–50 t/s; with two overlapping generations (or a second prefill) decode falls to single-digit t/s.

Expected on V620 for many llama.cpp configs — see Concurrent slots. Prefer separate instances / GPUs over high --parallel for multi-agent.