RDNA2 fork — Speculative decoding
WIP: Community-tested configs. See Verification status.
DFlash / MTP basics
--spec-type draft-dflash \
--spec-draft-model /path/to/dflash.gguf \
--spec-draft-n-max 6
--spec-draft-n-max is a workload knob; start at the draft model’s block size and tune.
DFlash2 + ngram (community-tested)
DFlash2 shines on real workloads (coding, agents, long context) more than short synthetic benches — community members saw DFlash2 alone score lower than MTP on a 20k bench (~30 vs ~40 t/s) but win on actual agent/coding sessions (~47 t/s with 700+ t/s prefill).
Draft model quant: use Q4_K_M for the DFlash2 draft GGUF — do not use Q8_0 for the drafter (same acceptance, slower).
Recommended draft GGUFs (Aug 2026):
Coding / ngram-map-k4v:
--spec-type draft-dflash,ngram-map-k4v \
--spec-draft-n-max 5 \
--spec-ngram-map-k4v-size-n 12 \
--spec-ngram-map-k4v-size-m 48
General / ngram-mod:
--spec-type draft-dflash,ngram-mod \
--spec-draft-n-max 5 \
--spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 48 \
--spec-ngram-mod-n-max 64
Use tg1024+ (not tg32) for realistic decode benchmarks. DFlash2 is more consistent at long context;
MTP acceptance tends to fall off. #llamacpp (Aug 26 2026): prefer the DFlash2 Q4_K_M drafter
over MTP when context grows — same advice as the draft-quant row above.
Prefill cost: DFlash tends to hurt prompt processing more than MTP. If PP is the bottleneck (short chats, many reconnects), A/B MTP first; if long-context decode matters more, keep DFlash2.
Qwen3.8 Flash-Next (experimental)
Community benches (4× V620, layer vs tensor split) show Flash-Next layer-split can run, while tensor split on experimental qwen4exp / Flash-Next branches is still rough. Fork maintainers have deferred dedicated Flash-Next work until upstream settles — expect experimental support via upstream merges only, not a polished gfx1030 profile.
#llamacpp / forum (Sep 2026): Flash-Next on llama.cpp often lands around ~15–30 t/s on multi-V620
hosts and is widely called out as weaker than the
vLLM Flash-Next recipe (~60–100+ t/s decode class
after the Sep prefill campaign). Prefer stable Qwen3.8-27B / MoE recipes for production TP on llama.cpp;
use vLLM for Flash-Next until upstream/fork gaps close.
MTP load: #llamacpp / #general (Sep 16–18): Unsloth Studio and some RDNA2 forks
often refused Flash-Next MTP until you built a tree with the upstream MTP PR, or used
Unsloth desktop. Community Sep 18: stock llama.cpp + MTP was the path that loaded for one
host when forks did not. #llamacpp (Sep 19): two public trees did load MTP when Studio’s
bundled llama.cpp would not (including versus that Studio fine-tune) —
markldn/llama.cpp-qwen4exp-rdnaport-latest
and okigan/llama.cpp qwen4exp-mtp-fix.
The same host could not get tensor parallel on either. #llamacpp (Sep 20): that host then
reported Studio’s bundled llama.cpp now loads MTP (called a same-day bundle fix) — re-try the
bundled binary before compiling a fork. Needs verify (one report). Public walkthrough:
Unsloth Qwen3.8-Flash-Next / MTP. Ubuntu 26.04
shipping ROCm 7.1 is a separate A/B from MTP (wiki still prefers 7.2.0 or 7.14.0 for
multi-GPU). 3× V620 + 32 GB host RAM was not enough; community later loaded 3× + MTP with
about 64 GB host RAM (SSD offload untested). 4× + 128 GB with Q4 weights / Q8 KV / Q8 MTP
is the class that got a larger server up.
#llamacpp (Sep 10–11 2026) community snapshots — still not a polished gfx1030 profile:
| Setup | Quant / backend | Prefill | Decode | Notes |
|---|---|---|---|---|
2× V620, LocalAI rocm-llama-cpp-development | Qwen3.8 Flash-Next Q4 | low | ~6 t/s | n-gram offloaded to NVMe; 64 GB host RAM |
| 2× V620, 64 GB VRAM | mudler/Qwen3.8-Flash-Next-APEX-GGUF compact (~85 GB GGUF) | ~370+ t/s | ~26 t/s | n-gram table on NVMe (no extra host RAM); 156k KV at q8 |
| 2× V620, layer split + CPU MoE | Flash-Next UD-IQ4_XS + F16 mmproj | ~150–200 t/s | ~25 t/s | #vllm-rdna / #llamacpp Sep 14–15 — see recipe below; vLLM not recommended on 2 cards |
| 2× V620, official llama.cpp ROCm server | Unsloth Flash-Next UD-IQ3_XXS + F16 mmproj, n-gram in host RAM | ~530 t/s (lower -ub; 600+ if it fits) | ~30 t/s empty / ~10 t/s at 80k+ ctx | #general / #forum Sep 15 — stock ROCm server, not the RDNA2 fork; Vulkan A/B was ~25% slower |
| 2× V620, llama.cpp Flash-Next | UD-IQ4_XS class (layer split) | ~300 t/s | ~25 t/s | #llamacpp Sep 16 — called normal for llama.cpp Flash-Next; drops toward ~200 PP / ~15 t/s on long agentic ctx |
| 2× V620, llama.cpp (no MTP) | Navin-Models/Qwen3.8-Flash-Next-Uncensored-AD-4.27-GGUF | ~550 t/s | ~29–30 t/s (about half on long/heavy) | #llamacpp Sep 28–29 — weights minus n-gram in 64 GB VRAM; 262k KV at q8. MTP headers still failed. Host was PCIe 3.0 / DDR4 (HP Z4 G4 class). Same thread also cited ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF as sitting between Unsloth Q3_XL and Q4_XS (no tok/s). #llamacpp Sep 30: another 2× host used the ISTA-DASLab coder pack and preferred it over 27B Q8 for one coding task (Needs verify). LocalAI YAML: serving |
| 2× V620, llama.cpp MTP | ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_xs-class + q4_0 ngrams | ~400 t/s | ~40 t/s | #llamacpp Sep 30 — ~48 GB host RAM. --batch-size 4096 --ubatch-size 1024 helped PP a little. Shorter-reasoning Swift; quality Needs verify |
| 2× V620, llama.cpp MTP, IQ3_XXS | Same Swift GSQ-RCO IQ3_XXS + BF16 mmproj + Q4_0 MTP | ~800 → ~650 t/s at ~30k | ~35 → ~20 t/s | #llamacpp Oct 2 — tensor split 1,1.1, -ot 'per_layer_token_embd\\.weight=CPU', --lazy-mode off --load-mode none, -c 71680, -b 2048 -ub 1024. Community: VRAM-bound. Recipe: serving |
| 3× V620, llama.cpp + MTP | Flash-Next | — | loaded | #llamacpp Sep 17 — ~64 GB host RAM; 32 GB did not load. SSD offload untested. |
| 4× V620, patched llama.cpp MTP | Q4 + Q8 KV + Q8 MTP | ~400 t/s | ~27–45 t/s | #llamacpp Sep 17 — MTP needed a compiled PR; others call 400 PP low vs ~700–800 PP on 2× at low ctx. Layer split can halve TG. |
| 4× V620, n-gram in host RAM | Flash-Next (non-APEX) | ~200–400 t/s | ~20–35 t/s | Context-dependent; PP/tg still below 27B on the same rig |
Host RAM for Flash-Next n-gram: keeping the n-gram table off storage is on the order of ~50 GB of system RAM (community, 4× V620). Budget that on top of OS + any CPU offload — or keep the table on fast NVMe and accept the slower path (APEX report: NVMe-resident n-gram was fine).
#vllm-rdna (Sep 14): if you must CPU/RAM-offload Flash-Next (2× V620, or 4× with a too-small
KV pool), llama.cpp is the better offload engine than Hub / resident vLLM. HEAD after
#146 (28 Oct) adds a vLLM expert-offload
wrapper — 2× serve. vLLM still wins on
4× when everything stays in VRAM.
2× V620 community recipe (IQ4_XS, layer split)
Sanitized from a community “fastest I got” serve line. -t / --threads-batch are host-specific
— set them to your cores. Layer-split ratios (-ts 53,47) and --n-cpu-moe 6 are starting points,
not a fork profile.
ROCBLAS_USE_HIPBLASLT=1 ./build/bin/llama-server \
-m /path/to/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
--alias flashnext \
-ngl 99 -sm layer -ts 53,47 \
-c 65000 --flash-attn on --parallel 1 \
--cache-type-k f16 --cache-type-v f16 \
--n-cpu-moe 6 \
--mmproj /path/to/mmproj-F16.gguf \
-ot "ple_ngram_embd=CPU" \
-t 8 --threads-batch 14 -b 2048 -ub 512 \
--spec-type ngram-mod \
--spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 12 \
--spec-ngram-mod-n-max 16 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
--host 0.0.0.0 --port 8081 --jinja --load-mode none
Community snapshot on that host: ~25 t/s decode, ~150–200 t/s prefill. Prefer the
RDNA2 fork over a random ROCm build. --load-mode none avoids
DAX mmap issues if the GGUF
sits on a special mount.
Full DFlash2 serve example (TP4, Qwen3.8-27B)
HSA_NO_SCRATCH_RECLAIM=1 GGML_HIP_RDNA2_AUTO=1 GGML_HIP_SAFE_STATE_IO=1 \
GGML_TP_SHARDED_OUTPUT=1 HSA_OVERRIDE_GFX_VERSION=10.3.0 \
./build/bin/llama-server \
-m ./models/qwen38-27b-q4s8/autoround/Qwen3.8-27B-Q4_0.gguf \
-ngl all --split-mode tensor --tensor-split 1,1,1,1 \
--device ROCm0,ROCm1,ROCm2,ROCm3 --flash-attn on \
--ctx-size 262144 --batch-size 8192 --ubatch-size 4096 \
--host 0.0.0.0 --port 8080 --metrics \
--reasoning-effort xhigh --reasoning-preserve \
--spec-type draft-dflash,ngram-map-k4v \
--spec-ngram-map-k4v-size-n 12 --spec-ngram-map-k4v-size-m 48 \
--spec-draft-n-max 5 \
-md ./models/qwen38-27b-q4s8/dflash2/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
--device-draft ROCm0 --parallel 1 --spec-draft-ubatch-size 4096 \
--cache-ram 65535
MTP on tensor-split setups
Pin the draft to one GPU while the main model stays tensor-split:
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-ngl 999 --spec-draft-device ROCm0
Community tuning tip (Qwen3.8-27B Q8, TP4, real prompts): --spec-draft-n-max 3 often beat
n=4 (higher acceptance / mean accepted length). A/B on your workload — do not trust 100% draft
acceptance in synthetic benches (usually a bad prompt).
MXFP4 quants (e.g. quark75/Qwen3.8-27B-MXFP4-GGUF) pair well with MTP on TP2 — see
Serving.
HIP sidecar speculative decoding is now an opt-in fork path (#llamacpp, Sep 2026):
export SPEC_SIDECAR=1 # moves MTP/DFlash logic outside the main llama.cpp process
Pair with --spec-draft-p-min 0.0 (fork maintainer tip). Sidecar work is still evolving — pull latest
edwinbrowwn/llama.cpp-rdna2 and prefer MTP over DFlash on the sidecar path while DFlash remains slow.
Track the fork README for flag churn; do not treat SPEC_SIDECAR as required for the built-in MTP flags
above.
Sidecar / DFlash gotchas (Sep 2026)
#llamacpp reports to know about:
| Issue | What happens | Mitigation |
|---|---|---|
| Draft GGUF identity mismatch | Sidecar probe fails with target GGUF model identity differs; falls back to native MTP / no sidecar | Prefer the same publisher family for target + draft (community: Unsloth Qwen3.8-27B Q8_0 loaded sidecar; some Bartowski / AtomicChat Q8_0 builds did not). Re-pull fork after identity-matching fixes |
DFlash2 MEMORY_APERTURE_VIOLATION | Crash in gemv_mmvq2_* / [dflash-sidecar] illegal memory access; server may enter target-only then abort | Reboot / clean env, pull latest DFlash sidecar fixes, A/B MTP instead of DFlash2; report host + quant on #llamacpp |
--spec-draft-p-min ≠ 0 | Community: non-zero p-min can effectively disarm MTP acceptance | Keep --spec-draft-p-min 0 (or 0.0) unless you have measured otherwise |
| Quark / MXFP4 lockups | Quark-AWQ-MXFP4 GGUFs can look great then wedge after long ctx | Prefer Q6+ / Q8 for long sessions; community: sub-Q6 feels unusable over long context |
| MTP + LCP slot reuse | HTTP 200, no tokens; inconsistent sequence positions / llama_decode(ctx_dft) after find_slot: non-consecutive | Disable MTP, force a fresh slot, or see MTP LCP position desync. --ctx-checkpoints 0 does not prevent this |