RDNA2 fork — Speculative decoding

WIP: Community-tested configs. See Verification status.

DFlash / MTP basics

  --spec-type draft-dflash \
  --spec-draft-model /path/to/dflash.gguf \
  --spec-draft-n-max 6

--spec-draft-n-max is a workload knob; start at the draft model’s block size and tune.

DFlash2 + ngram (community-tested)

DFlash2 shines on real workloads (coding, agents, long context) more than short synthetic benches — community members saw DFlash2 alone score lower than MTP on a 20k bench (~30 vs ~40 t/s) but win on actual agent/coding sessions (~47 t/s with 700+ t/s prefill).

Draft model quant: use Q4_K_M for the DFlash2 draft GGUF — do not use Q8_0 for the drafter (same acceptance, slower).

Recommended draft GGUFs (Aug 2026):

• incoai/Qwen3.8-27B-DFlash2-GGUF
• webhie/Qwen3.8-27B-Q4-AutoRound-Code-GGUF

Coding / ngram-map-k4v:

--spec-type draft-dflash,ngram-map-k4v \
  --spec-draft-n-max 5 \
  --spec-ngram-map-k4v-size-n 12 \
  --spec-ngram-map-k4v-size-m 48

General / ngram-mod:

--spec-type draft-dflash,ngram-mod \
  --spec-draft-n-max 5 \
  --spec-ngram-mod-n-match 24 \
  --spec-ngram-mod-n-min 48 \
  --spec-ngram-mod-n-max 64

Use tg1024+ (not tg32) for realistic decode benchmarks. DFlash2 is more consistent at long context; MTP acceptance tends to fall off. #llamacpp (Aug 26 2026): prefer the DFlash2 Q4_K_M drafter over MTP when context grows — same advice as the draft-quant row above.

Prefill cost: DFlash tends to hurt prompt processing more than MTP. If PP is the bottleneck (short chats, many reconnects), A/B MTP first; if long-context decode matters more, keep DFlash2.

Qwen3.8 Flash-Next (experimental)

Community benches (4× V620, layer vs tensor split) show Flash-Next layer-split can run, while tensor split on experimental qwen4exp / Flash-Next branches is still rough. Fork maintainers have deferred dedicated Flash-Next work until upstream settles — expect experimental support via upstream merges only, not a polished gfx1030 profile.

#llamacpp / forum (Sep 2026): Flash-Next on llama.cpp often lands around ~15–30 t/s on multi-V620 hosts and is widely called out as weaker than the vLLM Flash-Next recipe (~60–100+ t/s decode class after the Sep prefill campaign). Prefer stable Qwen3.8-27B / MoE recipes for production TP on llama.cpp; use vLLM for Flash-Next until upstream/fork gaps close.

#llamacpp (Sep 10–11 2026) community snapshots — still not a polished gfx1030 profile:

Host RAM for Flash-Next n-gram: keeping the n-gram table off storage is on the order of ~50 GB of system RAM (community, 4× V620). Budget that on top of OS + any CPU offload — or keep the table on fast NVMe and accept the slower path (APEX report: NVMe-resident n-gram was fine).

Full DFlash2 serve example (TP4, Qwen3.8-27B)

HSA_NO_SCRATCH_RECLAIM=1 GGML_HIP_RDNA2_AUTO=1 GGML_HIP_SAFE_STATE_IO=1 \
GGML_TP_SHARDED_OUTPUT=1 HSA_OVERRIDE_GFX_VERSION=10.3.0 \
./build/bin/llama-server \
  -m ./models/qwen38-27b-q4s8/autoround/Qwen3.8-27B-Q4_0.gguf \
  -ngl all --split-mode tensor --tensor-split 1,1,1,1 \
  --device ROCm0,ROCm1,ROCm2,ROCm3 --flash-attn on \
  --ctx-size 262144 --batch-size 8192 --ubatch-size 4096 \
  --host 0.0.0.0 --port 8080 --metrics \
  --reasoning-effort xhigh --reasoning-preserve \
  --spec-type draft-dflash,ngram-map-k4v \
  --spec-ngram-map-k4v-size-n 12 --spec-ngram-map-k4v-size-m 48 \
  --spec-draft-n-max 5 \
  -md ./models/qwen38-27b-q4s8/dflash2/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  --device-draft ROCm0 --parallel 1 --spec-draft-ubatch-size 4096 \
  --cache-ram 65535

MTP on tensor-split setups

Pin the draft to one GPU while the main model stays tensor-split:

--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-ngl 999 --spec-draft-device ROCm0

Community tuning tip (Qwen3.8-27B Q8, TP4, real prompts): --spec-draft-n-max 3 often beat n=4 (higher acceptance / mean accepted length). A/B on your workload — do not trust 100% draft acceptance in synthetic benches (usually a bad prompt).

MXFP4 quants (e.g. quark75/Qwen3.8-27B-MXFP4-GGUF) pair well with MTP on TP2 — see Serving.

HIP sidecar speculative decoding is now an opt-in fork path (#llamacpp, Sep 2026):

export SPEC_SIDECAR=1   # moves MTP/DFlash logic outside the main llama.cpp process

Pair with --spec-draft-p-min 0.0 (fork maintainer tip). Sidecar work is still evolving — pull latest edwinbrowwn/llama.cpp-rdna2 and prefer MTP over DFlash on the sidecar path while DFlash remains slow. Track the fork README for flag churn; do not treat SPEC_SIDECAR as required for the built-in MTP flags above.

Sidecar / DFlash gotchas (Sep 2026)

#llamacpp reports to know about: