Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Flash-Next (vLLM)

WIP: Benchmarks and serve lines are community-reported (#vllm-rdna). See Verification status. Day-to-day 27B serving stays on Recipes (Hub -extras or the recipe container). This page is only the Flash-Next track.

On this page

SectionWhat it answers
Status by card count2× / 3× / 4×, QSA, KV tightness
Intel AutoRoundW4A16 draft checkpoint and gotchas
Swift 1.5 Flash-NextShorter-reasoning AWQ / AutoRound packs
Serve linesFULL_AND_PIECEWISE, #17, PP3, MonoKernel, DFlash2, INT4 KV
Serve scriptsserve_rdna.sh RECIPE=flashnext-mtp2 and siblings
Fork progressHub vs HEAD vs open drafts

When something fails, use Flash-Next troubleshooting rather than the general vLLM list.

Qwen3.8 Flash-Next on vLLM

llama.cpp still struggles with Flash-Next on gfx1030 (upstream gaps). On 4× V620, community production is the Flash-Next vLLM stack on rdna_extras (v0.31.0) — not the old v0.27.1-extras* Hub tags. #announcements (11 Oct 2026): Hub blivioniag/vllm-rdna:v0.31.0-extras served Flash-Next MTP-2 (BF16 + INT4-disk). #vllm-rdna Sep 14–15: that work lives on opengfx1030/vllm-rdna rdna_extras; leapdragon/vllm-rdna2-qwen is the older published container / docs tree (still receives occasional commits; last push 11 Oct 2026). Prefer latest rdna_extras or the 0.31 Hub tag over llama.cpp until the RDNA2 llama.cpp fork catches up.

2× V620 / RAM offload: resident Flash-Next experts still do not fit two V620s. Merged #146 (28 Oct) is the first in-tree vLLM path: pinned-RAM expert banks + a bounded GPU slot cache, launched with tools/rdna/mono/serve_offload.sh on HEAD — 2× serve. Hub v0.31.0-extras lags. #vllm-rdna (Sep 14): without that HEAD launcher, use llama.cpp (--n-cpu-moe, -ot …=CPU). Community 2-card llama.cpp Flash-Next is still ~25 t/s decode / ~150–200 PP — far below the 4-card vLLM class. #vllm-rdna (16 Oct): MonoKernel alone does not make resident Flash-Next fit 2 cards.

3× V620: vLLM wants PP=3 (even TP). Current rdna_extras HEAD (merged #96, 13 Oct) ships RECIPE=flashnext-pp3 / flashnext-pp3-mtp2 — PP3 recipes. Hub v0.31.0-extras (11 Oct bake) does not include those recipes. Older Sep 15–16 pins corrupted PP3 decode on rdna_extras — troubleshooting. The Sep 17–18 leapdragon overlay is a snapshot — overlay.

4× V620 + MonoKernel (#94, 16 Oct): opt-in --monokernel on HEAD for Flash-Next decode ITL — MonoKernel. Hub v0.31.0-extras lags.

4× V620 + HEAD HIP defaults (#130, 19 Oct): Flash-Next recipes on current rdna_extras turn the #99 HIP set on (QSA / GDN / HC). Hub bake still has the older opt-in path — HIP defaults.

4× V620 + DFlash2 (#132, 21 Oct): RECIPE=flashnext-dflash2 is on HEAD. Paired A/B kept native MTP-2 faster; use DFlash2 only if you want that drafter — DFlash2.

4× V620 + rdna_extras graphs (#vllm-rdna Sep 16–25): on current HEAD, prefer FULL_AND_PIECEWISE (compile mode: 3) — uniform decode uses the FULL CUDA graph (same live path as #17 FULL_DECODE_ONLY); mixed/prefill use piecewise. See FULL_AND_PIECEWISE serve and fork docs/rdna2/V620-FULL-AND-PIECEWISE.md. #20 alone still redirected FULL→piecewise (~26–34 t/s decode); the keep-FULL fix (6c26c78d, #vllm-rdna Sep 24) turns that redirect off. #17 FULL_DECODE_ONLY remains a valid measured baseline — Sep 23 serve notes. Older hosts that saw FULL decode corrupt stayed on PIECEWISE — FULL-graph corruption / Sep 17 serve line. Community snapshot on that older recipe: ~3331 tok/s PP / ~73 tok/s TG @ 16k/1k, c=8 (Needs verify). There is no Q3 path on vLLM.

6× V620 Flash-Next (#vllm-rdna Sep 24): minimum 3 cards. Maintainer starting points: PP=3 + TP=2 (six GPUs) or PP=6. Prefer power-of-2 tensor parallel over PP=2 + TP=3. Disaggregated prefill/decode (two model loads) is a separate WIP — open #23. Needs verify — no published 6× table.

#vllm-rdna (Sep 19): pipeline parallel 4 on Flash-Next was reported to hit ~1950 tok/s prefill at 32k while decode collapsed. The same host dropped PP4 and continued on TP=4 (~1550 tok/s prefill at long context; decode ~30–35 t/s with MTP-2). Treat PP4 as a prefill experiment, not a 4-card default. Needs verify.

#vllm-rdna (Sep 20): opengfx1030/vllm-rdna#15 merged into rdna_extras. AMD QSA prefill scoring is bounded to the live prompt context (decode / missing metadata still use the capacity-wide path). The same PR folds V620 startup prerequisites from superseded PR #12. Combined-stack benches on TP=4 / PP=1 / EP=4 / MTP-2, Intel AutoRound INT4 experts, FP16 dense, original BF16 PLE in CPU RAM — Community / Needs verify (full host stack, not an isolated QSA A/B):

SuiteContextPrefill tok/sDecode tok/s
Regular16k2,01163.5
Coding16k2,03668.5
Regular32k2,02455.2
Regular64k1,97459.4
Coding64k1,98670.6
Regular128k1,85953.9
Coding128k1,83272.5

The PR records a 31–33% 16k/32k prefill lift versus an immediately collected ~1529–1543 tok/s baseline on that same combined stack. Community take: near-PP4 prefill with TP4 decode. TP2+PP2 was not tried. Fused QSA multi-step draft decode with MTP-3 did not help. INT8 decode shadows are not in this merge. Prefer Hub v0.31.0-extras or latest rdna_extras (commits after the 11 Oct bake pin still want a host venv).

#vllm-rdna (Sep 21): do not treat the ~1.8–2.0k PP table as the public-merge floor. Two other 4× hosts on rdna_extras after #15 landed ~1.07–1.45k tok/s prefill. Decode often did reach the PR band (~50–77 t/s) once CUDA graphs were on (one host: ~24 t/s eager vs ~66–76 t/s with graphs). One of those hosts had octachannel DDR4-3200, PCIe gen4 x16 on all four cards, and NVMe ~6.5 GB/s — so the gap is not just “slow RAM / gen3”. A second host (DDR4-2144 ECC, PCIe gen3, SATA SSD, ~96 GB RAM, int4 PLE in RAM) sat around ~1.1–1.2k PP / ~45–50 t/s with MTP-1. The PR author later said the 1350 → 1550 → 2000 prefill steps needed two local changes that were not on the git checkout others pulled, and planned to re-apply those from current rdna_extras through git. #vllm-rdna (Sep 22): the author later recovered ~1950 tok/s PP on a local tree and said something was still missing before those changes could land on the public branch. Treat ~1.1–1.5k PP as the #15-only public-merge class.

#vllm-rdna (Sep 23): that missing stack merged as opengfx1030/vllm-rdna#17 into rdna_extras. PR-head 16k (same llm-context-bench harness): 1,957.8 / 1,982.8 tok/s PP, decode ~69 / ~69 t/s. A second 4× host confirmed those cells (1,983.8 / 1,985.3 tok/s PP) after rebuilding TunableOp for its rocBLAS — mismatch. The #15 author-host ~2k table is no longer “unpublished”; #17 shipped in Hub v0.31.0-extras. Community / Needs verify on your topology.

#vllm-rdna (Sep 2026) ballpark on 4× V620 (host-dependent; fork author + community):

MetricEarlier recipeAfter Sep 4–6 prefill/decode work
Sustained prefill~580–700 tok/s~1000–1400 tok/s (fork author RESULTS.md ~1080–1180; #vllm-rdna Sep 13 community ~1100–1390 @ 32k–128k after latest Flash-Next pull)
Decode (single-stream)~50–64 tok/s~60–100+ tok/s class depending on MTP acceptance / prompt (fork RESULTS.md; community Sep 13 ~53–78 t/s on long suites, MTP=2)
vs llama.cpp Flash-Next—Community: container ~40 t/s vs llama.cpp ROCm ~18–19 t/s on the same host

Docs live under docs/rdna2/ (README.md, RESULTS.md, TROUBLESHOOTING.md, ROCR-CPU-FIX.md). Pull latest before re-benching — tags and container latest move with the prefill campaign.

Long-prompt stalls / timeouts: if large agentic prompts (tens of k tokens) hang or take many minutes while short prompts are fine — or GPUs sit at 100% util / ~40 W with TTFT swinging from seconds to minutes — try VLLM_USE_V2_MODEL_RUNNER=0. #vllm-rdna Sep 15: that switch unblocked a TP4 leapdragon container. Community: MoE on the 0.28 Flash-Next line wants the V1 runner until upstream 0.29 (V2 becomes the default). See vLLM troubleshooting.

KV / concurrency (4× 32 GB, #vllm-rdna Sep 14 + Sep 26–27): Flash-Next KV is expensive. Community ballpark ~24 GiB ≈ 300k tokens once the PLE sidecar is loaded — a 4× V620 box can still be tight (~280k tokens left on one host). The Sep 14 report that 128 GB host RAM offload was not enough (and that tool-call sessions re-prefilled) describes the pre-#24 / broken native path. Merged opengfx1030/vllm-rdna#24 (27 Sep) is the RAM-tier fix — community: three ~150k sessions while VRAM held ~270k, ~0.4 s to bring ~190k back, no full re-prefill on tool calls. PR capacity on a 48 GiB reserved tier: ~450k retained tokens. Needs verify on a second host; slower than fitting in VRAM. Hub tags lag. Details: native KV RAM offload. Concurrent streams still split decode (community: decode-only two streams often ~70–80 t/s; mixed prefill + decode drops decode to ~1–6 t/s and PP below 500). Prefer one stream plus prefix / radix cache unless you have measured overflow on #24. Merged #36 (5 Oct) adds opt-in VLLM_RDNA_DYNAMIC_PREFILL=1 so those interval / threshold knobs self-tune for mixed load — Needs verify, not a default serve recipe (troubleshooting).

#vllm-rdna (Sep 17): the logged hybrid KV token count can be ~2.5× too high versus a measured peak — size --max-model-len from a real request, not the banner (overstated pool). Prefix cache on rdna_extras needed e45dd5cb before repeat prompts actually hit (zero hits).

FA-RDNA2 vs QSA (#vllm-rdna 13 Oct): FA-RDNA2 is attention for any model that uses the normal backend selector — it is not dense-only. Flash-Next QSA full-attention layers bypass that selector (Triton QSA owner), so --attention-backend / VLLM_USE_RDNA2_FA only apply partially. Details: fork attention backends, troubleshooting.

Intel AutoRound Flash-Next (draft, #vllm-rdna Sep 10–11 2026)

A second Flash-Next track is the Intel W4A16 AutoRound checkpoint (Intel/Qwen3.8-Flash-Next-W4A16-AutoRound; community also cites the RTN sibling Intel/Qwen3.8-Flash-Next-W4A16-RTN-AutoRound). Weights are ~75 GB; plan on 4× V620. Community quality notes: Intel AutoRound beats some other INT4 Flash-Next packs on published tool-calling scores (example cited: 80.5 vs 76.0 for a different INT4). Treat those as publisher / community numbers, not wiki benches.

Draft integration: opengfx1030/vllm-rdna#5 (open, not in Hub -extras). PR validation used FP16 activations, CPU PLE / n-gram offload, and --max-num-batched-tokens 4096. Published short-run figures on 4× V620 (no P2P in the Discord report):

MetricCommunity / PR snapshot
Uncached 1024-token prefill~962 tok/s (vs ~535 BF16 on that host)
Short decode + MTP~41 tok/s, ~61% MTP accept
Prose / code 16–64k~950–980 tok/s PP, ~48–56 tok/s decode
128k after 4096 batched tokensPP stays ~950 tok/s class (was ~375 tok/s at 2048 scheduled tokens)
Community long-suite (Sep 16, 4× V620)Regular 32–128k: ~1296–1393 tok/s PP / ~56–60 tok/s TG; coding suites ~68–73 tok/s TG. Pin cited: Intel/Qwen3.8-Flash-Next-W4A16-AutoRound @ 4c67bf686b7f7fd386bae6b07ab59e8ff1d5b897. Further push hit int8 decode-shadow quality loss.
QSA live-context (rdna_extras #15, Sep 20)Author-host combined stack ~1830–2040 tok/s PP / ~54–73 tok/s TG. #vllm-rdna Sep 21: other 4× hosts on #15 only reproduced ~1.1–1.45k PP. The unpublished recovery merged as #17 (Sep 23) — see QSA merge.
Startup (warm-ish)~4 min vs earlier 8–10 min on the same host
fp16 KV fit~291k tokens on 4 cards (one report)

Gotchas (do not treat as a drop-in Hub image):

  • Default PLE is still the embedded BF16 n-gram table (~95 GiB tensor data) with --engram-config '{"cpu_offload":true}'. The historical group-16 INT4 sidecar was not end-to-end validated (#vllm-rdna saw incoherent generation on that older path). Merged #49 (11 Oct, rdna_extras) plus the v0.31.0 launcher PLE=int4|int4-disk path. #announcements (11 Oct): all three PLE modes tested; speed at parity. Hub v0.31.0-extras includes this.
  • --max-num-batched-tokens 2048 + chunked prefill was the suspected cause of the 128k PP cliff. Raise to 4096 (or see troubleshooting).
  • Concurrent varlen GDN prefill still has a fork kernel addressing bug (gdn_prefill_o_rdna2.cu reuses a global chunk index). Single-stream benches can look fine.
  • The PR is huge and dirty against rdna_extras — wait for a cleaned cherry-pick / image before calling it production. Day-to-day Flash-Next remains leapdragon/vllm-rdna2-qwen.
  • #vllm-rdna (12 Oct): latest rdna_extras can reject this Intel pack at startup even when PLE is unquantized (INCConfig / Qwen4Exp PLE embedding). Upstream vLLM #59990 is merged; the fork cherry-pick was still in queue. Prefer wtdcode AWQ + primitive-ai PLE-quant until that lands — see troubleshooting.

Also watch TheRock ROCR idle-CPU spin on ROCm 7.14 hosts.

Serve commands (4× PIECEWISE, #17, PP3, overlays) are on Flash-Next serve lines.

Swift 1.5 Flash-Next (#vllm-rdna Sep 25–26)

Public shorter-reasoning derivative of Qwen3.8 Flash-Next (ukisai/Swift-1.5-Qwen3.8-Flash-Next-W4A16-AWQ; AutoRound sibling ukisai/Swift-1.5-Qwen3.8-Flash-Next-W4A16-AutoRound). Publisher card (BF16 vs base, not a wiki bench): ~63% fewer median thinking tokens at xhigh with under 1% claimed accuracy loss. AWQ is compressed-tensors W4A16 — same runtime class as other Flash-Next INT4 packs, not Hub -extras.

#vllm-rdna (Sep 25–26) community first-look on 3× V620: the AWQ pack ran at about the same speed as the host’s usual AutoRound Flash-Next line (~1300 tok/s prefill / ~45 t/s decode) and was called more verbose. Treat tok/s as Needs verify. Two-card fit is unconfirmed (community: “won’t fit”).

AutoRound caveat: the Swift AutoRound config is 50 iterations / light tuning (512-token calibration). Community compared that to Intel AutoRound’s 200 iterations and noted Intel’s attention projections stay bf16 while the Swift AutoRound pack quantized them to int4. #vllm-rdna (Sep 26): Swift AutoRound was ~5% faster than the host’s usual Flash-Next line — expected if the pack is more aggressively quantized, not a quality win. #vllm-rdna (Sep 27): the same AutoRound pack looped badly on a later host — treat that as a quality warning, not a speed footnote. Do not treat “AutoRound vs AWQ quality” as settled from Discord — read the quant config, and prefer the AWQ sibling if you want a first try. Community / Needs verify.

llama.cpp sibling: ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF (#llamacpp Sep 30–Oct 2). Community 2× V620, IQ3_xs-class + q4_0 ngrams in ~48 GB host RAM, MTP: ~400 PP / ~40 TG (4k/1k batch). #llamacpp Oct 2 IQ3_XXS + CPU per_layer_token_embd + MTP: ~800 → ~650 PP at ~30k / ~35 → ~20 TG (2k/1k batch; community called it VRAM-bound). Needs verify. See speculative table and serve line.