Flash-Next serve lines

WIP / community. Copy from here only after you have read Flash-Next status (which card count, which graph mode). Failures: Flash-Next troubleshooting.

Path C — Flash-Next (rdna_extras / Hub 0.31)

For agentic / long-context Flash-Next on 4× V620, use the Flash-Next vLLM stack — not the 27B recipe presets and not the old v0.27.1-extras* / v0.28.0-extras* Hub tags:

• Source of truth: opengfx1030/vllm-rdna rdna_extras is the v0.31.0 line (#announcements 11 Oct 2026). The old 0.28 aggregation lives at rdna_extra/v0.28.0. Hub blivioniag/vllm-rdna:v0.31.0-extras (ROCm 7.14) served Flash-Next MTP-2 (BF16 + INT4-disk).
• In-tree launcher: tools/rdna/serve_rdna.sh RECIPE=flashnext (production tools/reasoning) or RECIPE=flashnext-mtp2 / flashnext-mtp0. The old tools/rdna/serve_gfx1030_flashnext.sh name is a thin alias of RECIPE=flashnext. #vllm-rdna Sep 21 pointed 2-card / GPTQ questions at that file. Do not copy host paths out of the tree. Prefer the recipe over agent-generated env dumps — drop VLLM_USE_BREAKABLE_CUDAGRAPH=1 and do not force VLLM_RDNA_FUSED_HC=0 on Flash-Next (#vllm-rdna Oct 10).
• Published container: prefer Hub v0.31.0-extras. The older leapdragon/vllm-rdna2-qwen tree remains for legacy GHCR / docs/rdna2 (occasional commits through 11 Oct 2026).
• Weights: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16. #vllm-rdna (Oct 11): cyankiwi/Qwen3.8-Flash-Next-AWQ-INT4 produced incoherent output on one 4× host; switching to the wtdcode AWQ + primitive-ai PLE pair is the documented path.
• PLE / n-gram: launcher PLE=bf16|int4|int4-disk + PLE_QUANT_DIR=…/ples_int4 (primitive-ai/Qwen3.8-Flash-Next-PLE-quant). #announcements (11 Oct): all three modes tested; speed at parity. Docker disk mode: -v /path/to/ples_int4:/ples_int4 -e VLLM_PLE_QUANT_DIR=/ples_int4 -e VLLM_PLE_CPU_OFFLOAD=1. No-P2P hosts: P2P=0 on the launcher (or VLLM_RDNA_AR=0).

Expect large host DRAM for the BF16 n-gram table (~95 GB class). INT4 pinned is a ~30 GB table; #vllm-rdna (Oct 11) corrected an inflated ~90 GB host-RAM figure — INT4 sidecar working set is closer to ~35 GB. int4-disk mmaps shards (MADV_RANDOM) so only the gather/dequant working set hits RAM. That is weight offload, not KV. Native KV RAM overflow was broken on one 4× host (Sep 14: 128 GB not enough); the fix merged as #24 (27 Sep) — KV offload. Long-prompt / intermittent stalls: try VLLM_USE_V2_MODEL_RUNNER=0 — troubleshooting. Throughput and KV tightness: Flash-Next status.

#vllm-rdna (Sep 17): there is no Q3 / GGUF-Q3 path on vLLM. Stay on W4A16 / AWQ / AutoRound. EXL3 ~3 bpw is still experimental — Quantization.

4× V620 Flash-Next serve (rdna_extras #17, Sep 23)

opengfx1030/vllm-rdna#17 merged 23 Sep 2026 into rdna_extras. This is the public landing of the unpublished ~1950 PP recovery after #15. Community / Needs verify. Not a Hub tag. Do not copy host paths from the PR body — use the in-tree docs (docs/rdna2/V620-GIT-FAST-SERVICE.md, docs/rdna2/V620-BASELINE-PORT-20260922.md) and tunableop/.

Operational facts that are safe to copy:

• Resident W4A16 MoE (VLLM_RDNA_MOE_RESIDENT=1): convert the checkpoint’s packed INT4 experts into the RDNA2 kernel layout once at load, keep that layout, and skip repacking every 4k prefill chunk. Precision stays W4A16. #vllm-rdna (Sep 29): a latest-rdna_extras host saw PP fall from ~2k → ~1450 with this unset; setting 1 restored ~2k. Treat a sudden PP cliff after an extras pull as a missing-resident-MoE check before blaming kernels.
• Mamba retirement: backport of vllm#55450. Without it, long prompts (100k+ class) hit a deterministic preemption loop (state blocks not retired across null gaps).
• Graphs (#17 measured): --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[3,6,12]}' with MTP-2. Prefill stays eager.
• On current HEAD, prefer FULL_AND_PIECEWISE instead — see FULL_AND_PIECEWISE (decode uses the same FULL path as this baseline; piecewise covers mixed/prefill).
• TunableOp is rocBLAS-hash locked. A mismatched CSV aborts the first GEMM or silently uses default algorithms — regenerate.
• Harness used for the PR numbers: GeorgeMA-Strong/llm-context-bench.

PR-head 16k (author) vs second-host confirm (#vllm-rdna Sep 23):

4× V620 Flash-Next serve (FULL_AND_PIECEWISE, Sep 24–25)

Current recommended graphs on latest rdna_extras. #vllm-rdna (Sep 24): maintainer confirmed full + piecewise is fixed. Fork-source write-up: docs/rdna2/V620-FULL-AND-PIECEWISE.md.

What changed:

Safe compile line (generalized; do not copy host paths):

--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[3,6,12]}'

In-tree launcher: tools/rdna2/serve_v620_piecewise.sh (V620_COMPILE_MODE=3, V620_CUDAGRAPH_MODE=FULL_AND_PIECEWISE). Older comments in that script still mention the #20-era ~2× decode drop — ignore those once you are on HEAD with the keep-FULL commit. Prefer this over bare PIECEWISE (no FULL graphs) and over the Sep 17 breakable/mode:0 workaround.

Without compilation / without VLLM_USE_BREAKABLE_CUDAGRAPH=1, a FULL_AND_PIECEWISE request downgrades to FULL_DECODE_ONLY (keeps FULL decode) instead of dropping every graph; a bare PIECEWISE request still becomes NONE.

rdna_extras HEAD (25 Sep, 700753d9): MTP-2 boots after the amdsmi / local-argmax draft fix, but author-host decode stays below MTP-0 (~46–49 vs ~61 t/s). Do not expect MTP to beat the no-draft line yet — MTP.

The older Sep 17 PIECEWISE host-venv block below is only for checkouts before #17 / the keep-FULL fix.

4× V620 Flash-Next MonoKernel (#94, 16 Oct)

Merged opengfx1030/vllm-rdna#94 (16 Oct) into rdna_extras. A MonoKernel is a persistent HIP kernel that runs a large part of one Flash-Next (qwen4_exp) decoder layer in one launch (k1 hyper-connection preamble, GDN/QSA decode cores, k2 FFN). Prefill, mixed batches, the MTP drafter, and TP all-reduce stay on today’s path. Off by default. Hub v0.31.0-extras lags. Recipes do not pass the flag yet.

Startup refuses --monokernel unless all of these hold: gfx10x GPU, Qwen4Exp (MTP drafter accepted), V2 model runner, PP=1, _rocm_C ops built, and MoE on symmetric W4A16 (CompressedTensorsWNA16RDNA2MoEMethod). Asymmetric experts are a follow-up (open #115). PP3 MonoKernel is open backlog (#116) — do not combine this switch with RECIPE=flashnext-pp3*.

# Build the in-tree HIP bits (from an rdna_extras checkout + venv):
bash tools/rdna/mono/build_tree.sh

MODEL=/path/to/flash-next VENV=/path/to/venv \
  EXTRA_ARGS="--monokernel" \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

Same switch: VLLM_ROCM_MONO_DECODE=1 (alias of --monokernel; backport of vllm#61082). A/B without the flag in the same session. Keep GPU_MAX_HW_QUEUES=4 (Flash-Next TP4 recipes already set it — #107); two queues share HIP queues and skew ranks. In-tree notes: tools/rdna/mono/README.md and docs/rdna/monokernel/. Reproduce A/B write-up: open #120.

PR #94 same-session A/B (4× V620, TP4, PLE int4, FULL_AND_PIECEWISE, temperature 0, HSA_FORCE_FINE_GRAIN_PCIE=1, GPU_MAX_HW_QUEUES=4) — Fork-source. Cells are prefill tok/s/req / decode tok/s/req / TTFT s / median ITL ms. For MTP-2, prefer ITL (decode tok/s moves with drafter acceptance):

#vllm-rdna (16 Oct) author restated the decode-side win as about 8–10% ITL at c=1 (with or without MTP), 5–6% at c=8 without MTP, and 0–3% at c=8 with MTP-2. Prefill / TTFT were same or better in the PR table. Serves passed probe / PPL / prefix / concurrency gates.

Community / Needs verify (#vllm-rdna 16–17 Oct): a second 4× host called MonoKernel slightly slower than the current HIP path, with one GPU at 100% VRAM and prefill under 1200 tok/s on the same cells — TunableOp mismatch suspected; author-host prefill stayed above 2.1k. A mixed RDNA2/RDNA3 host saw about 5–10% after adding the switch (not a V620 recipe). If PP collapses after --monokernel, regenerate TunableOp before blaming the kernel — TunableOp.

Merged #126 (18 Oct) moves the MonoKernel QSA indexer (store / epilogue / scoring) to HIP. That is on with --monokernel on HEAD; it does not change the default path by itself.

HEAD HIP defaults (#130, 19 Oct)

Merged #130 (19 Oct) into rdna_extras (closes #99). Flash-Next recipes on HEAD now default the HIP set on: VLLM_RDNA_ALL_HIP, VLLM_RDNA_QSA_HIP, VLLM_RDNA_HC_PREFILL_HIP, and VLLM_GDN_HIP_PREFILL=1. EXL3 recipes are unchanged. Hub v0.31.0-extras lags. Set those vars to 0 to get the older Triton paths back.

Author-host 4× TP4 (Flash-Next AWQ, PLE int4): HIP-on ITL −6 to −8% and prefill +3–7% vs the previous default (no --monokernel). With --monokernel, MTP-2 ITL was −2 to −4%. Clone HEAD — do not expect this from the 11 Oct Hub bake.

Do not pass num_speculative_tokens_per_batch_size (merged #127 opt-in). #vllm-rdna (22 Oct): that schedule caused token-repeat loops. Fixed MTP-2 (RECIPE=flashnext-mtp2) stays the daily path. #vllm-rdna (25–26 Oct): loops can survive MTP off if CPU / disk KV offload is restoring sessions — disable --kv-offloading-size while isolating — troubleshooting.

4× V620 Flash-Next DFlash2 (#132, 21 Oct)

Merged #132 (21 Oct). Opt-in drafter path, not a replacement for MTP-2. Recipe enables --monokernel.

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-dflash2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

• Target: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16.
• Draft: incoai/Qwen3.8-Flash-Next-DFlash2 (BF16 — FP16 draft produced NaNs / near-zero acceptance).
• Draft attention is FA-RDNA2 BLHNC (noncausal sliding window). Target stays HIP QSA.
• Default 3 draft tokens ([4,8,16,32] capture). For the measured 16k / 1k c=8 ladder: DFLASH_TOKENS=2 ([3,6,12,24]). PRINT=1 shows the argv before launch.

PR paired A/B (4× V620, TP4, same boot): native MTP-2 faster in all four cells (tuned DFlash2 aggregate output −29 / −12 / −24 / −7% at 1k c=1 / c=8 and 16k c=1 / c=8). #vllm-rdna (22 Oct): maintainer still has MTP-2 ahead on agentic load; community: DFlash2 can win short / single-user, MTP-2 under concurrency. Fork-source for the recipe; community contrast Needs verify. Not Hub.

2× V620 Flash-Next MoE offload (#146, 28 Oct)

Merged #146 (28 Oct). Resident Flash-Next experts still do not fit 2× V620. This is the first in-tree vLLM spill path: immutable pinned-RAM expert banks, a bounded GPU slot cache, and segmented FULL_AND_PIECEWISE graphs around host planning. #vllm-rdna (28 Oct): V1 landed.

This is not a serve_rdna.sh RECIPE= yet. Use the offload wrapper on HEAD:

PRINT=1 MODEL=/path/to/flash-next PLE_QUANT_DIR=/path/to/ples_int4 \
  bash tools/rdna/mono/serve_offload.sh

• Target: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16 (compressed-tensors symmetric INT4, group 128).
• Default layout: TP=2 / EP=2, --monokernel, 96 expert slots per layer/rank, native CPU KV tier (KV_OFFLOAD_GB; 0 disables KV offload for an expert-only A/B).
• Slot policy knobs: SLOTS, POLICY (lfu_lru or lru), DECAY.
• Offload launcher defaults VLLM_RDNA2_W4A16_DETERMINISTIC_DENSE=1 (Q32 dense accumulation). Resident 4× recipes keep their existing default.
• Design notes: docs/rdna2/moe-offloading-target.md. Do not copy host paths out of that file.

Author-host 2× V620, MTP-2, 16 GiB native CPU KV (PR + #vllm-rdna 28 Oct): 16k/1024 c=1 ~19.8 tok/s aggregate (TTFT ~17 s, ~30 decode); c=8 ~26.7 agg (mean TTFT ~106 s, KV pool at 100% observed occupancy); 32k/256 c=1 ~30 decode / TTFT ~34 s. Fork-source on that host. The PR does not claim a controlled resident-vs-offload speedup. Not Hub. Needs verify on a second host.

Without HEAD, llama.cpp RAM/MoE offload is still the 2-card path — llama.cpp 2×.

INT4 KV cache (#138, 25 Oct)

Merged #138 (25 Oct). Opt-in longer KV on the non-MonoKernel QSA path:

--kv-cache-dtype int4_fp16_block

Packed INT4 + FP16 per-32-channel block scales; native QSA dequant in registers. Other attention backends refuse the dtype. Default FP16 KV unchanged. Not Hub. MonoKernel + this dtype Needs verify. #vllm-rdna (21 Oct): INT8 KV restore (draft #71) remains the stated day-to-day priority. Details: Quantization, Fork progress.

4× V620 Flash-Next serve (rdna_extras, PIECEWISE, Sep 17)

Sanitized from a #vllm-rdna (Sep 17) host-venv recipe on 4× V620. Community / Needs verify — not a published Hub tag. Requires opengfx1030/vllm-rdna rdna_extras including the GDN sanitizer commit 388a61b6f, and a rocm_sdk venv (torch 2.12 + ROCm 7.14). Paths below are generalized.

Why this differs from the Hub -extras TP4 block: Flash-Next on current rdna_extras FULL decode graphs still corrupt output. Community: switch --compilation-config to PIECEWISE, keep --max-num-batched-tokens 2048, and cap --max-num-seqs at 4–6. See FULL-graph corruption. Do not copy this graph mode onto the 27B -extras recipe unless you are A/B testing.

Community snapshot on that host (16k prompt / 1k gen, concurrency 8): ~3331 tok/s prefill, ~72.7 tok/s decode, TTFT ~39 s. Treat as a single-host number.

#vllm-rdna (Sep 17–18): pin rdna_extras at or after e45dd5cb if you want prefix cache to actually hit, and set VLLM_RDNA_DENSE_GEMV=1 if skinny wvSplitK faults after the profile run. VLLM_USE_BREAKABLE_CUDAGRAPH=1 is the stable / no-compile path (~39 t/s class in one A/B); 0 plus compile can be faster once graphs are clean — see FULL-graph troubleshooting. Do not trust the logged hybrid KV token count — overstated pool.

#vllm-rdna (Sep 18): maintainer posted an updated host-venv production block that still keeps VLLM_USE_V2_MODEL_RUNNER=0 (V2 blocked on Qwen4Exp), VLLM_USE_BREAKABLE_CUDAGRAPH=1, and --max-num-batched-tokens 2048, but:

• uses --compilation-config FULL_AND_PIECEWISE. On Sep 18 trees that mode still executed as PIECEWISE on ROCm (rocm_full_executes_as_piecewise). On current HEAD the keep-FULL fix turns that redirect off — FULL_AND_PIECEWISE. Still do not use FULL-only on old trees — FULL-graph corruption
• raises --kv-cache-memory-bytes to 7000000000 (7 GiB) and --gpu-memory-utilization 0.90
• reported a 16× 16k concurrency test as fine (earlier --max-num-seqs tightness)
• kills leftover processes before relaunch — VLLM::Worker, VLLM::EngineCore, and PleOffloadWorker can survive an API/port kill (troubleshooting)
• custom all-reduce / PIX is optional — hosts without GPU↔GPU P2P should drop that block (vLLM falls back to PYNCCL) — no P2P

Prefer Hub v0.31.0-extras (or a matching host-venv clone of rdna_extras) for this serve line. Hub v0.28.0-extras is only for historical A/B against older Sep 2026 pastes.

MODEL="wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16"
export VLLM_PLE_QUANT_DIR="$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('primitive-ai/Qwen3.8-Flash-Next-PLE-quant'))")/ples_int4"

export VLLM_PLE_CPU_OFFLOAD=1
export VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600
# Low-RAM hosts only: PLE_OFFLOAD_PREFAULT=0  (skips RAM prefault; ~25% slower — see env-vars)
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
export VLLM_FORCE_CUSTOM_ALL_REDUCE=1   # only if GPU↔GPU P2P works
export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_USE_AOT_COMPILE=0
export VLLM_DISABLE_COMPILE_CACHE=1
export VLLM_USE_BREAKABLE_CUDAGRAPH=1   # compile off; A/B 0 after output is clean
export VLLM_RDNA_DENSE_GEMV=1           # if wvSplitK faults after profile
export VLLM_RDNA_FUSED_HC=0
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export VLLM_RDNA_FORCE_FP16=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_ENABLED=1
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export PYTORCH_TUNABLEOP_FILENAME="${PYTORCH_TUNABLEOP_FILENAME:-/path/to/tunableop_results.csv}"
export VLLM_BATCH_INVARIANT=0
export GPU_MAX_HW_QUEUES=2
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_P2P_LEVEL=pix
export RCCL_P2P_NET_DISABLE=1
export RCCL_P2P_BATCH_ENABLE=1
export NCCL_PROTO=Simple
export RCCL_MSCCL_ENABLE=0
export HSA_FORCE_FINE_GRAIN_PCIE=1

cd /tmp   # avoid sys.path collisions with a local vllm checkout
python -m vllm.entrypoints.cli.main serve "$MODEL" \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --max-num-seqs 6 \
  --max-num-batched-tokens 2048 \
  --kv-cache-memory-bytes 5000000000 \
  --gpu-memory-utilization 0.88 \
  --dtype float16 \
  --trust-remote-code \
  --enable-prefix-caching \
  --language-model-only \
  --skip-mm-profiling \
  --enable-expert-parallel \
  --block-size 16 \
  --distributed-timeout-seconds 1800 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE","compile_ranges_endpoints":[]}'

Shared P2P / all-reduce env is the same as Hub -extras TP4. Drop the custom-AR block if P2P is broken.

Concurrency vs batch: #vllm-rdna (Sep 16): --max-num-batched-tokens 2048 keeps prefill turns ~1 s at ~1300 tok/s; 8192 can block other streams for ~6 s. Raise toward 4096 only if you hit the 128k prefill cliff on Intel AutoRound — that 4096 knob and this PIECEWISE stability recipe are different bugs.

Vision (#vllm-rdna Sep 17–21 + the in-tree launcher): Flash-Next recipes pass the vision-cap feature, which still applies a pixel cap. --limit-mm-per-prompt '{"image":1}' alone is not enough (count is already 1). Without max_pixels, mm-profiling can feed a huge dummy image (~24.8M px) and the vision encoder SDPA math backend materializes a ~64 GiB L×L fp32 score matrix — startup OOM on 32 GB cards. max_pixels=1605632 keeps images up to about 1424×1424. serve_rdna.sh bakes those two flags into vision-cap.

--limit-mm-per-prompt '{"image":1}' \
--mm-processor-kwargs '{"max_pixels":1605632}'

#vllm-rdna (Sep 21): vision works on rdna_extras with those flags; community ~15–20 s per image. Runtime is a second OOM: vLLM may not reserve vision memory, so a tight KV plus an image can OOM after a clean start. #general (Sep 21): vision + PP3 is especially tight. Keep --language-model-only --skip-mm-profiling if you do not need images.

Maintainer QA playbook (not a second wiki): BlivionIaG/vllm-rdna-qa.

3× V620 Flash-Next PP3 (rdna_extras HEAD)

#general (13 Oct 2026) + merged #96: PP3 / MTP-2 is in-tree on current rdna_extras. Hub v0.31.0-extras (11 Oct bake) does not include these recipes — clone HEAD + host venv + serve scripts.

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-pp3 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

MTP-2 needs a W4A16 quantized drafter (tools/rdna/quant_mtp_experts.py); bf16 draft experts do not fit the last stage:

MODEL=/path/to/mtpq-model VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-pp3-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

Recipe facts (fork comments + #general 13 Oct maintainer paste):

#vllm-rdna (17 Oct): later HEAD merges were not re-benched against this 13 Oct table — treat the numbers as a snapshot, not a current HEAD guarantee.

Author-host 16k/1k (#general 13 Oct, Community — not wiki-tested):

Also merged #95: int8 dense shadows are built in row chunks so load no longer allocates a ~2.4 GiB fp32 temporary. PLE still lives on stage 0 — PLE=bf16 wants ~95 GB host RAM; smaller hosts use PLE=int4 / int4-disk.

Older Sep 15 leapdragon / overlay notes stay below as historical. If an older pin still corrupts PP3 decode — PP3 corruption. Sep 19 pin b33f9b6 is a different failure (graphs boot, small prefill KeyError) — workaround.

3× V620 Flash-Next PP3 + MTP (leapdragon era, Needs verify)

#vllm-rdna (Sep 15): one host fitted Qwen3.8-Flash-Next + MTP on 3× V620 32 GB using a leapdragon image (W4A16 + PLE int4 CPU offload), PP=3 / TP=1. Treat as a single-host write-up — some steps were local source patches. Prefer the HEAD recipes when you can clone current rdna_extras.

Operational facts from that era that still match the HEAD recipes:

• Baseline PP3 without MTP sat at ~30 GiB/card with --gpu-memory-utilization 0.95.
• The MTP head is instantiated on the last pipeline stage only, so that stage OOMs first.
• Uneven layers leave the last stage light. The PLE offload worker can inherit VLLM_PP_LAYER_PARTITION, see pp_size=1, and refuse (len(partitions)=3 ≠ pp_size=1) — unset that env in the PLE worker process.
• VLLM_ROCM_MOE_PADDING=0 (do not pad routed-expert weights).
• Upstream vllm#46994 (MTP under pipeline parallel on the V2 runner) merged Sep 2026.

The old local int8 row-chunk rewrite is now #95 on HEAD. VLLM_RDNA_DENSE_INT8_ONLY=1 is on both flashnext-pp3* recipes.

3× overlay: leapdragon + org MoE HIP (Sep 17–18)

#vllm-rdna (Sep 17–18): a temporary public overlay (alanoo81/flashnext-v620-pp3) ports opengfx1030’s fused W4A16 MoE HIP kernel (moe_gptq_gemm_rdna2) onto a leapdragon Flash-Next image so 3× V620 / PP=3 can keep leapdragon CUDA graphs + MTP while picking up org-class prefill. Not a published Hub tag; treat as Community / Needs verify.

Public README snapshot (same harness; 160 W cap; AWQ-W4A16 + int4 PLE in host RAM):

Community: use MTP for one user; from two streams up, leave MTP off (plain graphs scaled better in that write-up). Also backports vLLM #46994 and #54044 (MTP + graphs + prefix cache). Follow the overlay README — do not copy host paths from Discord.

#general (Sep 23): the overlay author is no longer working on the repo. Community still reports useful 3-card numbers (one host ~45 t/s decode / ~1100 tok/s PP; overlay write-up 120+ t/s under multi-agent load, even ~200 t/s with 5–6 agents). Long context can trip infinite KV-pool dump loops — the author capped 3 concurrent agents on long prompts. Treat the overlay as unmaintained / snapshot; prefer rdna_extras once PP3 is clean there.

#vllm-rdna (Sep 19): that same HIP MoE kernel was reported non-deterministic (~3.5% of top tokens change between identical runs; Triton MoE did not). #vllm-rdna (Sep 30): an fp32 deterministic path was described as opt-in (throughput cost); no public flag name. See fork MoE note. Needs verify.

#vllm-rdna (Sep 18): the published Flash-Next checkpoint can leave the MTP draft head’s 512 experts in bf16, so speculative decode runs those through the generic Triton MoE kernel. The overlay README adds scripts/quant_mtp_experts.py (offline W4A16, RTN int4 g128, same packing as the main experts) so the draft experts hit the same HIP kernel. Community on 3× V620 PP3: +4–7% single-stream decode (~65–66 t/s), no acceptance loss (~70→75%), and +60% wall throughput at 8 streams (~77 → ~123 t/s). Community / Needs verify — run the script from the overlay repo; do not invent a Hub tag for the derived checkpoint.

DeepSeek-V4 Flash (community, Needs verify)

#vllm-rdna (Sep 17): public INT4-W4A16 weights yiminyuan/DeepSeek-V4-Flash-0731-INT4-W4A16 plus a gfx1030 vLLM tree yiminyuan/vllm @ gfx1030/v0.28.0. Publisher intent: TP=4 / PP=2. This is not Hub -extras and not wiki-tested. #llamacpp also mentioned a RDNA2 TP4 DeepSeek-V4 serve (~22 t/s, kernel unpublished) — Needs verify.

#vllm-rdna (Oct 10): on the current extras / v0.31 test line, DeepSeek-V4 is expected to run once mixed prefill/decode stalls are fixed (suspected CUDA-graph capture range too narrow). That is not a serve recipe yet — stay on the yiminyuan tree or wait for the stall fix. GLM-5.3 on the same fork was asked and not answered. See mixed prefill.

Related

• Flash-Next status — throughput, KV, AutoRound
• Serve scripts — RECIPE=flashnext-mtp2 and the other in-tree configs
• Hub -extras TP4 — the non-Flash-Next 27B block
• Configuration
• vLLM forks