Introduction

Work in progress: This wiki is being actively expanded from Discord research and fork release notes. Many tuning recipes and throughput numbers are community-reported and have not been independently reproduced here. See Verification status for what is solid vs what still needs checking on your hardware.

Welcome to the GFX1030 Wiki — a focused, hands-on knowledge base for running LLM inference on AMD gfx1030 (RDNA2 / Navi 21) GPUs, with a strong bias toward the Radeon PRO V620 and the purpose-built tooling collected here.

gfx1030 is the LLVM/ROCm target for the Navi 21 “Sienna Cichlid” die. It powers the consumer Radeon RX 6800 / 6800 XT / 6900 XT / 6950 XT and the workstation/data-center PRO W6800 and PRO V620. These cards are officially supported by ROCm on Linux, which makes them a cost-effective platform for modern LLMs — but getting the most out of them (power tuning, multi-GPU P2P, and RDNA-tuned kernels) takes a bit of extra work. That’s what this wiki documents.

Read it in this order

1. Getting started — hardware check, ROCm, optional V620 tuning, first container.
2. llama.cpp or vLLM — which stack matches the model and card count.
3. The matching section: vLLM onboard (V620) / vLLM overview, or llama.cpp.
4. Troubleshooting when a command fails. Verification status says which numbers are solid.

Host firmware (the machine never POSTs) and Flash-Next serve lines are specialist pages. Skip them until you need them.

What this wiki focuses on

• Tuning — the v620_toolbox recipes: lowering the V620’s VBIOS-locked 250 W floor to 120 W, enabling GPU↔GPU PCIe Peer-to-Peer between multiple V620s (Fedora + AMD hosts; power tuning also on Ubuntu 26.04; Intel Ice Lake often sees no P2P win — A/B it), and optionally disabling Pro-card ECC for ~2 GB extra VRAM.
• llama.cpp — stock builds plus the RDNA2 fork (tensor parallel, MTP / DFlash). Usually the easiest first stack on V620.
• vLLM on RDNA — ready-to-run Docker images; recipes for Hub vs recipe container, and a separate Flash-Next page (blivioniag/vllm-rdna on a blivioniag/rocm-rdna PyTorch base), how they are built with opengfx1030/vllm-rdna-docker, and the official vLLM extras fork opengfx1030/vllm-rdna (rdna_extras, now including Flash-Next cherry-picks). Hub v0.31.0-extras tracks the 11 Oct 2026 bake pin; commits after that pin still want a host venv clone.

New here? Start with Getting Started. Card identity is on Supported Hardware.

Community Discord

Much of this wiki is distilled from the gfx1030 club Discord — live tuning tips, fork releases, and multi-GPU benches land there first.

• Server: gfx1030 club
• Join: https://discord.gg/mESex2aBp
• Useful channels: #vllm-rdna, #llamacpp, #general, #benchmarks, #harnesses, #lmcache, #motherboard

When turning Discord threads into docs, follow the privacy rules in Contributing.

Disclaimer: This is a community wiki, not affiliated with or endorsed by AMD. Kernel patches and power-cap changes are done at your own risk. Always cross-check against the official ROCm documentation. Contributions welcome — see Contributing.

llama.cpp or vLLM

Pick a stack before you follow a recipe. Both run on gfx1030; they are good at different jobs. Numbers below are community snapshots — see Verification status.

Which stack

New to the cards? Start with RDNA2 serving, then vLLM onboard when you need concurrency. Multi-GPU tensor parallel benefits from PCIe P2P.

What fits well on V620

#general (Sep 2026) consensus — RDNA2 has no matrix / tensor cores, so prefill on dense models is the weak spot (agentic “read a pile of files” workloads frustrate people even when decode looks fine):

Next

• Getting started
• vLLM recipes — Hub -extras vs recipe container
• llama.cpp overview

Getting Started

This page gets you from a bare Linux install to a working ROCm + PyTorch stack on a gfx1030 card, and explains how to preview this wiki locally.

1. Confirm your hardware

Make sure your card is a Navi 21 gfx1030 (or a related RDNA2 card):

lspci | grep -i vga
# After ROCm is installed:
rocminfo | grep -m1 -o 'gfx[0-9]*'

See Supported Hardware for the full card list. If the box never POSTs with several V620s (especially behind a PLX switch), read Host firmware before chasing ROCm.

2. Install ROCm

Follow Installing ROCm. At a high level:

# Add the amdgpu repo, then:
sudo apt install rocm
sudo usermod -aG render,video "$LOGNAME"
# Reboot, then verify:
rocminfo
clinfo | grep -i 'gfx\|Board'

3. (Optional) Tune the card

If you run one or more Radeon PRO V620 (or PRO W6800) cards, these tweaks are worth doing before you load models (Fedora or Ubuntu 26.04 for power cap; P2P is Fedora + AMD-validated — A/B on Intel, see When P2P helps):

• Power Tuning — drop the VBIOS-locked 250 W floor to 120 W and boot-cap at 180 W. Passthrough VM and need OD clocks instead? See Soft unlock (amd-v620-soft-unlock).
• Disabling ECC — Pro cards hide ~2 GB behind ECC; optional extra VRAM.
• Multi-GPU PCIe P2P — enable GPU↔GPU peer-to-peer for multi-card setups.

Baremetal power floor + P2P come from the v620_toolbox repo.

4. Run an inference stack (Docker)

The fastest path is the prebuilt images — no local ROCm/PyTorch/vLLM build required:

• Onboard vLLM on V620 — short happy path for Hub v0.31.0-extras or the fork launcher (serve_rdna.sh).
• Running vLLM (Docker) — blivioniag/vllm-rdna on a blivioniag/rocm-rdna PyTorch base.
• vLLM recipes — Hub -extras vs recipe container vs Flash-Next by card count.

Prefer GGUF and building from source? See Building & Running llama.cpp (ROCm or Vulkan).

Want to build the images yourself, or use the RDNA-tuned kernels? See Building the Images and The rdna_extras fork.

Need the same stack in a host uv venv instead of Docker? See Host venv (ROCm 7.2.0 index wheels, or 7.14.0 multi-arch wheels).

5. Smoke test

# Inside a rocm-rdna / vllm-rdna container, or a local ROCm PyTorch env:
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

You should see True and your Radeon card’s name. If not, see Troubleshooting.

6. Join the community Discord

Questions, live benches, and fork updates land first on gfx1030 club: https://discord.gg/mESex2aBp (#vllm-rdna, #llamacpp, #general, #benchmarks). See Useful resources.

Editors: local preview and page conventions are in Contributing.

Supported Hardware

gfx1030 is the LLVM target name for AMD’s Navi 21 GPU (RDNA2 generation, codename Sienna Cichlid). Several retail and workstation cards use this die and therefore report themselves as gfx1030.

Cards that natively report as gfx1030

Pro cards hide ~2 GB behind ECC. See Disabling ECC if you want the full 32 GB.

#general (Sep 15 2026): versus a 32 GB MI50 (gfx906), community first-pass on V620 was more prefill, less decode — PP “almost 2×”, TG behind but not far. That is a community A/B, not a wiki bench. Sibling notes: wiki-gfx906.

#general (Sep 17 2026): do not tensor-parallel a V620 with an NVIDIA card (community example: RTX 5090 + V620). Opposite strengths (RDNA2 has no matrix cores; the NVIDIA card has little VRAM per dollar) — mixed decode gains were small and prefill collapsed. Prefer a homogeneous V620 (or W6800) box, or run the NVIDIA card standalone.

Do not flash W6800 VBIOS onto a V620. #general (Sep 2026): pre-modded eBay cards and DIY flashes that load a W6800 BIOS on a V620 drop active CUs from 72 → 54 (W6800’s different CU layout). That is a large compute hit — stay on stock V620 BIOS. Cooling shroud mods are fine; BIOS swaps are not a free “Pro unlock.”

Need OverDrive / unlocked clocks in a passthrough VM without flashing? Prefer the reversible 4-byte PowerPlay soft unlock (Tamalero/amd-v620-soft-unlock) over a W6800 flash — see Power tuning → Soft unlock.

All of these are on the officially supported list for recent ROCm releases on Linux.

AMD BC-250 (gfx1013)

Headless mining APU board (Cyan Skillfish). RDNA2, reports as gfx1013 — not Navi 21 / gfx1030. Community ROCm and Vulkan work exists; most of this wiki still targets Navi 21.

Community references: AMD BC-250 docs, akandr/bc250-rocm.

RDNA2 relatives that can run gfx1030 code

The rest of the RDNA2 line uses a different LLVM target but shares the same ISA family. They are not on the official support matrix, but in practice they run gfx1030 kernels once you set HSA_OVERRIDE_GFX_VERSION=10.3.0 (see HSA_OVERRIDE for RDNA2 Cousins).

Architecture highlights (Navi 21)

• RDNA2 compute units with a native wavefront size of 32 (wave32), unlike GCN’s wave64.
• Infinity Cache (128 MB on Navi 21) that dramatically raises effective memory bandwidth.
• No dedicated matrix/tensor cores — RDNA2 predates the WMMA/matrix instructions added in RDNA3 (gfx11xx). Matrix math runs on the regular vector ALUs, so expect lower peak throughput than RDNA3 or CDNA cards, but very good price/performance for inference.
• FP16 is well supported; BF16 has limited/emulated support and is best avoided for hot paths (prefer float16). #general (Sep 28 2026): ComfyUI / image-video workflows often default to bf16. On V620 that path is HIP-emulated, so those pipelines are a poor fit — pick fp16 / int8 workflows, or use a card with native bf16. Community also called video generation a CUDA-world problem, not a V620 win. #general (30 Oct): training on V620 is the same constraint — prefer FP32 (or a native-bf16 card); community bf16-emulation tests had drawbacks.

How to check your GPU target

rocminfo | grep -i 'gfx\|Name'
# or, more directly:
rocminfo | grep -m1 -o 'gfx[0-9]*'

If the output shows gfx1030, everything in this wiki applies directly. If it shows gfx1031, gfx1032, etc., head to HSA_OVERRIDE for RDNA2 Cousins first.

Host firmware (large BAR / no POST)

4× V620 behind a PLX switch can fail to POST on older workstation BIOS (community: Dell Precision T5820 / T7820 / T7920) even when the same kit boots on a modern desktop board. That is firmware MMIO High / SR-IOV work — see Host firmware — not a ROCm bug.

Installing ROCm

gfx1030 (Navi 21) is on AMD’s officially supported list for recent ROCm releases on Linux, so in most cases a stock install “just works” — no override hacks required.

Always follow the official ROCm install guide for your exact distro and ROCm version. The commands below are a convenience summary and may drift over time.

Supported operating systems

ROCm’s RDNA2 support targets specific LTS releases. As of recent ROCm versions that typically means:

• Ubuntu 22.04 / 24.04 LTS
• RHEL / Rocky 9.x
• Debian 12 (community, less tested)
• Fedora 43 (wiki-validated for power / P2P); Fedora Server 44 + kernel 6.19 reported working for the V620 powerfix (community)

Check the system requirements page for the version you plan to install.

Ubuntu quick install (amdgpu-install)

# 1. Add the amdgpu package repository (replace VERSION with the ROCm release you want, e.g. 6.4.60400-1)
sudo apt update
wget https://repo.radeon.com/amdgpu-install/latest/ubuntu/jammy/amdgpu-install_VERSION_all.deb
sudo apt install ./amdgpu-install_VERSION_all.deb
sudo apt update

# 2. Install ROCm (compute use case)
sudo amdgpu-install --usecase=rocm

# 3. Add yourself to the render/video groups so you can access the GPU without root
sudo usermod -aG render,video "$LOGNAME"

# 4. Reboot
sudo reboot

Verify the install

rocminfo                       # should list your Navi 21 card and "Name: gfx1030"
clinfo | grep -i 'Board\|gfx'  # OpenCL view
rocm-smi                       # live clocks, temps, VRAM, power

If rocminfo shows Name: gfx1030, you’re done. If it shows gfx1031/gfx1032/etc., your card is a smaller RDNA2 die — continue to HSA_OVERRIDE for RDNA2 Cousins.

Multi-GPU: pin ROCm 7.2.0 or 7.14.0

For more than one card, stay on ROCm 7.2.0 (not 7.2.1+) or jump to 7.14.0. #vllm-rdna / #llamacpp report an RCCL bug from 7.2.1 upward (including mid-line builds such as 7.2.4) that shows up as soon as you leave a single GPU — tensor-parallel hangs, comm failures, AllReduce init failed, or cards dropping offline. The published vllm-rdna images already sit on those two bases for that reason.

Do not “upgrade within 7.2.x” on a multi-GPU box. If you are already on a broken 7.2.1–7.13 userspace, rebuild or pull a 7.2.0 or 7.14.0 image rather than debugging RCCL on the in-between releases. Ubuntu 26.04 can ship ROCm/HIP 7.1 (#llamacpp Sep 17) — treat that as an A/B, not the multi-GPU pin. ROCm 10 / TheRock packaging is interesting for newer cards; community llama.cpp benches on Ice Lake have run ROCm 10.0 with numbers close to 7.1 on that host. #vllm-rdna (29 Oct): a fresh Ubuntu 26.04 install can land /opt/rocm/core-10 (and core-10.1) with no core-7.14 folder — do not copy a 7.14 ROCM_HOME from host venv onto that tree. Same thread: ROCm 10 serves on at least one host; the Triton wheel for that userspace is unconfirmed. #vllm-rdna (30 Oct): maintainer called ROCm 10 and 7.14 the same for day-to-day serving; another host is already serving on 10 with no issues so far. That is not a reason to wipe a working 7.14 box. One 10.1 host-venv attempt failed before a clean 7.14 retry — Needs verify, not a 10.1 ban. Day-to-day gfx1030 multi-GPU advice is still: prefer 7.14.0 (or 7.2.0) unless you are deliberately lab-testing 10.x.

Notes & gotchas

• Kernel driver: ROCm relies on the amdgpu kernel module. Very new kernels sometimes ship a driver newer than your ROCm userspace expects; the amdgpu-dkms package from the amdgpu repo keeps them in sync.
• Secure Boot: if Secure Boot is enabled, the DKMS module must be signed or it will fail to load. The easiest path for a dev box is to disable Secure Boot.
• Multiple GPUs / iGPU present: if your CPU also has an RDNA2 iGPU, ROCm may enumerate it. Pin the discrete card with HIP_VISIBLE_DEVICES (see Reference).
• hipBLASLt: some libraries assume hipBLASLt, which historically had gaps on Navi 21. If a workload complains, try TORCH_BLAS_PREFER_HIPBLASLT=0 (PyTorch) or the workload’s equivalent flag.

HSA_OVERRIDE for RDNA2 Cousins

Only Navi 21 cards report as gfx1030. The smaller RDNA2 dies use different LLVM targets:

Many ROCm libraries only ship precompiled kernels for a subset of targets. Because every RDNA2 card shares the same instruction set family, you can tell ROCm to treat your card as gfx1030 and reuse the gfx1030 kernels.

The override

export HSA_OVERRIDE_GFX_VERSION=10.3.0

10.3.0 maps to gfx1030. Set it in the shell (or systemd unit / container env) before launching any HIP/ROCm program:

HSA_OVERRIDE_GFX_VERSION=10.3.0 python my_inference_script.py

To make it permanent for your user:

echo 'export HSA_OVERRIDE_GFX_VERSION=10.3.0' >> ~/.bashrc

Why this works (and its limits)

• RDNA2 GPUs (gfx1030–gfx1036) are binary-compatible enough that gfx1030 kernels execute correctly on the smaller dies for the vast majority of ML ops.
• It is still a workaround. AMD does not officially validate it, and you may hit edge cases — particularly in hand-tuned assembly kernels or libraries that query the exact arch at runtime.
• Performance-tuned kernels (e.g. in rocBLAS/Tensile) were tuned for Navi 21’s CU count and cache; on a smaller die they run correctly but may be sub-optimal.

Building instead of overriding

If you compile a library yourself, prefer building for your real target so you get correctly-tuned kernels, e.g.:

# Build for multiple RDNA2 targets at once
export PYTORCH_ROCM_ARCH="gfx1030;gfx1031;gfx1032"
# or for a single card
export AMDGPU_TARGETS=gfx1032

For llama.cpp:

cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release

Verifying it took effect

HSA_OVERRIDE_GFX_VERSION=10.3.0 python -c \
  "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

If PyTorch previously errored with "no kernel image is available for execution on the device" and now prints True, the override is working.

Host firmware (MMIO High / large BAR / SR-IOV)

WIP / community: Workstation BIOS work for 4× V620 + PLX. Reports are from #motherboard and #general (Sep 2026). Treat every offset as your dump, not a copy-paste from a PDF. See Verification status.

Large-BAR cards (V620 is 32 GB GDDR6, plus a huge prefetchable MMIO window) can fail to POST on older workstation firmware even when the same cards boot on a modern desktop board. This page is about getting the host to enumerate the GPUs — not about PCIe P2P after Linux is up.

When this applies

Community symptoms:

• Host POSTs fine with no GPUs (or one GPU) and hangs with 4× V620 behind a PLX / switch
  • bifurcation card.
• The same 4× V620 + PLX kit works on a recent consumer board (community: Z690-class) but not on a Dell Precision tower you want for more RAM / more slots.
• lspci never appears because firmware never hands off.

If Linux already sees all cards, skip this page and go to P2P / Getting started.

Dell Precision T5820 / T7820 / T7920 (community)

#motherboard + #general (Sep 12–14 2026): these Xeon-W / dual-socket Precision towers often need a hidden MMIO High / large-BAR change. A circulating public write-up is the Precision 7920 MMIO GPU Unlock Guide (GitHub attachment). Community: T7820 behaves like T5820 for this class of tweak.

Operational rules (do these)

1. Dump your own UEFI variables. Do not copy a hex blob from the PDF or from someone else’s BIOS version. Community first-fail: writing another host’s dump back.
2. Confirm the MMIO change without the PLX (cards in CPU slots, or fewer cards). Then add the switch.
3. Prefer EFI .nsh scripts over typing hex by hand. Community used setup_var.efi and UEFITool against the matching BIOS update package (example: T7820 2.52.1) to find hidden variables.
4. Community variable name (T5820): SocketCommonRcConfig, sub-field mmiohsize set to 0x03 or 0x04. Write base + size together; keep the existing base. Both sizes have been reported as “the one that worked” — A/B on your dump.
5. Do not factory-reset, clear CMOS, or pull the CMOS battery after a successful write — that clears the change. #general (Sep 23): a host that lost BIOS settings (flat CMOS battery suspected) came back with SR-IOV re-enabled and ReBAR off, then saw MMIO not allocated / fewer GPUs than before. Re-apply the known-good firmware settings before chasing ROCm.
6. After Linux boots, continue with P2P readiness.

Aperture size (community)

V620 VT-d / IOMMU windows reserve a very large prefetchable range per card (community order of magnitude: hundreds of GB of address space per GPU, not “just 32 GB VRAM”). A 64 GB MMIO High window can be too small for 4× V620. Community: there is often no 128 GB option; jump to 256 GB if the menu / variable allows it. Mixed large-BAR cards (e.g. adding 32 GB MI50s next to V620s) made a too-small window worse until 256 GB was set.

Dual-socket towers

T7820-class boards can route some slots to CPU2 or the chipset. Label the slots and keep all inference GPUs on one CPU’s root ports when you can. See Host topology.

What actually unblocked 4× V620 + PLX (one host)

Community (#motherboard, Sep 14 2026): Dell Precision T7820, 4× V620, LSI PEX880xx-class switch on a bifurcation card, guest Proxmox.

• MMIO High / mmiohsize work was necessary but not sufficient — POST still failed with the PLX populated.
• The change that then posted and booted was disabling SR-IOV via hidden UEFI variables located with UEFITool on that BIOS package.
• That SR-IOV fix is board-specific. Do not treat it as a recipe for ASUS / HP / consumer boards.

Other platforms (short)

Supermicro H12D

#general (Oct 6 2026): dual-SP3 H12D (community also uses H12D-8D as a P2P / bifurcation host) can fail to POST a V620 with a repeating debug-LED loop. Community first-success set:

1. SR-IOV — disabled (also reported required for 4 GPUs behind one PLX on this board).
2. Above 4G decode — on.
3. CSM — off.
4. ReBAR — on.

That is a menu-level combo, not the Dell hidden-mmiohsize path above. Do not factory-reset / clear CMOS after it works — on other boards that re-enables SR-IOV and turns ReBAR off. Needs verify on other H12D revisions.

BIOS editors and unsigned flashes void warranty and can brick the board. Use the vendor flashback / recovery path if you have one. This wiki does not publish byte patches.

Tools (public)

Related

• Supported hardware — card identity
• Multi-GPU P2P — after the host POSTs
• Getting started
• Useful resources

Power Tuning (120 W floor)

The Radeon PRO V620 (gfx1030) ships with its power limit floor locked to 250 W in the VBIOS. That’s wasteful for inference, where the card spends most of its time memory-bound. This page summarizes how to unlock a 120 W floor and apply a 180 W boot cap, following the powertuning/ feature of the v620_toolbox repo.

Validated on Fedora 43 (kernels 6.17.6 pre-7 / 7.1.7 post-7) and Ubuntu 26.04 LTS (kernel 7.0.0-30-generic). Community confirmation: Fedora Server 44 + kernel 6.19, 4× V620, powerfix + 180 W cap (cap_min=120 W). This involves patching and rebuilding a kernel module — do it at your own risk.

Why it’s needed

The V620’s VBIOS declares 250 W as its minimum power limit, and the amdgpu driver trusts that number, so any lower cap is rejected:

echo 180000000 | sudo tee /sys/class/hwmon/hwmonX/power1_cap
# -> Invalid argument

The SMU firmware actually accepts values down to 120 W — only the kernel is in the way.

#general (17 Oct 2026): that “Invalid argument” story is not universal. A Debian 12.12 / kernel 6.1 host accepted echo 180000000 to power1_cap without the toolbox patch (power1_cap_min stayed 0, power1_cap_max stayed 250 W). Temps dropped vs stock 250 W. The same thread pointed at the public r/linux_gaming report that AMD sysfs power limiting is effectively broken on Linux 6.7+. Maintainer: forgot the pre-7.x behavior; the 180 W write did lower temps from 250 W on that report; the toolbox path still works on 7.x.

Treat a successful write on an old kernel as a soft cap only — it does not unlock the 120 W floor (power1_cap_min=0 means the driver never advertised 120 W). After a 6.7+ or 7.x upgrade, expect the write to start failing again until you apply the patch. The supported unlock is still this page’s toolbox path.

The fix

A tiny patch to sienna_cichlid_get_power_limit() in drivers/gpu/drm/amd/pm/swsmu/smu11/sienna_cichlid_ppt.c clamps the reported minimum to 120 W — but only when the GPU matches the V620 reference board by PCI identity:

• PCI device 1002:73a1, subsystem 1002:0e34

It does not touch the VBIOS, pp_table, or pp_features, and works on any kernel ≥ 5.15 with any number of V620s in the host. The canonical patch is patches/v620-powercap-min-120W.patch (portable across pre-7 and post-7 kernels via noinline).

+		if (smu->adev->pdev->vendor == 0x1002 &&
+		    smu->adev->pdev->device == 0x73a1 &&
+		    smu->adev->pdev->subsystem_vendor == 0x1002 &&
+		    smu->adev->pdev->subsystem_device == 0x0e34)
+			sienna_cichlid_v620_min_powercap_fix(smu, min_power_limit);

Other gfx1030 boards (RX 6900 XT / 6800 use device 0x73bf) have different PCI IDs. To power-tune those, change the identity match in the patch accordingly.

Platform paths

Both platforms share the same 120 W floor patch logic, the same v620-cap-apply.sh runtime script, and the same v620-powercap.service boot cap. Follow the README in the folder for your distro.

Ubuntu quick path

git clone https://github.com/blivioniag/v620_toolbox.git
cd v620_toolbox/ubuntu_powertuning

# Install deps (see ubuntu_powertuning/README.md), then:
sudo cp v620-rebuild-amdgpu /usr/local/sbin/
sudo cp ../powertuning/scripts/v620-cap-apply.sh /usr/local/sbin/
sudo chmod 755 /usr/local/sbin/v620-rebuild-amdgpu /usr/local/sbin/v620-cap-apply.sh

sudo /usr/local/sbin/v620-rebuild-amdgpu "$(uname -r)"
sudo reboot

After reboot: sudo dmesg | grep -i 'V620 powerfix' and ../powertuning/scripts/v620-verify.sh. Install the systemd unit and optional kernel/postinst.d/v620-amdgpu hook so future kernel updates rebuild the module — see ubuntu_powertuning/README.md.

Secure Boot: disable it or sign the rebuilt module — unsigned overrides won’t load with SB on.

Two ways to apply it (Fedora)

The toolbox provides scripts under powertuning/scripts/:

1. Bake a kernel RPM — v620-kernel-bake.sh builds a Fedora dist-git kernel RPM with the patch (and the P2P kernel-config delta) baked in. Survives cleanly across reboots.
2. Out-of-tree module override — v620-module-install.sh builds a patched amdgpu.ko against your running kernel and installs it as an override. Re-run after every kernel update.

Then apply the runtime cap at boot:

• scripts/v620-cap-apply.sh — writes the 180 W cap (matches the V620 by PCI ID).
• systemd/v620-powercap.service — a oneshot unit that runs v620-cap-apply.sh 180 at boot.

Verify

# The kernel logs the fix on match:
sudo dmesg | grep -i 'V620 powerfix'

# The reported minimum is now 120 W (120000000 µW):
cat /sys/class/hwmon/hwmon*/power1_cap_min

# The toolbox's own check:
sudo ./powertuning/scripts/v620-verify.sh

v620-verify.sh confirms both the V620 powerfix dmesg marker and power1_cap_min=120000000. After the override, idle around ~7 W per card has been reported (Fedora 44, 4× V620). If idle is still ~50 W, the floor/cap path is not active — re-run verify.

Token cost vs stock 250 W

#llamacpp: at the stock 250 W floor, V620 inference can look ~15% more expensive per token than a high-end Nvidia reference; with a 180 W cap after the floor unlock, the same report called V620 ~20% cheaper per token, with falloff “undetectable” down to ~200 W and only small percentages below that. Cap at 160 / 140 W mainly for PSU headroom, not because 180 W is slow. #general (Sep 2026) similarly calls ~120–170 W the practical sweet spot — little gained above that for the extra watts.

Undervolt experiments exist; community: about −25 mV was stable for one host, while more aggressive settings started emitting random tokens. #vllm-rdna (27 Oct): looping !!!! tokens reproduced on one host with undervolt larger than ~20 mV or core above ~2479 MHz. Other community clocks that did not show that loop: 2350 MHz core / 1050 MHz memory at stock 250 W, and 2400 / 1075 MHz at 160 W. Treat UV / high-core as optional and A/B carefully — token loops.

Slot power and PSU transients

Unlike many gaming cards, a V620 can reach TDP from the PCIe slot once it has enumerated. That makes the +12 V rail and slot power delivery more sensitive than a 3080-class swap-in. Enumeration is separate: #general (29 Oct): cards with only one 6+2 / 8-pin aux cable plugged can light LEDs and still never appear in the OS. Community: dual 8-pin (a single 8-pin from the PSU can Y-split) and slot power. Missing aux connectors is a common “I have 8 cards but lspci only sees 5” miss.

• Prefill / tensor-split can trip old or miner PSUs even when average watts look fine — see llama.cpp PSU troubleshooting.

• If a new card hard-reboots the host on load, try a gentler SMU ramp before blaming the kernel:

  sudo rocm-smi --setperflevel standard   # community: ~80 W min / ~150 W max, softer ramp
  

  (STANDARD / standard — check rocm-smi --help on your ROCm; the enum name varies.)

• Cap at 160 W or 140 W if 180 W still trips protection.

Multi-card PSU sizing (community)

#forum / #general (Sep 2026): an 8× V620 @ 180 W build targets ~1440 W for GPUs alone, leaving headroom for CPU/board. One reported pick that avoids sketchy splitters: SilverStone HELA 2050 Platinum (SST-AX2050MCPT-A). US hosts often need a 240 V circuit for that class of load — stock 250 W × 8 on a 120 V / 20 A breaker is a non-starter. Prefer the 120–180 W caps for both thermals and wall power.

#general (Sep 22–23): 1600 W Platinum was enough for 5× V620 at stock 250 W (~1250 W GPU TDP). A 1200 W ATX 3.1 for 4× at stock is tight (1000 W GPU TDP before CPU/board). Same thread: 0.7 A blowers were enough at 180 W; stock 250 W full-power inference hit ~86 °C on that host. Community is still unsure whether rocm-smi caps catch brief transients — size the PSU for stock TDP if you have not applied the floor yet.

#general (31 Oct): an 8× V620 host saw IOMMU page faults then fatal PCIe errors during TP8 or model load. The six cards on a 1500 W server PSU → breakout board dropped from the OS; the two cards powered entirely by a workstation PSU stayed. Unresolved (power delivery vs riser/cable vs driver). Needs verify. A/B the disappearing group’s aux power and dmesg for IOMMU / AER — the first card to log the fault is not necessarily the bad one. See general troubleshooting.

See the full recipe, prerequisites, and the deep-dive docs (docs/POWERCAP.md) in the repo.

Soft unlock (passthrough VMs)

Different problem, different tool: Tamalero/amd-v620-soft-unlock unlocks OverDrive clock + power controls on a V620 that is PCIe-passthrough into a Linux VM (Proxmox / QEMU / libvirt). Stock passthrough often leaves OverDrive empty (LACT / CoreCtrl / sysfs), a fixed ~1825 MHz core pstate, and a hard 250 W power cap.

How soft unlock works (summary — follow the upstream README):

1. Guest kernel: amdgpu.ppfeaturemask=0xffffffff
2. Dump the card’s own VBIOS from the guest (/sys/kernel/debug/dri/*/amdgpu_vbios)
3. Patch with make_odcaps_rom.py (sets OD caps 0–3 + ATOM checksum; no ROMs shipped)
4. Attach as QEMU romfile= / Proxmox hostpci…,romfile= / libvirt <rom file=…/>
5. Verify pp_od_clk_voltage shows an OD_RANGE, then tune with sysfs / tools/v620 / LACT

Warnings from upstream (do not skip):

• Never write guest pp_features or sysfs pp_table on this card — can wedge the SMU; V620 has no FLR, so recovery is a host reboot.
• Soft unlock is per-VM (romfile=). Nothing is written to the physical flash; remove romfile= to revert.
• Passive baremetal ACPI VFCT delivery path is documented upstream as experimental / untested.
• Passive server cards need real airflow before raising clocks / 275 W.

#general: community reports of reliable use under Proxmox passthrough (weeks-long). This wiki has not independently validated TFLOPS or power-range claims — treat benches in the upstream README as community-reported.

Disabling ECC (Pro VRAM)

WIP: Procedure is documented for Radeon PRO W6800 in lunnova’s guide. #vllm-rdna and #general report the same ECC-on-by-default behavior on V620. Treat the two-reboot kernel-param path as community until you confirm rocm-smi on your own cards. See Verification status.

Workstation/data-center Navi 21 cards (PRO V620, PRO W6800) ship with on-board ECC enabled. That costs about 7% of VRAM (~2 GB on a 32 GB card). Consumer RX 6800 / 6900 cards have no ECC and already show the full 16 GB.

#vllm-rdna (pinned): extra ~2 GB per GPU if you disable it. Some members keep ECC on for bit-flip protection; others disable it for long-context KV cache.

#general (Sep 23): stress-test VRAM before turning ECC off. One returned V620 only showed a single correctable ECC error in dmesg; a Vulkan memtest-class load produced many errors in minutes. Community: the extra ~2 GB is not worth it if the card is already throwing ECC errors — leave ECC on and RMA / replace. Do not treat a quiet dmesg as a clean card.

#general (Sep 26): Flash-Next can surface ECC that a 27B serve does not. One host only saw correctable errors in dmesg while running Qwen3.8 Flash-Next; the same cards stayed quiet on Qwen3.8-27B. A follow-up Vulkan memtest of about 5–8 minutes per GPU (wait until temps stabilize) was enough to catch a bad card that then started erroring immediately. A replacement passed. Leave ECC on when dmesg reports errors even if a short 27B smoke looks fine.

Check whether ECC is on

sudo dmesg | grep -i ecc
# "MEM ECC is active" / "GECC is enabled"  → ECC is on
# "MEM ECC is not presented"               → consumer card / no ECC

rocm-smi --showmeminfo vram
# ~30.7 GB total on a 32 GB Pro card → ECC is eating the rest

Linux: amdgpu.ras_enable=0 (two reboots)

There is no Radeon control-panel toggle on Linux. The community path (lunnova, 6.x kernels) is:

1. Add the kernel parameter amdgpu.ras_enable=0.
2. Reboot twice, leaving the parameter in place both times.

GRUB (Ubuntu / Debian / Fedora):

# Ubuntu/Debian: edit GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub
# Fedora: sudo grubby --update-kernel=ALL --args="amdgpu.ras_enable=0"
sudo nano /etc/default/grub
# …add amdgpu.ras_enable=0 to GRUB_CMDLINE_LINUX_DEFAULT, then:
sudo update-grub   # Ubuntu/Debian
# Fedora uses grubby above instead of update-grub
sudo reboot

After the first reboot, dmesg should mention that GECC will be disabled on the next boot:

GECC will be disabled in next boot cycle if set amdgpu_ras_enable and/or amdgpu_ras_mask to 0x0

Reboot again. On the second boot you want:

amdgpu: GECC is disabled

and rocm-smi --showmeminfo vram should report the full ~32 768 MiB.

To turn ECC back on: remove the kernel parameter and reboot twice.

An older amdgpu-no-ecc.patch is not needed on 6.x kernels. Keep it only if you are still on 5.x and the two-reboot path does nothing — details in lunnova’s article.

Notes

• Consumer gfx1030 (RX 6800 / 6800 XT / 6900 XT / 6950 XT): skip this page.
• Secure Boot / signed kernels: same caveats as power tuning — unsigned module overrides will not load.
• This does not replace the 120 W power-cap patch. ECC and the VBIOS power floor are independent.

Related

• Supported Hardware
• Power Tuning
• Getting Started

Multi-GPU PCIe P2P

For multi-card V620 rigs, enabling GPU↔GPU PCIe Peer-to-Peer (P2P) lets the GPUs DMA directly to each other’s VRAM instead of bouncing through system memory — important for tensor/pipeline parallel serving. This page summarizes the pcie_p2p/ feature of the v620_toolbox repo.

Validated end-to-end on Fedora 43 + AMD CPU (EPYC 7452), 4× Radeon PRO V620. Result: rocminfo enumerates 5 HSA agents and amd-smi topology shows 12/12 GPU↔GPU P2P ENABLED.

Power-cap tuning on Ubuntu 26.04 is supported separately via ubuntu_powertuning/ — P2P recipes here remain Fedora-validated for now. Community Intel notes (Ice Lake) are in When P2P helps — and when it does not.

Two validated kernel paths

Both kernels can be installed on the same host simultaneously; toggle the default with grubby --set-default /boot/vmlinuz-<evr>. Both boot with the 180 W power cap from Power Tuning.

#vllm-rdna / #general (Sep 24): Ubuntu 26 may ship a 7.0-class kernel that sits in the gap between these two validated paths. Do not assume the Fedora Post-7 recipe applies unchanged — rebuild with the toolbox options (or stay on a Pre-7 / ≥7.1 build), then add iommu=pt, confirm ACS off (bit-level caveat), ReBAR / Above 4G on, CSM off. Power-cap helpers for Ubuntu 26 live under ubuntu_powertuning/; P2P itself remains Fedora-validated.

Key prerequisites

• AMD CPU — the KFD P2P path is wiki-validated on AMD (EPYC 7452). Intel Ice Lake can enumerate P2P (community) but often does not speed up inference — see below.
• ≥ 2 Radeon PRO V620 for a meaningful P2P topology (bench validated on 4).
• No pre-gfx1030 AMD GPU installed alongside the V620. ROCm 7.x dropped support for older ASIC families (gfx8xx Polaris, gfx900 Vega, …). If any pre-gfx1030 AMD GPU is present at boot, KFD registration fails and rocminfo bails with Failed to map remapped mmio page on gpu_mem 0. This only matters if your host actually has older AMD cards — remove them first.

Identify your V620 reference boards:

lspci -nn | grep '1002:73a1'    # one line per V620, subsystem 1002:0e34

The full 4-tuple 1002:73a1:1002:0e34 uniquely identifies the V620 reference board and is what the toolbox scripts match on. Other gfx1030 boards (RX 6900 XT / 6800) use device 0x73bf and different subsystem IDs — they’d need the identity match adjusted.

Readiness & verification

The toolbox ships diagnostics that gate on kernel config, hardware, ACS/IOMMU, and runtime state:

# Four-gate readiness check (kernel + hardware + ACS/IOMMU + runtime)
sudo ./powertuning/scripts/v620-p2p-readiness.sh

# End-to-end P2P verification
sudo ./pcie_p2p/scripts/verify-p2p.sh

A healthy system shows:

rocminfo | grep -c '^  Name:.*gfx1030'     # one per V620
amd-smi topology                            # all GPU<->GPU pairs: P2P ENABLED

For the full recipe (kernel config deltas, BIOS/IOMMU settings, what can go wrong), read pcie_p2p/README.md and the knowledge base powertuning/docs/AMD_P2P.md.

ACS is several bits, not one switch

Wiki and toolbox prose still say “ACS off”. #general (25 Oct 2026): that is not a single enable bit. PCIe Access Control Services is a per-port control word (ACSCtl). Clearing every bit is a different recipe from “P2P Request Redirect is off.”

Community host that passed V620↔V620 after the toolbox readiness script still reported ACSCtl = 0x0019 on the relevant switch / root ports (nine ports on that machine):

Same thread, two recipes — do not pick a winner without a second host + verify-p2p.sh:

Inspect without copying a personal BDF. Replace 00:00.0 with your switch or CPU root port (not the GPU):

sudo setpci -s 00:00.0 ECAP_ACS+0x06.w
# 0019 = the community-passing mask above

Then run v620-p2p-readiness.sh and verify-p2p.sh. Treat a readiness “ACS off” line as incomplete until you have amd-smi topology P2P ENABLED and a working TP serve. Needs verify on a second topology.

When P2P helps — and when it does not

amd-smi topology saying P2P ENABLED is not the same as faster tokens. Always A/B with NCCL_P2P_DISABLE=1. Community reports (#general, Ice Lake 4× V620 host):

If bandwidth tests pass but inference regresses:

export NCCL_P2P_DISABLE=1          # llama.cpp / RCCL tensor parallel

On the RDNA2 fork, the README also documents a flag to disable P2P all-reduce fusion (GGML_HIP_GFX1030_P2P_ALLREDUCE=off). Verified ~25 GB/s P2P between V620 pairs does not guarantee faster token generation if links are narrow or the host is a single monolithic PCIe root.

Intel: do not disable IOMMU to “help” P2P

On Ice Lake, disabling VT-d / IOMMU broke P2P even though some generic docs say IOMMU-off is more permissive. Leave IOMMU on, then run the toolbox readiness script and ACS checks. Fedora Server 44 + kernel 6.19 + the powerfix has been reported working on 4× V620 (community; not the Fedora 43 AMD validation).

PLX / PCIe switches

• Intra-switch P2P can stay full-width (e.g. 4× Gen4 x16 behind one PLX 88096). The host↔switch uplink (typically one x16) is the bottleneck. #general (Sep 21): a single gen4 x16 uplink to a PEX88096 is still “full speed to the switch”; cards talk to each other on the switch bus. Consumer X570 boards that expose two CPU x16 (bifurcated x8x8 gen4 when both slots are filled) were called a cheaper host than a workstation board if you only need that uplink.
• #general (Sep 19): one gen4 x16 on the host → community suggestion is a PEX88096 broken out to SlimSAS and 8 risers. Alternative board: dual-SP3 Supermicro H12D-8D, bifurcate all 4 PCIe ports to x8x8 gen4. 6 GPUs is still awkward (not a clean TP size; the open question was TP=8 vs TP=2 × PP=3). One camp says buy 2/4/8 only; another says any even count can work if you plan the split — do not add a third card assuming TP=3.
• 4-slot vs 5-slot switch boards (#general, Sep 2026): a common 5-slot PLX only fits double-width cards; a 4-slot board can take triple-width GPUs (future cooler upgrades). Pick the board for card thickness, not just GPU count.
• 5 GPUs: community prefers TP=4 plus a standalone fifth card (draft / embedder / second model). Pipeline-parallel on 5 can work but is “not optimal.” Odd counts still have TP3 crash notes.
• Without tensor parallel, a PLX box is usually slower than native CPU lanes: higher latency and less aggregate host bandwidth.
• Do not tensor-split across a daisy-chained pair of switches — that single inter-switch link is a TP bottleneck. Prefer TP inside one switch and pipeline-parallel between switches (vLLM can do TP+PP that way; llama.cpp generally cannot).
• Community: ACS often needs extra kernel cmdline fiddling; pcie surprise link down crashes were fixed by putting a small fan on the PLX heatsink (these boards often ship with no airflow notes).
• Workstation hosts that do not POST with V620 + PLX (Dell Precision MMIO High / SR-IOV; Supermicro H12D POST loop): Host firmware.
• External PCIe expansion (#general, Sep 2026 — e.g. Cubix Xpander–class enclosures): each group of cards may get full-width slots behind the switch, while only one x16 (often PCIe 3.0) returns to the host. Community: 8× GPU on that pattern was fine at low context but fell to ~3 t/s around 50–60k context. Prefer 4 GPUs on one switch + P2P / TP inside the switch over stretching TP across a narrow host uplink.

Host topology

Community reports, not wiki-benched:

Layer split vs tensor split (quick)

Community explainer (#benchmarks thread, Sep 2026):

• Layer / pipeline split — VRAM grows with each card, but only one GPU works the model at a time (sequential stages). Less GPU↔GPU traffic.
• Tensor (row) split — all GPUs work together each step; needs much more interconnect bandwidth and often becomes PCIe-limited on narrow links.

On fat CPU roots (gen4 x16 × N), try tensor first. Behind a narrow switch uplink, prefer smaller TP domains or layer/PP splits.

Cabling / risers (community)

Not wiki-benched — common #general notes:

• SlimSAS / SFF-8654 cables: PCIe gen3 is usually fine at ~70 cm; gen4 needs testing per cable/insulation. Prefer known-good gen4 kits over the cheapest Amazon/eBay passive ribbon risers.
• Passive PCIe risers: several community reports of timeouts / inability to hold gen4 x16 with cheap passive risers under multi-GPU load, while the same slots work with cards seated directly. Brand-name gen4 risers (e.g. ADT-class) are often the next step — A/B one card first.
• V620 + blower shroud length: community measure ~37 cm with a common EFH-08E12W-style fan shroud installed — plan chassis / PLX slot spacing accordingly.
• 5-slot PCIe switch → ATX tray: #other (Sep 30) published a Printables adapter (PCIe switch to ATX mounting adapter) for a 5-slot PCIe 4.0 switch on an ATX hole pattern. Community: it matched one common PLX board even when the listing photos differed. Mechanical only — not a P2P or POST fix.

vLLM on RDNA — Overview

WIP: This section is actively expanded from Discord and fork release notes. See Verification status before treating benchmarks as gospel.

The quickest way to serve LLMs on gfx1030 / RDNA is the prebuilt blivioniag/vllm-rdna images on a blivioniag/rocm-rdna ROCm + PyTorch base. A single image targets seven RDNA architectures (gfx1030 through RDNA4).

Not sure whether to use vLLM or llama.cpp? Start with llama.cpp or vLLM.

Where to start

Image variants

Image tags are refreshed in place when fixes land — always docker pull before debugging. Confirm your -extras image includes the latest extras commits (AWQ dispatch, GDN HIP, TP graph fix). #vllm-rdna (Aug 31 2026): blivioniag/vllm-rdna:v0.27.1-extras was refreshed again — re-pull even if you already had that tag.

Official source: Kernel work lives in opengfx1030/vllm-rdna (rdna_extras = v0.31.0 as of 11 Oct 2026). Day-to-day serving is Hub blivioniag/vllm-rdna:v0.31.0-extras. First-time path: Onboard on V620. Older v0.27.1-extras* / v0.28.0-extras* tags stay on Hub for A/B. #vllm-rdna Sep 14–15: Flash-Next commits are in rdna_extras; the standalone leapdragon/vllm-rdna2-qwen branch is no longer the moving target. Prefer the 0.31 Hub tag or a host venv over the old leapdragon image. Details: Fork landscape.

Upstream vLLM: Official GPU docs still omit Navi 21 / gfx1030. Keep the community fork until upstream documents it.

Related

• Choose a stack — llama.cpp vs vLLM, and what fits on a V620.
• Flash-Next — card counts, AutoRound, and serve lines.
• Multi-GPU PCIe P2P — important for tensor parallel.
• Environment variables — cheat-sheet.
• Community recipe book (mirror opengfx1030/vllm-rdna2-recipe) — presets, patches, concurrent MTP PRs.

Onboard vLLM on V620

WIP: Happy path below follows the #announcements v0.31.0 release (11 Oct 2026) and day-to-day practice in #vllm-rdna. See Verification status.

Get from a bare V620 host to an OpenAI-compatible vllm serve with the community opengfx1030/vllm-rdna fork (rdna_extras = upstream vLLM 0.31.0 + RDNA HIP kernels). Prefer this page over the deeper vLLM docs until the first request succeeds.

Not sure vLLM is the right stack? Read llama.cpp or vLLM first.

What you need

Optional but worth it on V620 before long serves: power cap, ECC, PCIe P2P for multi-GPU. Full host walkthrough: Getting started.

Pick a path

#announcements (11 Oct 2026): Hub v0.31.0-extras is the published 0.31 bake (ROCm 7.14, torch 2.12, Triton 3.7.1). Older v0.27.1-extras / v0.28.0-extras tags remain on Hub for comparison only.

Path A — Docker Hub -extras (fastest)

docker pull docker.io/blivioniag/vllm-rdna:v0.31.0-extras

docker run -it --rm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  docker.io/blivioniag/vllm-rdna:v0.31.0-extras \
  vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
    --dtype float16 \
    --max-model-len 8192 \
    --language-model-only --skip-mm-profiling --trust-remote-code

• Always docker pull before debugging — tags refresh in place.
• Prefer --dtype float16. RDNA2 BF16 is weak/emulated.
• Multi-GPU: add --tensor-parallel-size N. Prefer power-of-2 TP. Enable P2P when your host supports it; if P2P is broken, set NCCL_P2P_DISABLE=1 (and see Path B P2P=0).
• First boot is slow (Triton / torch.compile). Mount caches for restarts — Configuration.

Smoke test

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"cyankiwi/Qwen3.8-27B-AWQ-INT4","messages":[{"role":"user","content":"Hi from V620"}]}'

Flash-Next in Docker (4× V620)

#announcements (11 Oct): the same v0.31.0-extras image can serve Flash-Next. You need local weights plus the PLE table (wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16 and primitive-ai/Qwen3.8-Flash-Next-PLE-quant). Mount the INT4 PLE directory and pass the fork env:

# Paths are examples — use your local checkpoints.
docker run -it --rm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc host \
  -p 8000:8000 \
  -v /path/to/Qwen3.8-Flash-Next-AWQ:/model \
  -v /path/to/Qwen3.8-Flash-Next-PLE-quant/ples_int4:/ples_int4 \
  -e VLLM_PLE_QUANT_DIR=/ples_int4 \
  docker.io/blivioniag/vllm-rdna:v0.31.0-extras \
  vllm serve /model --dtype float16 --tensor-parallel-size 4

For disk PLE (lower pinned RAM), also set -e VLLM_PLE_CPU_OFFLOAD=1. Prefer Path B’s RECIPE=flashnext-mtp2 once you want the full tuned argv (graphs, MTP, features). Details: Flash-Next, Serve scripts.

Path B — Fork checkout + host venv

Use this when you want HEAD of opengfx1030/vllm-rdna (rdna_extras) or the in-tree launcher. Build steps live on Host venv (ROCm 7.14.0 multi-arch wheels are the usual match for the 0.31 Docker bake).

git clone -b rdna_extras https://github.com/opengfx1030/vllm-rdna.git
cd vllm-rdna
# Build torch + FlashAttention + causal-conv1d + editable vLLM — see host-venv.md

After import vllm works in the venv, serve from that checkout:

# Day-to-day dense 27B (local checkpoint or set HF_HUB_OFFLINE=0)
MODEL=/path/to/Qwen3.8-27B-AWQ VENV=/path/to/venv-7.14.0 \
  bash tools/rdna/serve_rdna.sh RECIPE=27b-awq

# Flash-Next MTP=2 on 4× V620 (local weights + INT4 PLE)
MODEL=/path/to/Qwen3.8-Flash-Next-AWQ VENV=/path/to/venv-7.14.0 \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4 \
  HOST=0.0.0.0 PORT=8888

• MODEL and VENV are required. Recipes do not bake in home directories or IPs.
• Dry-run: PRINT=1 … RECIPE=27b-awq prints env + argv without launching.
• Hosts without working PCIe P2P: add P2P=0 (#vllm-rdna Oct 11 — disables rdna_ar and sets NCCL_P2P_DISABLE=1).
• Other recipes: flashnext, flashnext-mtp0, 27b-exl3, dsv4-flash, full — Serve scripts.

Which model for your card count

Full matrix: Recipes. Workload fit: What fits on V620.

If something breaks

Ask in Discord #vllm-rdna — invite.

Next steps

• Recipes — Hub vs recipe container vs Flash-Next by card count
• Configuration — env vars, CUDA graphs, Compose
• Quantization — AWQ / GPTQ / MTP / KV
• vLLM forks — kernel landscape and PR status
• Building images — bake your own from vllm-rdna-docker

Running vLLM (Docker)

WIP: See Verification status.

Image matrix

Base images — blivioniag/rocm-rdna

Bake target base-rocm714 pins torch 2.12.0+rocm7.14.0. The docker-repo README still says 2.13.0 for this tag — treat that README cell as stale; confirm with python -c "import torch; print(torch.__version__)" inside a pulled container.

These are a general-purpose ROCm PyTorch base for RDNA — useful on their own if you just want a working torch on a Radeon card.

vLLM images — blivioniag/vllm-rdna

The -extras tags ship the rdna_extras fork (hand-written RDNA2 HIP kernels: FlashAttention, quantized GEMM, MoE, GDN, …). Check Docker Hub for the current tag list. Tags are refreshed in place — docker pull before debugging.

#announcements (11 Oct 2026): v0.31.0-extras is the current extras image — rdna_extras is now the v0.31.0 port, and that bake served Flash-Next MTP-2. First-time path: Onboard on V620. Older Hub tags (v0.27.1-extras*, v0.28.0-extras*) do not track HEAD. For kernels newer than the bake, clone the org repo and follow Host venv + serve_rdna.sh. Use rocm-rdna:7.14.0 if you only need a ROCm packaging base.

Multi-GPU: pick a 7.2.0 or 7.14.0 tag, not a host ROCm in the 7.2.1–7.13 gap. RCCL on those in-between releases is reported broken with more than one card — see Installing ROCm.

Every image bakes these PYTORCH_ROCM_ARCH targets: gfx1030;gfx1100;gfx1101;gfx1150;gfx1151;gfx1200;gfx1201.

Run it

docker run -it --rm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  docker.io/blivioniag/vllm-rdna:v0.31.0-extras \
  vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
    --dtype float16 \
    --max-model-len 8192 \
    --language-model-only --skip-mm-profiling --trust-remote-code

First-time V620 walkthrough (Docker + fork launcher): Onboard on V620.

• Give the container the GPU with --device /dev/kfd --device /dev/dri and the video and render groups. Missing render is a common #vllm-rdna cause of Failed to infer device type.
• Prefer --dtype float16. RDNA2 has weak/emulated BF16; letting vLLM pick bf16 from a model’s config.json can trigger slow float32 fallbacks.
• The 27B AWQ example is the current #vllm-rdna day-to-day dense pick (Sep 2026). For a smaller smoke test, swap in any instruct model that fits VRAM. Model / card-count matrix: Recipes.
• For a non-Navi-21 RDNA2 card (gfx1031/1032/…), add -e HSA_OVERRIDE_GFX_VERSION=10.3.0. See HSA_OVERRIDE.
• Multi-GPU: add --tensor-parallel-size N; enabling PCIe P2P helps a lot here.

Query the OpenAI-compatible endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"cyankiwi/Qwen3.8-27B-AWQ-INT4","messages":[{"role":"user","content":"Hi from gfx1030!"}]}'

Next steps

• Choose Hub vs recipe container vs Flash-Next: Recipes
• Tune env vars and CUDA graphs: Configuration
• Pick a quant format: Quantization
• Kernel details: rdna_extras fork

vLLM recipes (pick a path)

WIP: Stack picks and tok/s figures are community-reported (#vllm-rdna, Sep 2026). See Verification status. Prefer this page when you want “which image / which model / how many cards” — not a kernel deep-dive.

Discord keeps asking the same questions: which Docker stack, which Hugging Face weights, and whether 1× V620 is enough. This page consolidates those answers. Details for env vars and CUDA graphs live on Configuration; fork history on vLLM forks.

Which stack?

#vllm-rdna (Sep 2026): the recipe repo is treated as a parts pile (compose/env/patches against pristine vLLM 0.27.1). New Flash-Next work lives on opengfx1030/vllm-rdna rdna_extras (Sep 14–15 cherry-picks); vllm-rdna2-qwen is the older published container. Official kernel development is the same org extras repo — Hub -extras tags may lag HEAD until bake retargets.

Do not mix host ROCm userspace into the recipe container — the image carries its own stack. Mounting host ROCm into it is a common break (recipe TROUBLESHOOTING.md).

Pick by card count

Also see What fits well on V620.

#vllm-rdna (Oct 8), for llama.cpp migrants: --pipeline-parallel-size N is the vLLM analogue of llama.cpp layer split (-sm layer); --tensor-parallel-size N is the analogue of tensor split (-sm row / -ts). vLLM can combine them (example: TP=2 + PP=2 on four cards). Prefer power-of-2 TP. Flash-Next on four cards still wants TP=4, not PP=4 (PP=4 can look great on long prefill then tank decode — table above).

Gemma 4 note

Community reports Gemma 4 ~26B still fails or is unfinished on current gfx1030 vLLM paths (#vllm-rdna, Sep 13 2026). Prefer Qwen / Ornith until someone posts a working recipe.

Model cheat sheet

Small-VRAM experiment: Ornith 9B EXL3 (~6.8 GB) vs AWQ (~9 GB) — experimental, not in published -extras tags yet. See Quantization.

Path A — Hub -extras (recommended default)

docker pull docker.io/blivioniag/vllm-rdna:v0.31.0-extras

docker run -it --rm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  docker.io/blivioniag/vllm-rdna:v0.31.0-extras \
  vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
    --dtype float16 \
    --max-model-len 8192 \
    --language-model-only --skip-mm-profiling --trust-remote-code

Short V620 walkthrough: Onboard on V620.

• Add --tensor-parallel-size N for multi-GPU. Enable PCIe P2P when possible.
• Prefer --dtype float16. Mount Triton / torch-compile caches for faster restarts — Configuration.
• Force the native W4A16 path when needed: VLLM_DISABLED_KERNELS=ExllamaLinearKernel,TritonW4A16LinearKernel — confirm Using RDNA2W4A16LinearKernel in logs.

Full env block and Compose (GPTQ + MTP): Configuration.

Hub -extras TP4 Qwen3.8-27B AWQ

Sanitized from a #vllm-rdna (Aug 31 2026) bench recipe on 4× V620 with working P2P / custom all-reduce. Drop the custom-AR block if P2P is broken on your board — use VLLM_DISABLE_CUSTOM_ALL_REDUCE=1 instead (Configuration).

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export VLLM_RDNA_FORCE_FP16=1
export VLLM_USE_RDNA2_FA=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_ENABLED=1
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_BATCH_INVARIANT=0
export GPU_MAX_HW_QUEUES=2

# Only when GPU↔GPU P2P works:
export VLLM_FORCE_CUSTOM_ALL_REDUCE=1
export NCCL_P2P_LEVEL=pix
export RCCL_P2P_NET_DISABLE=1
export RCCL_P2P_BATCH_ENABLE=1
export NCCL_PROTO=Simple
export RCCL_MSCCL_ENABLE=0

cd /tmp   # avoid sys.path collisions with a local vllm checkout
vllm serve cyankiwi/Qwen3.8-27B-AWQ-INT4 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 20480 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.88 \
  --dtype float16 \
  --language-model-only --skip-mm-profiling --trust-remote-code \
  --enable-prefix-caching --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Host-venv TP=2 Qwen3.8-27B AWQ (community, Sep 26)

#vllm-rdna (Sep 26): one host cloned opengfx1030/vllm-rdna and built from source (not a Hub tag), then served cyankiwi AWQ-INT4 at TP=2. Same short test on both models; answers coherent, no NaNs. Community / Needs verify.

Day-to-day 27B on 4 cards still uses the TP4 block or a recipe preset. This table is only the two-card / source-build snapshot.

#vllm-rdna (Oct 6): a different 2× host ran unquantized fp16 Qwen3.8-27B at TP=2 with MTP and reported ~40–48 t/s single-stream. A shorter-reasoning Swift 1.5 27B (not the Flash-Next Swift packs) on the same card count was ~48 t/s single-stream and ~70 t/s across two instances. Community / Needs verify. Concurrency notes for wvSplitK vs stacked LLMM1: troubleshooting.

Path B — Recipe container (27B / 122B presets)

Fastest path when you want the recipe book’s measured knobs without installing ROCm on the host:

IMG=ghcr.io/leapdragon/vllm-rdna2-recipe:0.27.1-rocm7.2.3-gfx1030
docker pull "$IMG"
docker run --rm "$IMG" list-presets

docker run -d --name vllm-rdna2 --network=host \
  --device /dev/kfd --device /dev/dri \
  --group-add "$(getent group render | cut -d: -f3)" \
  --group-add "$(getent group video | cut -d: -f3)" \
  --ipc=host --ulimit memlock=-1 --security-opt seccomp=unconfined \
  -e ROCR_VISIBLE_DEVICES=0,1 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMG" preset:qwen38-27b-gptq

First boot cold-compiles for ~10–15 minutes; mount /compile-cache and Triton caches for ~3 min warm boots — see the recipe containers/README.md. Community ballpark on 2× V620: ~40–49 tok/s decode when TunableOp rows are seeded; ~27 t/s flat if they are missing (recipe troubleshooting).

122B builds need the repo wrapper / one-time weight prep — not a bare preset: one-liner.

Path C — Flash-Next

Flash-Next (Qwen3.8, PLE sidecar, 3×/4× V620 serve lines, AutoRound, DeepSeek-V4) lives on Flash-Next. This page stays the picker for Hub -extras and the recipe-container presets.

GLM-5.3-Flash (community, Needs verify)

#vllm-rdna (Sep 30): a community 8× V620 host served public wtdcode/GLM-5.3-Flash-AWQ-W4A16 (AWQ W4A16 of zai-org/GLM-5.3-Flash; routed experts INT4, attention / vision / MTP left BF16) on a local vLLM 0.30 fork. TP=8, MTP-2, P2P off (PLX), 150 W caps, dual-socket Broadwell-EP class. This is not Hub -extras and not rdna_extras HEAD.

Org draft opengfx1030/vllm-rdna#2 is a runtime-unverified Glm5Next load path on a side branch (later/glm53-flash-awq); HIP gates default off. #vllm-rdna (Sep 30): at that time day-to-day extras were still 0.28 and the maintainer paused the 0.30 rebase, inviting a PR of the community 0.30 work. #announcements (11 Oct 2026): day-to-day is now Hub v0.31.0-extras — re-check GLM on that line before treating the Sep 30 pause as current. V2 model runner can work on the local-fork line.

Community snapshot (one host — Needs verify):

gfx1030-relevant lessons (do not treat the unpublished kernel list as a Hub recipe):

• Serve --dtype float16, not bf16 — no bf16 hardware. Same quality; community +68% decode on this host.
• Vision: MIOpen compiled the downsample conv per image size (3–4 min; killed the engine once). Community used conv-as-matmul (158 s → 4 ms). Separate from the Flash-Next max_pixels OOM — troubleshooting.
• Indexer / tiled-fp8 / fused mHC / skinny-int4 MoE work lived on the local fork.

Not a drop-in for 4× Flash-Next hosts. #general (25 Oct): 1× V620 + host-RAM offload is also a poor fit. Community Maya snapshot ~200 PP / ~13 TG on one host (Needs verify, not recently rechecked). Public 18B active implies a theoretical decode ceiling around 30 t/s before MTP if experts stayed on-GPU; real offload will not get close.

Related

• Serve scripts — in-tree serve_rdna.sh after a host-venv build
• Running (Docker) — image matrix and minimal docker run
• Flash-Next — Path C serve lines
• Configuration — env vars, CUDA graphs, Compose
• Quantization — GPTQ/AWQ, KV, MTP
• vLLM forks — rdna_extras vs Flash-Next vs recipe ports
• Useful resources

Flash-Next (vLLM)

WIP: Benchmarks and serve lines are community-reported (#vllm-rdna). See Verification status. Day-to-day 27B serving stays on Recipes (Hub -extras or the recipe container). This page is only the Flash-Next track.

On this page

When something fails, use Flash-Next troubleshooting rather than the general vLLM list.

Qwen3.8 Flash-Next on vLLM

llama.cpp still struggles with Flash-Next on gfx1030 (upstream gaps). On 4× V620, community production is the Flash-Next vLLM stack on rdna_extras (v0.31.0) — not the old v0.27.1-extras* Hub tags. #announcements (11 Oct 2026): Hub blivioniag/vllm-rdna:v0.31.0-extras served Flash-Next MTP-2 (BF16 + INT4-disk). #vllm-rdna Sep 14–15: that work lives on opengfx1030/vllm-rdna rdna_extras; leapdragon/vllm-rdna2-qwen is the older published container / docs tree (still receives occasional commits; last push 11 Oct 2026). Prefer latest rdna_extras or the 0.31 Hub tag over llama.cpp until the RDNA2 llama.cpp fork catches up.

2× V620 / RAM offload: resident Flash-Next experts still do not fit two V620s. Merged #146 (28 Oct) is the first in-tree vLLM path: pinned-RAM expert banks + a bounded GPU slot cache, launched with tools/rdna/mono/serve_offload.sh on HEAD — 2× serve. Hub v0.31.0-extras lags. #vllm-rdna (Sep 14): without that HEAD launcher, use llama.cpp (--n-cpu-moe, -ot …=CPU). Community 2-card llama.cpp Flash-Next is still ~25 t/s decode / ~150–200 PP — far below the 4-card vLLM class. #vllm-rdna (16 Oct): MonoKernel alone does not make resident Flash-Next fit 2 cards.

3× V620: vLLM wants PP=3 (even TP). Current rdna_extras HEAD (merged #96, 13 Oct) ships RECIPE=flashnext-pp3 / flashnext-pp3-mtp2 — PP3 recipes. Hub v0.31.0-extras (11 Oct bake) does not include those recipes. Older Sep 15–16 pins corrupted PP3 decode on rdna_extras — troubleshooting. The Sep 17–18 leapdragon overlay is a snapshot — overlay.

4× V620 + MonoKernel (#94, 16 Oct): opt-in --monokernel on HEAD for Flash-Next decode ITL — MonoKernel. Hub v0.31.0-extras lags.

4× V620 + HEAD HIP defaults (#130, 19 Oct): Flash-Next recipes on current rdna_extras turn the #99 HIP set on (QSA / GDN / HC). Hub bake still has the older opt-in path — HIP defaults.

4× V620 + DFlash2 (#132, 21 Oct): RECIPE=flashnext-dflash2 is on HEAD. Paired A/B kept native MTP-2 faster; use DFlash2 only if you want that drafter — DFlash2.

4× V620 + rdna_extras graphs (#vllm-rdna Sep 16–25): on current HEAD, prefer FULL_AND_PIECEWISE (compile mode: 3) — uniform decode uses the FULL CUDA graph (same live path as #17 FULL_DECODE_ONLY); mixed/prefill use piecewise. See FULL_AND_PIECEWISE serve and fork docs/rdna2/V620-FULL-AND-PIECEWISE.md. #20 alone still redirected FULL→piecewise (~26–34 t/s decode); the keep-FULL fix (6c26c78d, #vllm-rdna Sep 24) turns that redirect off. #17 FULL_DECODE_ONLY remains a valid measured baseline — Sep 23 serve notes. Older hosts that saw FULL decode corrupt stayed on PIECEWISE — FULL-graph corruption / Sep 17 serve line. Community snapshot on that older recipe: ~3331 tok/s PP / ~73 tok/s TG @ 16k/1k, c=8 (Needs verify). There is no Q3 path on vLLM.

6× V620 Flash-Next (#vllm-rdna Sep 24): minimum 3 cards. Maintainer starting points: PP=3 + TP=2 (six GPUs) or PP=6. Prefer power-of-2 tensor parallel over PP=2 + TP=3. Disaggregated prefill/decode (two model loads) is a separate WIP — open #23. Needs verify — no published 6× table.

#vllm-rdna (Sep 19): pipeline parallel 4 on Flash-Next was reported to hit ~1950 tok/s prefill at 32k while decode collapsed. The same host dropped PP4 and continued on TP=4 (~1550 tok/s prefill at long context; decode ~30–35 t/s with MTP-2). Treat PP4 as a prefill experiment, not a 4-card default. Needs verify.

#vllm-rdna (Sep 20): opengfx1030/vllm-rdna#15 merged into rdna_extras. AMD QSA prefill scoring is bounded to the live prompt context (decode / missing metadata still use the capacity-wide path). The same PR folds V620 startup prerequisites from superseded PR #12. Combined-stack benches on TP=4 / PP=1 / EP=4 / MTP-2, Intel AutoRound INT4 experts, FP16 dense, original BF16 PLE in CPU RAM — Community / Needs verify (full host stack, not an isolated QSA A/B):

The PR records a 31–33% 16k/32k prefill lift versus an immediately collected ~1529–1543 tok/s baseline on that same combined stack. Community take: near-PP4 prefill with TP4 decode. TP2+PP2 was not tried. Fused QSA multi-step draft decode with MTP-3 did not help. INT8 decode shadows are not in this merge. Prefer Hub v0.31.0-extras or latest rdna_extras (commits after the 11 Oct bake pin still want a host venv).

#vllm-rdna (Sep 21): do not treat the ~1.8–2.0k PP table as the public-merge floor. Two other 4× hosts on rdna_extras after #15 landed ~1.07–1.45k tok/s prefill. Decode often did reach the PR band (~50–77 t/s) once CUDA graphs were on (one host: ~24 t/s eager vs ~66–76 t/s with graphs). One of those hosts had octachannel DDR4-3200, PCIe gen4 x16 on all four cards, and NVMe ~6.5 GB/s — so the gap is not just “slow RAM / gen3”. A second host (DDR4-2144 ECC, PCIe gen3, SATA SSD, ~96 GB RAM, int4 PLE in RAM) sat around ~1.1–1.2k PP / ~45–50 t/s with MTP-1. The PR author later said the 1350 → 1550 → 2000 prefill steps needed two local changes that were not on the git checkout others pulled, and planned to re-apply those from current rdna_extras through git. #vllm-rdna (Sep 22): the author later recovered ~1950 tok/s PP on a local tree and said something was still missing before those changes could land on the public branch. Treat ~1.1–1.5k PP as the #15-only public-merge class.

#vllm-rdna (Sep 23): that missing stack merged as opengfx1030/vllm-rdna#17 into rdna_extras. PR-head 16k (same llm-context-bench harness): 1,957.8 / 1,982.8 tok/s PP, decode ~69 / ~69 t/s. A second 4× host confirmed those cells (1,983.8 / 1,985.3 tok/s PP) after rebuilding TunableOp for its rocBLAS — mismatch. The #15 author-host ~2k table is no longer “unpublished”; #17 shipped in Hub v0.31.0-extras. Community / Needs verify on your topology.

#vllm-rdna (Sep 2026) ballpark on 4× V620 (host-dependent; fork author + community):

Docs live under docs/rdna2/ (README.md, RESULTS.md, TROUBLESHOOTING.md, ROCR-CPU-FIX.md). Pull latest before re-benching — tags and container latest move with the prefill campaign.

Long-prompt stalls / timeouts: if large agentic prompts (tens of k tokens) hang or take many minutes while short prompts are fine — or GPUs sit at 100% util / ~40 W with TTFT swinging from seconds to minutes — try VLLM_USE_V2_MODEL_RUNNER=0. #vllm-rdna Sep 15: that switch unblocked a TP4 leapdragon container. Community: MoE on the 0.28 Flash-Next line wants the V1 runner until upstream 0.29 (V2 becomes the default). See vLLM troubleshooting.

KV / concurrency (4× 32 GB, #vllm-rdna Sep 14 + Sep 26–27): Flash-Next KV is expensive. Community ballpark ~24 GiB ≈ 300k tokens once the PLE sidecar is loaded — a 4× V620 box can still be tight (~280k tokens left on one host). The Sep 14 report that 128 GB host RAM offload was not enough (and that tool-call sessions re-prefilled) describes the pre-#24 / broken native path. Merged opengfx1030/vllm-rdna#24 (27 Sep) is the RAM-tier fix — community: three ~150k sessions while VRAM held ~270k, ~0.4 s to bring ~190k back, no full re-prefill on tool calls. PR capacity on a 48 GiB reserved tier: ~450k retained tokens. Needs verify on a second host; slower than fitting in VRAM. Hub tags lag. Details: native KV RAM offload. Concurrent streams still split decode (community: decode-only two streams often ~70–80 t/s; mixed prefill + decode drops decode to ~1–6 t/s and PP below 500). Prefer one stream plus prefix / radix cache unless you have measured overflow on #24. Merged #36 (5 Oct) adds opt-in VLLM_RDNA_DYNAMIC_PREFILL=1 so those interval / threshold knobs self-tune for mixed load — Needs verify, not a default serve recipe (troubleshooting).

#vllm-rdna (Sep 17): the logged hybrid KV token count can be ~2.5× too high versus a measured peak — size --max-model-len from a real request, not the banner (overstated pool). Prefix cache on rdna_extras needed e45dd5cb before repeat prompts actually hit (zero hits).

FA-RDNA2 vs QSA (#vllm-rdna 13 Oct): FA-RDNA2 is attention for any model that uses the normal backend selector — it is not dense-only. Flash-Next QSA full-attention layers bypass that selector (Triton QSA owner), so --attention-backend / VLLM_USE_RDNA2_FA only apply partially. Details: fork attention backends, troubleshooting.

Intel AutoRound Flash-Next (draft, #vllm-rdna Sep 10–11 2026)

A second Flash-Next track is the Intel W4A16 AutoRound checkpoint (Intel/Qwen3.8-Flash-Next-W4A16-AutoRound; community also cites the RTN sibling Intel/Qwen3.8-Flash-Next-W4A16-RTN-AutoRound). Weights are ~75 GB; plan on 4× V620. Community quality notes: Intel AutoRound beats some other INT4 Flash-Next packs on published tool-calling scores (example cited: 80.5 vs 76.0 for a different INT4). Treat those as publisher / community numbers, not wiki benches.

Draft integration: opengfx1030/vllm-rdna#5 (open, not in Hub -extras). PR validation used FP16 activations, CPU PLE / n-gram offload, and --max-num-batched-tokens 4096. Published short-run figures on 4× V620 (no P2P in the Discord report):

Gotchas (do not treat as a drop-in Hub image):

• Default PLE is still the embedded BF16 n-gram table (~95 GiB tensor data) with --engram-config '{"cpu_offload":true}'. The historical group-16 INT4 sidecar was not end-to-end validated (#vllm-rdna saw incoherent generation on that older path). Merged #49 (11 Oct, rdna_extras) plus the v0.31.0 launcher PLE=int4|int4-disk path. #announcements (11 Oct): all three PLE modes tested; speed at parity. Hub v0.31.0-extras includes this.
• --max-num-batched-tokens 2048 + chunked prefill was the suspected cause of the 128k PP cliff. Raise to 4096 (or see troubleshooting).
• Concurrent varlen GDN prefill still has a fork kernel addressing bug (gdn_prefill_o_rdna2.cu reuses a global chunk index). Single-stream benches can look fine.
• The PR is huge and dirty against rdna_extras — wait for a cleaned cherry-pick / image before calling it production. Day-to-day Flash-Next remains leapdragon/vllm-rdna2-qwen.
• #vllm-rdna (12 Oct): latest rdna_extras can reject this Intel pack at startup even when PLE is unquantized (INCConfig / Qwen4Exp PLE embedding). Upstream vLLM #59990 is merged; the fork cherry-pick was still in queue. Prefer wtdcode AWQ + primitive-ai PLE-quant until that lands — see troubleshooting.

Also watch TheRock ROCR idle-CPU spin on ROCm 7.14 hosts.

Serve commands (4× PIECEWISE, #17, PP3, overlays) are on Flash-Next serve lines.

Swift 1.5 Flash-Next (#vllm-rdna Sep 25–26)

Public shorter-reasoning derivative of Qwen3.8 Flash-Next (ukisai/Swift-1.5-Qwen3.8-Flash-Next-W4A16-AWQ; AutoRound sibling ukisai/Swift-1.5-Qwen3.8-Flash-Next-W4A16-AutoRound). Publisher card (BF16 vs base, not a wiki bench): ~63% fewer median thinking tokens at xhigh with under 1% claimed accuracy loss. AWQ is compressed-tensors W4A16 — same runtime class as other Flash-Next INT4 packs, not Hub -extras.

#vllm-rdna (Sep 25–26) community first-look on 3× V620: the AWQ pack ran at about the same speed as the host’s usual AutoRound Flash-Next line (~1300 tok/s prefill / ~45 t/s decode) and was called more verbose. Treat tok/s as Needs verify. Two-card fit is unconfirmed (community: “won’t fit”).

AutoRound caveat: the Swift AutoRound config is 50 iterations / light tuning (512-token calibration). Community compared that to Intel AutoRound’s 200 iterations and noted Intel’s attention projections stay bf16 while the Swift AutoRound pack quantized them to int4. #vllm-rdna (Sep 26): Swift AutoRound was ~5% faster than the host’s usual Flash-Next line — expected if the pack is more aggressively quantized, not a quality win. #vllm-rdna (Sep 27): the same AutoRound pack looped badly on a later host — treat that as a quality warning, not a speed footnote. Do not treat “AutoRound vs AWQ quality” as settled from Discord — read the quant config, and prefer the AWQ sibling if you want a first try. Community / Needs verify.

llama.cpp sibling: ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF (#llamacpp Sep 30–Oct 2). Community 2× V620, IQ3_xs-class + q4_0 ngrams in ~48 GB host RAM, MTP: ~400 PP / ~40 TG (4k/1k batch). #llamacpp Oct 2 IQ3_XXS + CPU per_layer_token_embd + MTP: ~800 → ~650 PP at ~30k / ~35 → ~20 TG (2k/1k batch; community called it VRAM-bound). Needs verify. See speculative table and serve line.

Related

• Serve lines
• Recipes — Hub -extras, Swift, and the TP=2 27B snapshot
• Quantization — MTP unbreak, AutoRound vs Swift
• Flash-Next troubleshooting
• Choose a stack

Flash-Next serve lines

WIP / community. Copy from here only after you have read Flash-Next status (which card count, which graph mode). Failures: Flash-Next troubleshooting.

Path C — Flash-Next (rdna_extras / Hub 0.31)

For agentic / long-context Flash-Next on 4× V620, use the Flash-Next vLLM stack — not the 27B recipe presets and not the old v0.27.1-extras* / v0.28.0-extras* Hub tags:

• Source of truth: opengfx1030/vllm-rdna rdna_extras is the v0.31.0 line (#announcements 11 Oct 2026). The old 0.28 aggregation lives at rdna_extra/v0.28.0. Hub blivioniag/vllm-rdna:v0.31.0-extras (ROCm 7.14) served Flash-Next MTP-2 (BF16 + INT4-disk).
• In-tree launcher: tools/rdna/serve_rdna.sh RECIPE=flashnext (production tools/reasoning) or RECIPE=flashnext-mtp2 / flashnext-mtp0. The old tools/rdna/serve_gfx1030_flashnext.sh name is a thin alias of RECIPE=flashnext. #vllm-rdna Sep 21 pointed 2-card / GPTQ questions at that file. Do not copy host paths out of the tree. Prefer the recipe over agent-generated env dumps — drop VLLM_USE_BREAKABLE_CUDAGRAPH=1 and do not force VLLM_RDNA_FUSED_HC=0 on Flash-Next (#vllm-rdna Oct 10).
• Published container: prefer Hub v0.31.0-extras. The older leapdragon/vllm-rdna2-qwen tree remains for legacy GHCR / docs/rdna2 (occasional commits through 11 Oct 2026).
• Weights: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16. #vllm-rdna (Oct 11): cyankiwi/Qwen3.8-Flash-Next-AWQ-INT4 produced incoherent output on one 4× host; switching to the wtdcode AWQ + primitive-ai PLE pair is the documented path.
• PLE / n-gram: launcher PLE=bf16|int4|int4-disk + PLE_QUANT_DIR=…/ples_int4 (primitive-ai/Qwen3.8-Flash-Next-PLE-quant). #announcements (11 Oct): all three modes tested; speed at parity. Docker disk mode: -v /path/to/ples_int4:/ples_int4 -e VLLM_PLE_QUANT_DIR=/ples_int4 -e VLLM_PLE_CPU_OFFLOAD=1. No-P2P hosts: P2P=0 on the launcher (or VLLM_RDNA_AR=0).

Expect large host DRAM for the BF16 n-gram table (~95 GB class). INT4 pinned is a ~30 GB table; #vllm-rdna (Oct 11) corrected an inflated ~90 GB host-RAM figure — INT4 sidecar working set is closer to ~35 GB. int4-disk mmaps shards (MADV_RANDOM) so only the gather/dequant working set hits RAM. That is weight offload, not KV. Native KV RAM overflow was broken on one 4× host (Sep 14: 128 GB not enough); the fix merged as #24 (27 Sep) — KV offload. Long-prompt / intermittent stalls: try VLLM_USE_V2_MODEL_RUNNER=0 — troubleshooting. Throughput and KV tightness: Flash-Next status.

#vllm-rdna (Sep 17): there is no Q3 / GGUF-Q3 path on vLLM. Stay on W4A16 / AWQ / AutoRound. EXL3 ~3 bpw is still experimental — Quantization.

4× V620 Flash-Next serve (rdna_extras #17, Sep 23)

opengfx1030/vllm-rdna#17 merged 23 Sep 2026 into rdna_extras. This is the public landing of the unpublished ~1950 PP recovery after #15. Community / Needs verify. Not a Hub tag. Do not copy host paths from the PR body — use the in-tree docs (docs/rdna2/V620-GIT-FAST-SERVICE.md, docs/rdna2/V620-BASELINE-PORT-20260922.md) and tunableop/.

Operational facts that are safe to copy:

• Resident W4A16 MoE (VLLM_RDNA_MOE_RESIDENT=1): convert the checkpoint’s packed INT4 experts into the RDNA2 kernel layout once at load, keep that layout, and skip repacking every 4k prefill chunk. Precision stays W4A16. #vllm-rdna (Sep 29): a latest-rdna_extras host saw PP fall from ~2k → ~1450 with this unset; setting 1 restored ~2k. Treat a sudden PP cliff after an extras pull as a missing-resident-MoE check before blaming kernels.
• Mamba retirement: backport of vllm#55450. Without it, long prompts (100k+ class) hit a deterministic preemption loop (state blocks not retired across null gaps).
• Graphs (#17 measured): --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[3,6,12]}' with MTP-2. Prefill stays eager.
• On current HEAD, prefer FULL_AND_PIECEWISE instead — see FULL_AND_PIECEWISE (decode uses the same FULL path as this baseline; piecewise covers mixed/prefill).
• TunableOp is rocBLAS-hash locked. A mismatched CSV aborts the first GEMM or silently uses default algorithms — regenerate.
• Harness used for the PR numbers: GeorgeMA-Strong/llm-context-bench.

PR-head 16k (author) vs second-host confirm (#vllm-rdna Sep 23):

4× V620 Flash-Next serve (FULL_AND_PIECEWISE, Sep 24–25)

Current recommended graphs on latest rdna_extras. #vllm-rdna (Sep 24): maintainer confirmed full + piecewise is fixed. Fork-source write-up: docs/rdna2/V620-FULL-AND-PIECEWISE.md.

What changed:

Safe compile line (generalized; do not copy host paths):

--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[3,6,12]}'

In-tree launcher: tools/rdna2/serve_v620_piecewise.sh (V620_COMPILE_MODE=3, V620_CUDAGRAPH_MODE=FULL_AND_PIECEWISE). Older comments in that script still mention the #20-era ~2× decode drop — ignore those once you are on HEAD with the keep-FULL commit. Prefer this over bare PIECEWISE (no FULL graphs) and over the Sep 17 breakable/mode:0 workaround.

Without compilation / without VLLM_USE_BREAKABLE_CUDAGRAPH=1, a FULL_AND_PIECEWISE request downgrades to FULL_DECODE_ONLY (keeps FULL decode) instead of dropping every graph; a bare PIECEWISE request still becomes NONE.

rdna_extras HEAD (25 Sep, 700753d9): MTP-2 boots after the amdsmi / local-argmax draft fix, but author-host decode stays below MTP-0 (~46–49 vs ~61 t/s). Do not expect MTP to beat the no-draft line yet — MTP.

The older Sep 17 PIECEWISE host-venv block below is only for checkouts before #17 / the keep-FULL fix.

4× V620 Flash-Next MonoKernel (#94, 16 Oct)

Merged opengfx1030/vllm-rdna#94 (16 Oct) into rdna_extras. A MonoKernel is a persistent HIP kernel that runs a large part of one Flash-Next (qwen4_exp) decoder layer in one launch (k1 hyper-connection preamble, GDN/QSA decode cores, k2 FFN). Prefill, mixed batches, the MTP drafter, and TP all-reduce stay on today’s path. Off by default. Hub v0.31.0-extras lags. Recipes do not pass the flag yet.

Startup refuses --monokernel unless all of these hold: gfx10x GPU, Qwen4Exp (MTP drafter accepted), V2 model runner, PP=1, _rocm_C ops built, and MoE on symmetric W4A16 (CompressedTensorsWNA16RDNA2MoEMethod). Asymmetric experts are a follow-up (open #115). PP3 MonoKernel is open backlog (#116) — do not combine this switch with RECIPE=flashnext-pp3*.

# Build the in-tree HIP bits (from an rdna_extras checkout + venv):
bash tools/rdna/mono/build_tree.sh

MODEL=/path/to/flash-next VENV=/path/to/venv \
  EXTRA_ARGS="--monokernel" \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

Same switch: VLLM_ROCM_MONO_DECODE=1 (alias of --monokernel; backport of vllm#61082). A/B without the flag in the same session. Keep GPU_MAX_HW_QUEUES=4 (Flash-Next TP4 recipes already set it — #107); two queues share HIP queues and skew ranks. In-tree notes: tools/rdna/mono/README.md and docs/rdna/monokernel/. Reproduce A/B write-up: open #120.

PR #94 same-session A/B (4× V620, TP4, PLE int4, FULL_AND_PIECEWISE, temperature 0, HSA_FORCE_FINE_GRAIN_PCIE=1, GPU_MAX_HW_QUEUES=4) — Fork-source. Cells are prefill tok/s/req / decode tok/s/req / TTFT s / median ITL ms. For MTP-2, prefer ITL (decode tok/s moves with drafter acceptance):

#vllm-rdna (16 Oct) author restated the decode-side win as about 8–10% ITL at c=1 (with or without MTP), 5–6% at c=8 without MTP, and 0–3% at c=8 with MTP-2. Prefill / TTFT were same or better in the PR table. Serves passed probe / PPL / prefix / concurrency gates.

Community / Needs verify (#vllm-rdna 16–17 Oct): a second 4× host called MonoKernel slightly slower than the current HIP path, with one GPU at 100% VRAM and prefill under 1200 tok/s on the same cells — TunableOp mismatch suspected; author-host prefill stayed above 2.1k. A mixed RDNA2/RDNA3 host saw about 5–10% after adding the switch (not a V620 recipe). If PP collapses after --monokernel, regenerate TunableOp before blaming the kernel — TunableOp.

Merged #126 (18 Oct) moves the MonoKernel QSA indexer (store / epilogue / scoring) to HIP. That is on with --monokernel on HEAD; it does not change the default path by itself.

HEAD HIP defaults (#130, 19 Oct)

Merged #130 (19 Oct) into rdna_extras (closes #99). Flash-Next recipes on HEAD now default the HIP set on: VLLM_RDNA_ALL_HIP, VLLM_RDNA_QSA_HIP, VLLM_RDNA_HC_PREFILL_HIP, and VLLM_GDN_HIP_PREFILL=1. EXL3 recipes are unchanged. Hub v0.31.0-extras lags. Set those vars to 0 to get the older Triton paths back.

Author-host 4× TP4 (Flash-Next AWQ, PLE int4): HIP-on ITL −6 to −8% and prefill +3–7% vs the previous default (no --monokernel). With --monokernel, MTP-2 ITL was −2 to −4%. Clone HEAD — do not expect this from the 11 Oct Hub bake.

Do not pass num_speculative_tokens_per_batch_size (merged #127 opt-in). #vllm-rdna (22 Oct): that schedule caused token-repeat loops. Fixed MTP-2 (RECIPE=flashnext-mtp2) stays the daily path. #vllm-rdna (25–26 Oct): loops can survive MTP off if CPU / disk KV offload is restoring sessions — disable --kv-offloading-size while isolating — troubleshooting.

4× V620 Flash-Next DFlash2 (#132, 21 Oct)

Merged #132 (21 Oct). Opt-in drafter path, not a replacement for MTP-2. Recipe enables --monokernel.

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-dflash2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

• Target: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16.
• Draft: incoai/Qwen3.8-Flash-Next-DFlash2 (BF16 — FP16 draft produced NaNs / near-zero acceptance).
• Draft attention is FA-RDNA2 BLHNC (noncausal sliding window). Target stays HIP QSA.
• Default 3 draft tokens ([4,8,16,32] capture). For the measured 16k / 1k c=8 ladder: DFLASH_TOKENS=2 ([3,6,12,24]). PRINT=1 shows the argv before launch.

PR paired A/B (4× V620, TP4, same boot): native MTP-2 faster in all four cells (tuned DFlash2 aggregate output −29 / −12 / −24 / −7% at 1k c=1 / c=8 and 16k c=1 / c=8). #vllm-rdna (22 Oct): maintainer still has MTP-2 ahead on agentic load; community: DFlash2 can win short / single-user, MTP-2 under concurrency. Fork-source for the recipe; community contrast Needs verify. Not Hub.

2× V620 Flash-Next MoE offload (#146, 28 Oct)

Merged #146 (28 Oct). Resident Flash-Next experts still do not fit 2× V620. This is the first in-tree vLLM spill path: immutable pinned-RAM expert banks, a bounded GPU slot cache, and segmented FULL_AND_PIECEWISE graphs around host planning. #vllm-rdna (28 Oct): V1 landed.

This is not a serve_rdna.sh RECIPE= yet. Use the offload wrapper on HEAD:

PRINT=1 MODEL=/path/to/flash-next PLE_QUANT_DIR=/path/to/ples_int4 \
  bash tools/rdna/mono/serve_offload.sh

• Target: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16 (compressed-tensors symmetric INT4, group 128).
• Default layout: TP=2 / EP=2, --monokernel, 96 expert slots per layer/rank, native CPU KV tier (KV_OFFLOAD_GB; 0 disables KV offload for an expert-only A/B).
• Slot policy knobs: SLOTS, POLICY (lfu_lru or lru), DECAY.
• Offload launcher defaults VLLM_RDNA2_W4A16_DETERMINISTIC_DENSE=1 (Q32 dense accumulation). Resident 4× recipes keep their existing default.
• Design notes: docs/rdna2/moe-offloading-target.md. Do not copy host paths out of that file.

Author-host 2× V620, MTP-2, 16 GiB native CPU KV (PR + #vllm-rdna 28 Oct): 16k/1024 c=1 ~19.8 tok/s aggregate (TTFT ~17 s, ~30 decode); c=8 ~26.7 agg (mean TTFT ~106 s, KV pool at 100% observed occupancy); 32k/256 c=1 ~30 decode / TTFT ~34 s. Fork-source on that host. The PR does not claim a controlled resident-vs-offload speedup. Not Hub. Needs verify on a second host.

Without HEAD, llama.cpp RAM/MoE offload is still the 2-card path — llama.cpp 2×.

INT4 KV cache (#138, 25 Oct)

Merged #138 (25 Oct). Opt-in longer KV on the non-MonoKernel QSA path:

--kv-cache-dtype int4_fp16_block

Packed INT4 + FP16 per-32-channel block scales; native QSA dequant in registers. Other attention backends refuse the dtype. Default FP16 KV unchanged. Not Hub. MonoKernel + this dtype Needs verify. #vllm-rdna (21 Oct): INT8 KV restore (draft #71) remains the stated day-to-day priority. Details: Quantization, Fork progress.

4× V620 Flash-Next serve (rdna_extras, PIECEWISE, Sep 17)

Sanitized from a #vllm-rdna (Sep 17) host-venv recipe on 4× V620. Community / Needs verify — not a published Hub tag. Requires opengfx1030/vllm-rdna rdna_extras including the GDN sanitizer commit 388a61b6f, and a rocm_sdk venv (torch 2.12 + ROCm 7.14). Paths below are generalized.

Why this differs from the Hub -extras TP4 block: Flash-Next on current rdna_extras FULL decode graphs still corrupt output. Community: switch --compilation-config to PIECEWISE, keep --max-num-batched-tokens 2048, and cap --max-num-seqs at 4–6. See FULL-graph corruption. Do not copy this graph mode onto the 27B -extras recipe unless you are A/B testing.

Community snapshot on that host (16k prompt / 1k gen, concurrency 8): ~3331 tok/s prefill, ~72.7 tok/s decode, TTFT ~39 s. Treat as a single-host number.

#vllm-rdna (Sep 17–18): pin rdna_extras at or after e45dd5cb if you want prefix cache to actually hit, and set VLLM_RDNA_DENSE_GEMV=1 if skinny wvSplitK faults after the profile run. VLLM_USE_BREAKABLE_CUDAGRAPH=1 is the stable / no-compile path (~39 t/s class in one A/B); 0 plus compile can be faster once graphs are clean — see FULL-graph troubleshooting. Do not trust the logged hybrid KV token count — overstated pool.

#vllm-rdna (Sep 18): maintainer posted an updated host-venv production block that still keeps VLLM_USE_V2_MODEL_RUNNER=0 (V2 blocked on Qwen4Exp), VLLM_USE_BREAKABLE_CUDAGRAPH=1, and --max-num-batched-tokens 2048, but:

• uses --compilation-config FULL_AND_PIECEWISE. On Sep 18 trees that mode still executed as PIECEWISE on ROCm (rocm_full_executes_as_piecewise). On current HEAD the keep-FULL fix turns that redirect off — FULL_AND_PIECEWISE. Still do not use FULL-only on old trees — FULL-graph corruption
• raises --kv-cache-memory-bytes to 7000000000 (7 GiB) and --gpu-memory-utilization 0.90
• reported a 16× 16k concurrency test as fine (earlier --max-num-seqs tightness)
• kills leftover processes before relaunch — VLLM::Worker, VLLM::EngineCore, and PleOffloadWorker can survive an API/port kill (troubleshooting)
• custom all-reduce / PIX is optional — hosts without GPU↔GPU P2P should drop that block (vLLM falls back to PYNCCL) — no P2P

Prefer Hub v0.31.0-extras (or a matching host-venv clone of rdna_extras) for this serve line. Hub v0.28.0-extras is only for historical A/B against older Sep 2026 pastes.

MODEL="wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16"
export VLLM_PLE_QUANT_DIR="$(python -c "from huggingface_hub import snapshot_download; print(snapshot_download('primitive-ai/Qwen3.8-Flash-Next-PLE-quant'))")/ples_int4"

export VLLM_PLE_CPU_OFFLOAD=1
export VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600
# Low-RAM hosts only: PLE_OFFLOAD_PREFAULT=0  (skips RAM prefault; ~25% slower — see env-vars)
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
export VLLM_FORCE_CUSTOM_ALL_REDUCE=1   # only if GPU↔GPU P2P works
export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_USE_AOT_COMPILE=0
export VLLM_DISABLE_COMPILE_CACHE=1
export VLLM_USE_BREAKABLE_CUDAGRAPH=1   # compile off; A/B 0 after output is clean
export VLLM_RDNA_DENSE_GEMV=1           # if wvSplitK faults after profile
export VLLM_RDNA_FUSED_HC=0
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export VLLM_RDNA_FORCE_FP16=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_ENABLED=1
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export PYTORCH_TUNABLEOP_FILENAME="${PYTORCH_TUNABLEOP_FILENAME:-/path/to/tunableop_results.csv}"
export VLLM_BATCH_INVARIANT=0
export GPU_MAX_HW_QUEUES=2
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_P2P_LEVEL=pix
export RCCL_P2P_NET_DISABLE=1
export RCCL_P2P_BATCH_ENABLE=1
export NCCL_PROTO=Simple
export RCCL_MSCCL_ENABLE=0
export HSA_FORCE_FINE_GRAIN_PCIE=1

cd /tmp   # avoid sys.path collisions with a local vllm checkout
python -m vllm.entrypoints.cli.main serve "$MODEL" \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --max-num-seqs 6 \
  --max-num-batched-tokens 2048 \
  --kv-cache-memory-bytes 5000000000 \
  --gpu-memory-utilization 0.88 \
  --dtype float16 \
  --trust-remote-code \
  --enable-prefix-caching \
  --language-model-only \
  --skip-mm-profiling \
  --enable-expert-parallel \
  --block-size 16 \
  --distributed-timeout-seconds 1800 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE","compile_ranges_endpoints":[]}'

Shared P2P / all-reduce env is the same as Hub -extras TP4. Drop the custom-AR block if P2P is broken.

Concurrency vs batch: #vllm-rdna (Sep 16): --max-num-batched-tokens 2048 keeps prefill turns ~1 s at ~1300 tok/s; 8192 can block other streams for ~6 s. Raise toward 4096 only if you hit the 128k prefill cliff on Intel AutoRound — that 4096 knob and this PIECEWISE stability recipe are different bugs.

Vision (#vllm-rdna Sep 17–21 + the in-tree launcher): Flash-Next recipes pass the vision-cap feature, which still applies a pixel cap. --limit-mm-per-prompt '{"image":1}' alone is not enough (count is already 1). Without max_pixels, mm-profiling can feed a huge dummy image (~24.8M px) and the vision encoder SDPA math backend materializes a ~64 GiB L×L fp32 score matrix — startup OOM on 32 GB cards. max_pixels=1605632 keeps images up to about 1424×1424. serve_rdna.sh bakes those two flags into vision-cap.

--limit-mm-per-prompt '{"image":1}' \
--mm-processor-kwargs '{"max_pixels":1605632}'

#vllm-rdna (Sep 21): vision works on rdna_extras with those flags; community ~15–20 s per image. Runtime is a second OOM: vLLM may not reserve vision memory, so a tight KV plus an image can OOM after a clean start. #general (Sep 21): vision + PP3 is especially tight. Keep --language-model-only --skip-mm-profiling if you do not need images.

Maintainer QA playbook (not a second wiki): BlivionIaG/vllm-rdna-qa.

3× V620 Flash-Next PP3 (rdna_extras HEAD)

#general (13 Oct 2026) + merged #96: PP3 / MTP-2 is in-tree on current rdna_extras. Hub v0.31.0-extras (11 Oct bake) does not include these recipes — clone HEAD + host venv + serve scripts.

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-pp3 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

MTP-2 needs a W4A16 quantized drafter (tools/rdna/quant_mtp_experts.py); bf16 draft experts do not fit the last stage:

MODEL=/path/to/mtpq-model VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-pp3-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

Recipe facts (fork comments + #general 13 Oct maintainer paste):

#vllm-rdna (17 Oct): later HEAD merges were not re-benched against this 13 Oct table — treat the numbers as a snapshot, not a current HEAD guarantee.

Author-host 16k/1k (#general 13 Oct, Community — not wiki-tested):

Also merged #95: int8 dense shadows are built in row chunks so load no longer allocates a ~2.4 GiB fp32 temporary. PLE still lives on stage 0 — PLE=bf16 wants ~95 GB host RAM; smaller hosts use PLE=int4 / int4-disk.

Older Sep 15 leapdragon / overlay notes stay below as historical. If an older pin still corrupts PP3 decode — PP3 corruption. Sep 19 pin b33f9b6 is a different failure (graphs boot, small prefill KeyError) — workaround.

3× V620 Flash-Next PP3 + MTP (leapdragon era, Needs verify)

#vllm-rdna (Sep 15): one host fitted Qwen3.8-Flash-Next + MTP on 3× V620 32 GB using a leapdragon image (W4A16 + PLE int4 CPU offload), PP=3 / TP=1. Treat as a single-host write-up — some steps were local source patches. Prefer the HEAD recipes when you can clone current rdna_extras.

Operational facts from that era that still match the HEAD recipes:

• Baseline PP3 without MTP sat at ~30 GiB/card with --gpu-memory-utilization 0.95.
• The MTP head is instantiated on the last pipeline stage only, so that stage OOMs first.
• Uneven layers leave the last stage light. The PLE offload worker can inherit VLLM_PP_LAYER_PARTITION, see pp_size=1, and refuse (len(partitions)=3 ≠ pp_size=1) — unset that env in the PLE worker process.
• VLLM_ROCM_MOE_PADDING=0 (do not pad routed-expert weights).
• Upstream vllm#46994 (MTP under pipeline parallel on the V2 runner) merged Sep 2026.

The old local int8 row-chunk rewrite is now #95 on HEAD. VLLM_RDNA_DENSE_INT8_ONLY=1 is on both flashnext-pp3* recipes.

3× overlay: leapdragon + org MoE HIP (Sep 17–18)

#vllm-rdna (Sep 17–18): a temporary public overlay (alanoo81/flashnext-v620-pp3) ports opengfx1030’s fused W4A16 MoE HIP kernel (moe_gptq_gemm_rdna2) onto a leapdragon Flash-Next image so 3× V620 / PP=3 can keep leapdragon CUDA graphs + MTP while picking up org-class prefill. Not a published Hub tag; treat as Community / Needs verify.

Public README snapshot (same harness; 160 W cap; AWQ-W4A16 + int4 PLE in host RAM):

Community: use MTP for one user; from two streams up, leave MTP off (plain graphs scaled better in that write-up). Also backports vLLM #46994 and #54044 (MTP + graphs + prefix cache). Follow the overlay README — do not copy host paths from Discord.

#general (Sep 23): the overlay author is no longer working on the repo. Community still reports useful 3-card numbers (one host ~45 t/s decode / ~1100 tok/s PP; overlay write-up 120+ t/s under multi-agent load, even ~200 t/s with 5–6 agents). Long context can trip infinite KV-pool dump loops — the author capped 3 concurrent agents on long prompts. Treat the overlay as unmaintained / snapshot; prefer rdna_extras once PP3 is clean there.

#vllm-rdna (Sep 19): that same HIP MoE kernel was reported non-deterministic (~3.5% of top tokens change between identical runs; Triton MoE did not). #vllm-rdna (Sep 30): an fp32 deterministic path was described as opt-in (throughput cost); no public flag name. See fork MoE note. Needs verify.

#vllm-rdna (Sep 18): the published Flash-Next checkpoint can leave the MTP draft head’s 512 experts in bf16, so speculative decode runs those through the generic Triton MoE kernel. The overlay README adds scripts/quant_mtp_experts.py (offline W4A16, RTN int4 g128, same packing as the main experts) so the draft experts hit the same HIP kernel. Community on 3× V620 PP3: +4–7% single-stream decode (~65–66 t/s), no acceptance loss (~70→75%), and +60% wall throughput at 8 streams (~77 → ~123 t/s). Community / Needs verify — run the script from the overlay repo; do not invent a Hub tag for the derived checkpoint.

DeepSeek-V4 Flash (community, Needs verify)

#vllm-rdna (Sep 17): public INT4-W4A16 weights yiminyuan/DeepSeek-V4-Flash-0731-INT4-W4A16 plus a gfx1030 vLLM tree yiminyuan/vllm @ gfx1030/v0.28.0. Publisher intent: TP=4 / PP=2. This is not Hub -extras and not wiki-tested. #llamacpp also mentioned a RDNA2 TP4 DeepSeek-V4 serve (~22 t/s, kernel unpublished) — Needs verify.

#vllm-rdna (Oct 10): on the current extras / v0.31 test line, DeepSeek-V4 is expected to run once mixed prefill/decode stalls are fixed (suspected CUDA-graph capture range too narrow). That is not a serve recipe yet — stay on the yiminyuan tree or wait for the stall fix. GLM-5.3 on the same fork was asked and not answered. See mixed prefill.

Related

• Flash-Next status — throughput, KV, AutoRound
• Serve scripts — RECIPE=flashnext-mtp2 and the other in-tree configs
• Hub -extras TP4 — the non-Flash-Next 27B block
• Configuration
• vLLM forks

vLLM Configuration

WIP: Env recipes and throughput numbers are community-reported. See Verification status.

Recommended environment (#vllm-rdna)

These settings are commonly used in the gfx1030 Discord for -extras images on ROCm 7.14:

export VLLM_TARGET_DEVICE=rocm
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export VLLM_RDNA_FORCE_FP16=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_ENABLED=0          # or 1 for autotuning (see compose below)
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export GPU_MAX_HW_QUEUES=2
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_BATCH_INVARIANT=0
export HIP_FORCE_DEV_KERNARG=1
export RCCL_MSCCL_ENABLE=0
export VLLM_USE_RDNA2_FA=1                  # extras images: native RDNA2 FlashAttention
export VLLM_USE_V2_MODEL_RUNNER=1           # +17% vs V1 reported on gfx1030 `-extras`
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export SAFETENSORS_FAST_GPU=1               # faster safetensors → GPU load (`#vllm-rdna` Sep 2026)

VLLM_USE_V2_MODEL_RUNNER=1 remains the usual default on Hub -extras. On the Flash-Next fork, long agentic prompts that stall or timeout with V2 enabled have been fixed in-community by setting VLLM_USE_V2_MODEL_RUNNER=0 instead — see Flash-Next long-prompt stalls.

GPU_MAX_HW_QUEUES=2 is the launcher default. Merged #107 (16 Oct): Flash-Next TP4 recipes (flashnext-mtp2 / flashnext-mtp0 / flashnext) set 4. That is not a Hub -extras compose change — leave 2 on the 27B Docker example unless you measure TP4 Flash-Next on HEAD. See env vars.

Custom all-reduce / P2P (two community stacks)

Pick one path — do not mix both:

If custom all-reduce misbehaves (stalls, bad latency on V620), fall back to the disable path. Some community recipes now run a short all-reduce self-test at boot and auto-fallback — pull latest recipe / fork notes.

Prefer --attention-backend RDNA_ATTN (or VLLM_USE_RDNA2_FA=1) over ROCM_ATTN on -extras. #vllm-rdna reports ROCM_ATTN sitting in AMD Triton flash-attention compile for hours (RCCL and Triton also fight each other). FA_RDNA2 may not show up on older -extras images or GPTQ models that still auto-select ROCM_ATTN — that is expected on hybrid GDN; see fork. On Flash-Next, QSA full-attention layers ignore these flags (Triton QSA path) — partial FA.

For multi-GPU TP on current -extras images, if AOT compile cache replay causes device-bound errors, add:

export VLLM_USE_AOT_COMPILE=0
export VLLM_DISABLE_COMPILE_CACHE=1

RDNA_ATTN / VLLM_USE_RDNA2_FA steer vLLM away from the generic AMD Triton flash-attention path, which can be slower or crash on some Qwen head sizes. See rdna_extras fork for kernel details. Full env cheat-sheet: Reference.

Don’t force backends or quantization unless you’re A/B testing — or avoiding a ROCM_ATTN Triton hang. Let vLLM read the model’s config.json unless that auto-selects the slow AMD Triton FA path.

CUDA graphs (preferred over --enforce-eager)

On current -extras images, CUDA graphs are the fast path — you generally should not use --enforce-eager. Graph capture can take a while on first boot (Triton JIT + torch-compile cache), but steady-state throughput is much higher once warmed up.

Recommended graph config (fork-author reported on Qwen3.8-27B-AWQ-INT4, TP4, 4× V620 — not wiki-reproduced):

vllm serve /path/to/model \
  --dtype float16 \
  --tensor-parallel-size 4 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --language-model-only \
  --skip-mm-profiling \
  --trust-remote-code \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Reported steady-state throughput on current -extras (1024/512 bench, TP4):

Alternative that also avoids --enforce-eager:

--compilation-config '{"mode": "NONE", "cudagraph_mode": "FULL", "compile_ranges_endpoints": []}'

To disable graphs entirely (debugging only):

--compilation-config '{"cudagraph_mode": "NONE"}'

If graph capture still crashes, fall back to --enforce-eager — but try the updated image first (docker pull blivioniag/vllm-rdna:v0.31.0-extras). See vLLM troubleshooting.

Flash-Next on rdna_extras (Sep 16–25): on current HEAD, prefer cudagraph_mode=FULL_AND_PIECEWISE with compile mode: 3 (capture e.g. [3,6,12] + MTP-2) — uniform decode uses the FULL graph; mixed/prefill use piecewise — serve notes. #17 FULL_DECODE_ONLY (mode: 0) remains a valid decode baseline. #20 alone still redirected FULL→piecewise (~26–34 t/s); keep-FULL commit 6c26c78d turns that off. Older Sep 16–17 hosts that saw FULL decode corrupt stayed on PIECEWISE — FULL-graph troubleshooting. The table above is the 27B -extras path, not Flash-Next.

VLLM_USE_BREAKABLE_CUDAGRAPH=1 in that recipe disables torch.compile (mode=NONE). Community Sep 17 A/B: keep it for the no-corruption path, then try 0 if you want compile speed back.

Cache volumes (first boot is slow)

Mount these so Triton and torch-compile artifacts persist across container restarts:

volumes:
  - ~/.cache/huggingface:/root/.cache/huggingface
  - ~/.triton/cache:/root/.triton/cache          # ~3 GB compiled Triton kernels
  - ~/.triton/dump:/root/.triton/dump
  - ~/.triton/llvm:/root/.triton/llvm
  - ~/.cache/vllm/torch_compile_cache:/root/.cache/vllm/torch_compile_cache  # ~700 MB

First startup compiles kernels and can take many minutes. Subsequent boots reuse the cache.

For recipe / Flash-Next stacks that want the compile cache (not the multi-GPU AOT-disable workaround above), #vllm-rdna (Sep 2026) recommends keeping the cache on and pointing it at a persistent directory:

export VLLM_DISABLE_COMPILE_CACHE=0
export VLLM_CACHE_ROOT=/path/to/persistent/vllm-cache   # mount this in Docker

Community reports ~5 min vs ~10 min subsequent startups once the cache is warm — still slow cold, but better than rebuilding every boot. Do not mix this with the TP VLLM_DISABLE_COMPILE_CACHE=1 workaround unless you have verified your image needs that disable path.

On Flash-Next long-context serves, if prefill falls off a cliff around 128k while decode stays flat, raise --max-num-batched-tokens to 4096 before blaming kernels — see 128k prefill cliff. On current rdna_extras Flash-Next TP serves, keep 2048 first if FULL graphs are corrupting — 4096 is the Intel AutoRound scheduler knob, not the graph-corruption fix.

SAFETENSORS_FAST_GPU=1 is also commonly set (and already present in some vllm-rdna Dockerfiles) to speed weight load into GPU memory — see AMD vLLM optimization notes.

Docker Compose example (GPTQ + MTP + CUDA graphs)

This #vllm-rdna setup reached ~24 output t/s on TP2 with a GPTQ model. Key points: GPTQ hits the native RDNA2W4A16LinearKernel, CUDA graphs via --compilation-config, and VLLM_DISABLED_KERNELS forces the RDNA2 quant path:

services:
  server:
    image: blivioniag/vllm-rdna:v0.31.0-extras
    network_mode: host
    ipc: host
    devices: [/dev/kfd, /dev/dri]
    group_add: [video, render]
    security_opt: [label=disable]
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - ~/.triton/cache:/root/.triton/cache
      - ~/.triton/dump:/root/.triton/dump
      - ~/.triton/llvm:/root/.triton/llvm
      - ~/.cache/vllm/torch_compile_cache:/root/.cache/vllm/torch_compile_cache
    environment:
      VLLM_TARGET_DEVICE: rocm
      VLLM_ROCM_USE_AITER: "0"
      VLLM_USE_RDNA2_FA: "1"
      PYTORCH_TUNABLEOP_ENABLED: "1"
      PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED: "0"
      VLLM_WORKER_MULTIPROC_METHOD: spawn
      GPU_MAX_HW_QUEUES: "2"
      VLLM_USE_V2_MODEL_RUNNER: "1"
      VLLM_DISABLED_KERNELS: ExllamaLinearKernel,TritonW4A16LinearKernel
    command: >
      serve btbtyler09/Qwen3.8-27B-GPTQ-4bit
      --served-model-name "Qwen 27B"
      --host 0.0.0.0 --port 8091
      --max-model-len 262144
      --gpu-memory-utilization 0.85
      --kv-cache-dtype float16
      --speculative-config '{"method":"mtp","num_speculative_tokens":4}'
      --tensor-parallel-size 2
      --dtype float16
      --max-num-seqs 4
      --language-model-only --skip-mm-profiling --trust-remote-code
      --enable-auto-tool-choice --tool-call-parser qwen3_coder
      --enable-prefix-caching --enable-chunked-prefill
      --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Confirm the kernel is active in logs: Using RDNA2W4A16LinearKernel for AutoGPTQLinearMethod.

vLLM Quantization on gfx1030

WIP: Throughput numbers are community-reported. See Verification status.

GPTQ vs AWQ

On older images (before the AWQ→RDNA2 dispatch fix), AWQ fell through to Triton/Exllama and could stall at ~4–5 t/s on a 27B. Pull the latest -extras image and confirm Using RDNA2W4A16LinearKernel in startup logs. Qwen3.8-27B AWQ also needs the fork’s head_size=256 FlashAttention path — without it, FA falls back or never lists RDNA_ATTN.

Kernel dispatch details: rdna_extras fork.

#vllm-rdna (14 Oct) + merged #98: RDNA2 W4A16 MoE no longer synthesizes symmetric zero points. Asymmetric compressed-tensors checkpoints (e.g. cyankiwi/*-AWQ-INT4, group 32, int8 zeros) used to miss the native experts kernel and fall back to generic Triton WNA16 MoE. Asymmetric 8-bit checkpoints still fall back. Confirm RDNA2W4A16MoEExperts (or dense RDNA2W4A16LinearKernel) in logs. Not in Hub v0.31.0-extras (11 Oct bake) — clone HEAD.

KV-cache dtype

Prefer float16. Do not plan on --kv-cache-dtype fp8 for QSA / Flash-Next until the extras fork re-enables a gfx1030 path. #vllm-rdna (Sep 23–24): community is testing INT8 KV via upstream quantized KV cache (RDNA2 has no FP8) aiming at more concurrent long context (e.g. 4× 256k). That is not the same as INT8 decode shadows, and QSA backends still do not list int8 KV — treat as Needs verify, not a recipe.

MTP speculative decoding

MTP (--speculative-config '{"method":"mtp","num_speculative_tokens":N}') can boost throughput on GPTQ models with CUDA graphs enabled. Acceptance rates dropped after a v0.27.1 speculator update (~0.25), but base decode speed remains good — worth testing on your model. Example in Configuration.

MTP is not free at high concurrency. A #vllm-rdna TP4 matrix on Qwen3.6-35B-A3B-FP16 (4× V620, --enforce-eager, 16k/1k-style bench) reported MTP-2 +17% output tok/s at c=1, but −53% at c=8. Use MTP for latency-critical single-stream; leave it off for batched throughput.

#vllm-rdna (Sep 15): RDNA2W4A16MoEExperts was reported to raise usable concurrency on MoE W4A16 — confirm the kernel in logs. Pipeline-parallel MTP needs upstream vllm#46994 (merged; next vLLM release). A 3× V620 Flash-Next + MTP squeeze is the in-tree flashnext-pp3-mtp2 recipe on HEAD (PP3); the older leapdragon overlay is historical. Hub v0.31.0-extras lags.

#vllm-rdna (Sep 18): on Flash-Next, MTP can block the int4 PLE fused decode path. If logs show Triton GDN / PLE fallback and decode is stuck in the teens of t/s, A/B MTP off before blaming P2P. Separately, the MTP draft head may still be bf16 MoE even when the main experts are W4A16 — quantize those draft experts offline if you stay on MTP (3× overlay). #vllm-rdna Sep 18: no MTP on the Hub v0.28.0-extras line; that work is aimed at the rdna_extra/v0.29.0 rebase.

rdna_extras HEAD 700753d9 (25 Sep) unbreaks MTP draft inference on gfx1030: amdsmi_shut_down() exceptions no longer mask a successful MoE config lookup (that used to kill the drafter on ranks 1–3 and deadlock NCCL), and Qwen4Exp MTP uses local-argmax draft sampling when speculative_config.use_local_argmax_reduction=true. Author-host 4× V620 W4A16 Flash-Next MTP-2 then boots (FA + FULL_DECODE_ONLY ~46 t/s; Triton + FULL_AND_PIECEWISE ~49 t/s) but stays below MTP-0 (~61 t/s) — documented ~0.73–0.85× penalty. #vllm-rdna (Sep 25): community still reports the same “boots, not faster than MTP-0” class.

30632b2 (27 Sep) captures the draft decode graph at max_num_reqs (MTP-2 c=8 was eager). Author-host: +18% at 8×1k / +5% at 8×16k; still ~0.87× vs MTP-0 at 16k. One later host with a matching TunableOp table reported MTP above MTP-0 — treat that as Needs verify, not a retraction of the 0.73–0.87× class. Present in Hub v0.31.0-extras; commits after the bake pin still want clone + host venv.

Merged #26: concurrent MTP-2 (9–16 rows) can stay on HIP MoE with VLLM_ROCM_MOE_SKINNY_MAX_M=16. Default remains 8. See prefill vs decode.

#vllm-rdna (Oct 8): on current rdna_extras MTP-2, if temperature > 0, set "draft_sample_method":"probabilistic" inside --speculative-config. The greedy / local-argmax draft path (700753d9) is the default; non-zero temp without the probabilistic sampler is the community-reported miss. Example:

--speculative-config '{"method":"mtp","num_speculative_tokens":2,"draft_sample_method":"probabilistic"}'

Same thread: day-to-day rdna_extras MTP-2 was ~60–90 t/s on one host (community; not a recipe). Hub v0.31.0-extras includes the MTP=2 path; A/B acceptance on your host.

INT4 on gfx1030 (no native int4 ALUs)

RDNA2 has no hardware int4 matrix units. The -extras W4A16 kernels use vdot2 on fp16 with on-the-fly dequant — int4 weights packed and processed via dp4a-style instructions. Both GPTQ and AWQ dense now hit the same native HIP kernel on current -extras images.

Recent fork work on hybrid GDN models (Qwen3.8-27B-AWQ-INT4, TP4) reported ~93 output tok/s with CUDA graphs (1024/512), ~331 total tok/s at 8 concurrent requests (16k/512), and prefill peaks of 1450–1573 tok/s — with the full HIP GDN prefill + decode chain replacing Triton JIT.

Tips

• Lower --gpu-memory-utilization (e.g. 0.9 → 0.8) if KV-cache allocation OOMs on 16 GB cards.
• For GPTQ or AWQ on -extras: set VLLM_DISABLED_KERNELS=ExllamaLinearKernel,TritonW4A16LinearKernel and watch logs for RDNA2W4A16LinearKernel.
• On older images, AWQ could fall through to Triton (~4–5 t/s on a 27B). Pull latest -extras and confirm the native kernel is active before blaming the quant format.
• Don’t force --attention-backend or --quantization — let vLLM auto-select unless A/B testing.
• Mount Triton and torch-compile caches (see Configuration).

Experimental: EXL3 and Quark (#vllm-rdna, Sep–Oct 2026)

#announcements (11 Oct 2026): EXL3 is listed among the v0.31.0 extras features, and the launcher ships RECIPE=27b-exl3. Older Hub tags (v0.27.1-extras*, v0.28.0-extras*) do not have that bake. Treat EXL3 as experimental even on 0.31 — mul1-tested path; A/B before production. AMD Quark is still not a drop-in on current -extras.

EXL3: 3inst only (#vllm-rdna, Sep 19)

#general (Sep 18) and #vllm-rdna (Sep 19): the gfx1030 EXL3 kernel on rdna_extras HEAD was 3inst, not mul1. 3inst is easier to execute at inference, not higher quality. Mixed bit-widths are still not wired: MTP at 6 bpw, lm_head at 8 bpw, and vision in bf16 will not load together. On published Hub -extras, keep uniform 3 bpw and the head in bf16.

#vllm-rdna (Sep 24): rdna_extra/v0.29.0 now implements mul1, so existing EXL3 packs can load. HIP EXL3 work is still in testing. Needs verify — do not assume Hub tags have this.

#vllm-rdna (Sep 29–30) + rdna_extras commits (30 Sep–1 Oct): EXL3 is a weight format — you do not need the exllamav3 runtime. mul1 decode / K=1..8 from rdna_extra/v0.30.0 is now on rdna_extras HEAD and GPU-run on gfx1030 (215/215 EXL3 tests; Qwen3.8-27B 3.00 bpw mul1 TP=4 FULL_AND_PIECEWISE + RDNA_ATTN coherence pass). Author-host 74315f4: 1k/512 ~18.9 t/s (c=1) / ~81.8 t/s agg (c=8); 16k/1k ~13.7 t/s / ~830 tok/s PP (c=1). Needs verify on other hosts. Not in Hub -extras. Draft #32 is the earlier kernel-port PR; the loader rewrite landed as follow-up commits on HEAD.

Stale inductor artifacts: VLLM_DISABLE_COMPILE_CACHE does not clear TORCHINDUCTOR_CACHE_DIR — troubleshooting.

#vllm-rdna (Oct 2): current rdna_extras HEAD is tested for mul1 only. The maintainer has not run 3inst. Prefer a mul1 pack on HEAD. Public community mul1 27B: quark75/Qwen3.8-27B-EXL3-3.0bpw (use with the 27b-exl3 serve recipe). A calibrated 3inst Flash-Next upload exists — BlivionIaG/Qwen3.8-Flash-Next-EXL3-3bpw-3inst — but was not GPU-tested on that date. Do not treat it as a recipe.

A mixed 3+6 bpw pack around 50 GB was called tight for 2× V620 and still needed calibration. Experimental kernels had only been tried on an Ornith 1.5 9B quant — not confirmed on 2× Flash-Next. #general (Sep 18) and #vllm-rdna (Oct 2): older 0.27 / 0.28 images can unpack an EXL3 download (one ~12.5 GB pack ballooned to ~54 GB) and OOM — size host RAM, not just the download. Flash-Next EXL3 is still not a 1× V620 path without heavy RAM offload.

An uncalibrated public 3inst upload was posted and immediately flagged for rework — do not treat it as a stable checkpoint: BlivionIaG/Qwen3.8-Flash-Next-EXL3-3bpw-3inst-uncalibrated. The later calibrated 3inst repo above is still Needs verify.

#vllm-rdna (27 Oct): EXL3 vs AWQ on current gfx1030 kernels — EXL3 wins size and quality; AWQ wins speed and complexity. Day-to-day testing still uses AWQ. EXL3 kernel work is paused in favor of a new RDNA-oriented weight format (BlivionIaG/hi2). Public README: rdna_hint_flex stores each row as INT2 or INT4 (INT8 on request) plus block scales, an optional smoother, and an order-16 Hadamard; you set a quality target and the tool reports the bit rate (--target-bpw 3.0 is the compact recipe). HIP kernels and the vLLM loader live in the vLLM RDNA2 fork, not in the hi2 repo. #vllm-rdna (30 Oct): 3 bpw was called a bit aggressive; 3.5 bpw may hold quality better but QAD is slow. There is no vLLM RECIPE= / Hub bake yet. Do not treat hi2 as a serve path.

Prefer GPTQ/AWQ on published images until EXL3/Quark land in a tagged Docker build.

#vllm-rdna (Sep 29): W4A8 started as an explore path (public opengfx1030/vllm-rdna#9, w4a8-wiring branch). #vllm-rdna (Oct 1): W4A8 is in rdna_extras HEAD (VLLM_RDNA2_W4A8_SDOT4 / serve-script W4A8=1) but was not much faster than W4A16 dequant-on-the-fly — the stated win then was more context, not decode. #vllm-rdna (Oct 9): a later kernel on the v0.31.0 test line was called somewhat competitive vs W4A16. Merged #140 (26 Oct): 27b-awq recipe defaults W4A8=1 on HEAD (int8-activation prefill GEMMs for M≥33; decode stays W4A16). Author-host 4× TP4: 16k c=1 PP 860 → 1075 tok/s; c=8 aggregate out 42.5 → 52.3; paired PPL ~unchanged. Fork-source on that host; Hub v0.31.0-extras may still leave the knob off — check PRINT=1. Set W4A8=0 to force W4A16 prefill. EXL3 still needs optimizations. W8A8 was called theoretically the fastest int8 layout, not a current recipe.

INT4 KV cache (int4_fp16_block, HEAD)

Merged #138 (25 Oct): opt-in --kv-cache-dtype int4_fp16_block — packed INT4 payload + FP16 per-32-channel block scales (plus a 4-bit zero-point in the low nibble of each scale word). The RDNA2 QSA path reads/writes natively (dequant in registers into the same FP16 LDS tile). Backends that cannot pack the record refuse the dtype up front. Default FP16 KV is unchanged.

Not in Hub v0.31.0-extras. MonoKernel + this dtype still Needs verify. #vllm-rdna (21 Oct): maintainer priority for longer context is still restoring the INT8 KV HIP path (draft #71), not replacing day-to-day FP16 with experimental packs. Snapshot: Fork progress.

Intel AutoRound W4A16 (Flash-Next, #vllm-rdna Sep 2026)

Not in published -extras tags. Community + draft opengfx1030/vllm-rdna#5 are exercising Intel W4A16 AutoRound Flash-Next:

Community take: Intel AutoRound sits between Unsloth Q5_XL and Q6_XL on quality while the weight pack is tens of GB smaller (~75 GB). That still wants four 32 GB cards plus a large CPU PLE / n-gram table (embedded BF16 table is ~95 GiB — offload to RAM).

Do not assume the historical group-16 INT4 PLE sidecar matches published AutoRound benches (those used BF16 PLE). Merged #49 (11 Oct) plus the v0.31.0 launcher (PLE=int4|int4-disk) is the current INT4 path. See overview.

#vllm-rdna (12 Oct): latest rdna_extras can fail at construction on this Intel pack (NotImplementedError / INCConfig for the unquantized PLE table). Upstream vLLM #59990 is merged; the gfx1030 cherry-pick was still queued. Troubleshooting.

#vllm-rdna (Sep 20): merged opengfx1030/vllm-rdna#15 used this Intel AutoRound pack with the unquantized / original BF16 PLE table (~100 GB class in RAM), not wtdcode AWQ + primitive-ai PLE-quant. INT8 decode shadows remain experimental — they are excluded from that merge (one host reported ~42 → ~55 t/s without MTP; Needs verify, and quality loss was already seen on Sep 16).

vLLM forks (RDNA2 / gfx1030)

WIP: Kernel behavior below is traced to fork source commits; published -extras Docker images may lag HEAD until rebuilt. See Verification status.

Fork landscape

#vllm-rdna (Sep 2026): the official gfx1030 vLLM source is the shared-org repo. File new PRs and issues there. Flash-Next work lives on rdna_extras. Published Docker Hub v0.31.0-extras landed 11 Oct 2026.

Day-to-day serving: pull blivioniag/vllm-rdna v0.31.0-extras (#announcements 11 Oct 2026). Source of truth for new kernel work is opengfx1030/vllm-rdna rdna_extras (now the v0.31.0 port). Older Hub tags stay for A/B. Do not assume org GHCR tags exist yet.

Companion (not an engine fork): leapdragon/vllm-rdna2-recipe (mirror opengfx1030/vllm-rdna2-recipe) collects compose/env recipes and open PRs (concurrent MTP, etc.). Wiki entry point: Recipes. #vllm-rdna (Sep 2026): the Flash-Next author treats the recipes repo as a parts pile (less active new work). Day-to-day Flash-Next optimization lives on org rdna_extras (Hub v0.31.0-extras); Concurrent-MTP PRs on the recipes repo are still worth cherry-picking into that stack.

Progress snapshot

For a dated Hub vs HEAD vs backlog table (MonoKernel, HIP defaults, PP3, INT4 KV, W4A8, open drafts), see Fork progress (26 Oct 2026). The chronicle below keeps per-PR detail.

Consolidation status (Sep 3–26 2026)

#vllm-rdna / public merges chronicle — expect this to move quickly:

Official-fork feature set called out in-channel (HIP): MXFP4, AWQ INT4, GPTQ INT4, GDN prefill/decode linear attention, FA-RDNA2, one-shot all-reduce, FULL_AND_PIECEWISE, EXL3, MTP=2. Treat Hub v0.31.0-extras as the current extras bake; clone HEAD for work after that tag. Readable snapshot: Fork progress.

The rdna_extras fork

The official extras source is opengfx1030/vllm-rdna on the default rdna_extras branch (formerly personal blivioniag/vllm rdna2_extras — public repo 404 today). It adds hand-written RDNA2 HIP kernels and the vLLM plumbing to dispatch to them.

Current Hub extras bakes (v0.28.0-extras, v0.31.0-extras) clone this org branch via vllm-rdna-docker (VLLM_REPOSITORY=…/opengfx1030/vllm-rdna.git, VLLM_REF=rdna_extras). Older v0.26 / v0.27.1-extras* bake targets still name the removed personal BlivionIaG/vllm @ rdna2_extras clone. Pull Hub v0.31.0-extras for serving; clone the org repo to contribute.

Why it exists: RDNA2 (gfx1030) has no matrix/WMMA cores — those arrived with RDNA3 (gfx11xx). So quantized GEMM, attention, and MoE have to be implemented efficiently on RDNA2’s regular vector ALUs. This fork does exactly that (with RDNA3 WMMA variants where it makes sense).

INT4/W4A16 on gfx1030 does not use native int4 compute. The RDNA2 kernels dequant on the fly using vdot2 fp16 instructions with int4 packed via dp4a. Community testing found no faster INT8-based HIP kernel vs fp16 for these paths — fp16 dequant is the winning approach on V620.

Branches

Historical names on the old personal fork (feat/enable-gfx1030, perf/rdna2_w4a16, rdna2_extras) are superseded. Only the older v0.26 / v0.27.1-extras* bake targets still name that removed repo; current extras bakes use VLLM_REF=rdna_extras on opengfx1030/vllm-rdna.

What’s in it

The custom device code lives under csrc/rocm/*_rdna2.cu (with _rdna3 variants where WMMA applies), wired into vLLM through Python kernel/layer modules and covered by targeted tests under tests/kernels/.

Attention

• fa_rdna2.cu — a FlashAttention kernel tuned for RDNA2, exposed via vllm/v1/attention/ops/fa_rdna2_backend.py and the vllm/v1/attention/backends/rdna_attn.py backend. Now supports head_size=256 on gfx10x (needed for Qwen3.8-27B-AWQ-INT4; previously capped at 128 and caused Triton GDN compile hangs). #vllm-rdna (Sep 22–24): when the extras fork went public the maintainer dropped the FP16 FA pieces to focus on validating W4A16. In-tree fa_rdna2 is still the org target, but it is currently disabled / not working after the #17 stack changes (prefix-cache + piecewise first). Independent 2kiss/flash-attention-rdna2 is stale — do not treat it as a drop-in. Needs verify.
• sparse_mla_rdna2.cu + rocm_rdna2_mla_sparse.py — sparse MLA (multi-head latent attention, DeepSeek-style).
• indexer_paged_mqa_rdna2.cu — paged MQA indexer.

Quantized GEMM (mixed-precision linear)

• q_gemm_rdna2.cu / q_gemm_rdna2_prefill.cu (+ q_gemm_rdna2_common.cuh) — W4A16 quantized GEMM for decode and prefill.
• q_gemm_w8a16_fp8_rdna2.cu, gemm_w8a8_fp8_dense_rdna2.cu — FP8 W8A16 / W8A8 paths.
• qdq_4_rdna2.cuh, qdq_8_rdna2.cuh, qdq_fp8_rdna2.cuh — quant/dequant helpers.
• Python: model_executor/kernels/linear/mixed_precision/rdna2_w4a16.py, .../scaled_mm/rdna2_w8a16_fp8*.py, rdna2_w8a8_fp8.py, and the rdna_hybrid_w4a16.py selector.
• EXL3 (in-tree on rdna_extras; #announcements 11 Oct 2026 lists it on the v0.31.0 extras bake / RECIPE=27b-exl3): exl3_hadamard.cu, exl3_dot2_*.cu, plus vllm/.../quantization/exl3.py. Still experimental. #vllm-rdna (Sep 24): rdna_extra/v0.29.0 added mul1 so existing EXL3 packs can load. #vllm-rdna (Sep 29–30) + rdna_extras commits (30 Sep–1 Oct): mul1 / K=1..8 plus the v0.30 loader rewrite are on HEAD and GPU-run on gfx1030 (215/215 tests; 27B 3.00 bpw mul1 TP=4 graphs). #vllm-rdna (Oct 2): that path is mul1-tested only — 3inst packs are untested. Draft #32 was the earlier kernel-port PR. Older Hub v0.27.1-extras* tags do not have this — Quantization.

MoE (mixture of experts)

• moe_q_gemm_rdna2.cu, moe_w8a16_rdna2.cu, moe_w8a16_fp8_rdna2.cu — quantized expert GEMMs.
• Python experts fused_moe/experts/rdna2_mxfp4_moe.py, rdna2_w8a16_fp8_moe.py, and compressed_tensors MoE glue (..._fp8_rdna2, ..._w4a4_mxfp4_rdna2, ..._wna16_rdna2).

#vllm-rdna (Sep 19): a quality A/B reported the fused HIP MoE kernel is not deterministic. On the same server and the same tokens, about 3.5% of the most-probable tokens changed between runs, costing about +0.2–0.3% perplexity versus the best reference that host could run. The Triton MoE path was bit-identical across repeats. Cause not isolated (fp16 accumulation order vs a real race). Proposed, not landed as of Sep 19: replace the atomic_add_pk4_f16 CAS loop with global_atomic_add_f32 (fewer atomics, possibly worse bandwidth). #vllm-rdna (Sep 30): an fp32 deterministic MoE mode was described as opt-in because it cuts throughput — flag name was not posted; do not invent an env var. Default stays the faster path. Needs verify — do not switch kernels on this note alone.

GDN (gated delta-net / linear attention)

Kernels for gated-delta-net models (e.g. Qwen3.8-27B hybrid linear attention). As of Aug 2026 the full GDN chain is hand-ported to HIP — no Triton JIT on the hot path:

This replaces the Triton FLA GDN path that previously caused 16k/1k prefill hangs and slow decode on hybrid models. Recent tuning commits improved delta_h register pressure, o-kernel block-vector width, and prep-kernel vectorization.

Using it

The easiest path is a prebuilt -extras image — no compilation required:

docker run -it --rm \
  --device /dev/kfd --device /dev/dri --group-add video --group-add render \
  --security-opt seccomp=unconfined --ipc host -p 8000:8000 \
  docker.io/blivioniag/vllm-rdna:v0.31.0-extras \
  vllm serve <model> --dtype float16

See Running (Docker) for the full run recipe and Building images for how the -extras variant is produced (VLLM_VARIANT=extras-fork, current extras targets VLLM_REF=rdna_extras on opengfx1030/vllm-rdna).

Kernel dispatch on gfx1030

On -extras images, vLLM picks kernels based on quantization format:

Check startup logs for lines like Using RDNA2W4A16LinearKernel for AutoGPTQLinearMethod. If you see TritonW4A16LinearKernel or ExllamaLinearKernel instead, the RDNA2 quant path isn’t active.

Attention backends

• VLLM_USE_RDNA2_FA=1 — enables the custom fa_rdna2.cu FlashAttention backend on Hub -extras. #vllm-rdna (Sep 15): on current Flash-Next / rdna_extras HEAD, native FA is still WIP — community A/B used VLLM_USE_RDNA2_FA=0 (and sometimes VLLM_GDN_HIP_PREFILL=0) when chasing PP3 correctness. Do not copy that disable onto a working 0.27.1 -extras 27B serve.
• --attention-backend RDNA_ATTN — alternative RDNA-tuned attention path (useful for Qwen models with head size 256 where generic AMD Triton FA is slow or broken).
• FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE — enables AMD Triton FA as a fallback; often slower on gfx1030.
• Optional FA-RDNA2 knobs on rdna_extras (rdna_attn.py, Oct 2026): VLLM_FA_RDNA2_GQA_MODE (default subgroup), VLLM_FA_RDNA2_SPLIT_DECODE (default 1). A/B only — not a serve recipe.

#vllm-rdna (13 Oct): FA-RDNA2 is general attention, not “dense models only”. On Flash-Next, full-attention layers are owned by the QSA path (Qwen4ExpQSA*), which picks its Triton backend directly and does not consult --attention-backend / VLLM_USE_RDNA2_FA. Seeing Triton in the stack for those layers is expected. A full switch would need HIP QSA split-K attention plus INT4 PLE n-gram lookup — still open work (community cited fork backlog around #64). Prefer not treating FA flags as a Flash-Next speed lever until that lands — troubleshooting.

On v0.27.1, hybrid GDN models may still auto-select ROCM_ATTN even with VLLM_USE_RDNA2_FA=1 for the attention layers — that’s expected. The GDN linear-attention layers now use the native HIP kernels above (not Triton FLA). For graph capture issues, see CUDA graphs and vLLM troubleshooting.

CUDA graph capture (TP comm fix)

A common _SimpleCData.__new__ crash during V2 cudagraph capture on multi-GPU TP setups was fixed by marking TP communication wrappers with allow_in_graph. With current -extras images, CUDA graphs are the preferred fast path — you should not need --enforce-eager for this class of failure anymore. On multi-GPU, if AOT compile cache replay misbehaves, try VLLM_USE_AOT_COMPILE=0 VLLM_DISABLE_COMPILE_CACHE=1.

Disabling fallback kernels

export VLLM_DISABLED_KERNELS=ExllamaLinearKernel,TritonW4A16LinearKernel

This is the main lever for forcing GPTQ onto the native RDNA2 W4A16 path. The variable accepts a comma-separated list of kernel class names registered in vLLM’s linear-kernel registry.

To test Exllama instead (still relevant on some quants per #general):

--linear-backend exllama

Omit VLLM_DISABLED_KERNELS when testing Exllama — it competes with the RDNA2 kernel for dispatch.

Building from source (advanced)

Create the venv and install torch first — the ROCm 7.2.0 and 7.14.0 commands are on Host venv. Dockerfile.vllm then does an editable install (not requirements/dev.txt):

git clone -b rdna_extras https://github.com/opengfx1030/vllm-rdna.git
cd vllm-rdna
export PYTORCH_ROCM_ARCH=gfx1030
export ROCM_HOME="${ROCM_HOME:-/opt/rocm}"
VLLM_USE_PRECOMPILED=0 ENABLE_CK=0 FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
  uv pip install -e . --no-build-isolation

On ROCm 7.2.0 only, add --torch-backend=rocm7.2. On 7.14.0, leave it off and install the multi-arch device wheels first. Details: Host venv.

The kernels have their own tests, e.g.:

pytest tests/kernels/quantization/test_rdna2_w4a16.py
pytest tests/kernels/attention/test_fa_rdna2_shape_sweep.py

These kernels are actively evolving. Treat the fork as experimental, pin to a known-good image tag for reproducible serving, and file issues on opengfx1030/vllm-rdna if you hit correctness or performance problems.

vLLM fork progress (RDNA2 / gfx1030)

WIP snapshot: Readable status of opengfx1030/vllm-rdna rdna_extras. Dated 31 Oct 2026 from public merges + #vllm-rdna / #announcements. Kernel file lists and older chronicle rows stay on Fork landscape. See Verification status.

How to read this

Published Hub tags lag HEAD. If a row says “Not Hub”, do not expect it on blivioniag/vllm-rdna:v0.31.0-extras until a newer extras bake.

Shipped in Hub v0.31.0-extras (11 Oct)

#announcements (11 Oct 2026): rdna_extras = upstream vLLM 0.31.0 + RDNA port. Bake: ROCm 7.14, torch 2.12, Triton 3.7.1.

Author-host 4× V620 TP=4 (probes / PPL / prefix cache / concurrency, 0 GPU faults): Flash-Next AWQ ~2000 PP, 45 (MTP-0) / 69 (MTP-2) TG, ~205 agg @ 8 users — Fork-source. Details: fork.md v0.31 row.

On HEAD since the bake (13–28 Oct)

Prefer these over Hub when you need the feature. Serve lines: Flash-Next serve, Serve scripts.

Flash-Next / serving

Correctness / graphs / AR

Memory / dense / KV

Tooling / cleanup (HEAD hygiene)

#97 prefix-cache off flag; #100–#106 / #88 dead-path cleanup; #92 TunableOp grid expand; #141 stale-doc / lint fixes.

Open backlog (selected)

Do not treat drafts as recipes. Full list: open PRs on rdna_extras.

#vllm-rdna (21 Oct): community DeepSeek-V4-Flash correctness write-up on a fork of rdna_extras (bf16→fp16 cast, logits all-gather, EP routing). In-tree RECIPE=dsv4-flash still Needs verify — serve.

Practical defaults (31 Oct)

Related

• Fork landscape + chronicle — repos, kernels, full consolidation table
• Flash-Next / Serve lines
• Serve scripts
• Verification

Host venv (no Docker)

WIP: Commands below follow opengfx1030/vllm-rdna-docker (Dockerfile.base, Dockerfile.vllm, docker-bake.hcl) and the opengfx1030/vllm-rdna rdna_extras tree. See Verification status.

Stable day-to-day Docker is Hub v0.31.0-extras — Onboard / Running (Docker). Use this page when you want kernels newer than that bake, serve_rdna.sh, or a host Python 3.12 uv venv with no image rebuild. You need a ROCm toolchain with hipcc. For more than one GPU, pin ROCm 7.2.0 or 7.14.0 unless you are deliberately lab-testing ROCm 10 — Installing ROCm. #vllm-rdna (29 Oct): a fresh Ubuntu 26.04 host can ship /opt/rocm/core-10 (and core-10.1) with no core-7.14 tree. Do not copy the 7.14 ROCM_HOME path blindly. Community: ROCm 10 serves; the maintainer has not pinned a Triton wheel for that userspace yet. #vllm-rdna (30 Oct): ROCm 10 and 7.14 were called the same for day-to-day serving — do not reinstall just to chase 10.x. One 10.1 host-venv attempt failed (libnuma / optional HIP package path mess) before a clean 7.14 retry — Needs verify. Stay on the bake pins below unless you are A/B-ing 10.x. TunableOp already has a thin ROCm 10 profile — serve scripts.

The image bake lists seven gfx targets. A gfx1030-only venv can set PYTORCH_ROCM_ARCH=gfx1030.

ROCm 7.2.0

#general (1 Aug 2026) plus bake target base-rocm720. Wheels come from the PyTorch ROCm 7.2 index. That index does not publish rocm-sdk-device-* or amd-torch-device-* — do not add them.

uv venv --python 3.12 venv-7.2.0
source venv-7.2.0/bin/activate

uv pip install --pre --force-reinstall \
  --index-url https://download.pytorch.org/whl/rocm7.2 \
  "torch==2.12.0" "triton==3.5.1" torchvision torchaudio

The image pins triton==3.5.1. The same #general thread also had triton-rocm==3.7.1 installed beside it. Triton is a wheel either way — do not build it from source for this path.

Then build FlashAttention, causal-conv1d, and vLLM. On 7.2.0 only, add --torch-backend=rocm7.2 to the vLLM uv pip install.

ROCm 7.14.0

Do not swap the 7.2 index for https://download.pytorch.org/whl/rocm7.14. On Python 3.12 x86_64 that index has no matching torchaudio wheel (#vllm-rdna, 21 Sep 2026).

Bake target base-rocm714 uses the multi-arch index and then installs the per-GPU device packages. triton==3.7.1 on that index is a wheel (triton-3.7.1+git….rocm7.14.0), not a source build. gfx103X still has no aotriton, so PyTorch’s SDPA flash path stays off; that is separate from the Triton package.

docker-bake.hcl pins torch 2.12.0+rocm7.14.0 for base-rocm714. The docker-repo README still claims PyTorch 2.13.0 for published 7.14.0 — that README cell is stale; the image matrix matches the bake (2.12.0). Recreate with the pins below; if you are matching an image you already pulled, use whatever torch.__version__ that container prints (a true 2.13 wheel would pair with torchvision==0.28.0+rocm7.14.0 and amd-torch-device-gfx1030==2.13.0+rocm7.14.0 on the same index).

#vllm-rdna (17 Oct): Triton 3.9+ removes RDNA1/RDNA2. Stay on the bake pin (triton==3.7.1 here, 3.5.1 on ROCm 7.2). Do not uv pip install -U triton. The rdna_extras HIP path is the long-term escape; a 3.9 wheel will not grow gfx1030 support back.

uv venv --python 3.12 venv-7.14.0
source venv-7.14.0/bin/activate

uv pip install --pre --force-reinstall \
  --index-url https://repo.amd.com/rocm/whl-multi-arch/ \
  "torch==2.12.0+rocm7.14.0" \
  "torchvision==0.27.0+rocm7.14.0" \
  "torchaudio==2.11.0+rocm7.14.0" \
  "triton==3.7.1"

uv pip install \
  --index-url https://repo.amd.com/rocm/whl-multi-arch/ \
  "rocm-sdk-device-gfx1030==7.14.0" \
  "amd-torch-device-gfx1030==2.12.0+rocm7.14.0"

What [device-gfx1030] is

On a multi-arch index package, torch[device-gfx1030] is only a shortcut for the two wheels above (amd-torch-device-gfx1030 and rocm-sdk-device-gfx1030). The Docker file installs those packages by name. It does not pass the extra.

Without them, an index torch looks for code objects under the venv’s _rocm_sdk_* tree and kernel launch fails with hipErrorInvalidImage (Dockerfile.base). /opt/rocm/lib/rocblas/library is not the tree that wheel consults.

Wheels you built yourself

leapdragon/vllm-rdna2-qwen tools/rdna2/build-torch-rocm714.sh compiles PyTorch, Triton, and torchvision with PYTORCH_ROCM_ARCH=gfx1030. Those .whl files already are gfx1030 builds. They have no device-gfx1030 extra. Passing it makes uv warn and does nothing — for example torch==2.12.0+git6bbd260 and torchvision==0.27.1+… from ~/wheels/rdna2/.

Install the files. Do not add amd-torch-device-* on top:

uv pip install ~/wheels/rdna2/torch-*.whl \
  ~/wheels/rdna2/triton-*.whl \
  ~/wheels/rdna2/torchvision-*.whl

Library path (index / rocm-sdk venv)

The 7.14 image writes these into ldconfig. A host venv needs them in the environment before vllm serve (#vllm-rdna serve blocks). Skip this if you installed a from-source gfx1030 wheel and have no _rocm_sdk_* directories.

SP="$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
export LD_LIBRARY_PATH="$SP/_rocm_sdk_libraries/lib:$SP/_rocm_sdk_core/lib/host-math/lib:$SP/_rocm_sdk_core/lib/rocm_sysdeps/lib:$SP/_rocm_sdk_core/lib/core/lib:$SP/torch/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
export ROCM_HOME="${ROCM_HOME:-/opt/rocm}"
export HIP_PATH="$ROCM_HOME"

Point ROCM_HOME at the tree that contains bin/hipcc. The image base is rocm/dev-ubuntu-22.04:7.14.0-full (/opt/rocm). A TheRock layout is often /opt/rocm/core-7.14.

Do not pass --torch-backend=rocm7.14 when installing vLLM. That flag selects the PyTorch rocm7.14 index, which is the one missing the 3.12 x86_64 torchaudio wheel.

Build into the venv

Same three steps on 7.2.0 and 7.14.0, after torch is importable. Dockerfile.vllm is the source for the flags and the causal-conv1d pin. The 1 Aug #general paste left the FlashAttention clone as a placeholder and did not pin causal-conv1d; the image build fills both in as below.

FlashAttention

gfx1030 is not on the Dao-AILab allowlist (native, gfx90a, gfx942, gfx950, gfx1100–gfx1201). Leave it out of GPU_ARCHS. gfx1030 uses Triton FA at runtime (FLASH_ATTENTION_TRITON_AMD_ENABLE).

git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git submodule update --init
export ENABLE_CK=0
export PREBUILD_KERNELS=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
GPU_ARCHS="gfx1100;gfx1101;gfx1150;gfx1151;gfx1200;gfx1201" \
  uv pip install --no-build-isolation .
cd ..

causal-conv1d

HIP_ARCHITECTURES is the variable this commit reads (not GPU_ARCHS). The image pins 4f6ae4e26ae5fe8af9372f8d312ab25cc4595223.

CAUSAL_CONV1D_FORCE_BUILD=TRUE \
HIP_ARCHITECTURES=gfx1030 \
ROCM_PATH="${ROCM_HOME:-/opt/rocm}" \
  uv pip install --no-build-isolation \
  "git+https://github.com/Dao-AILab/causal-conv1d.git@4f6ae4e26ae5fe8af9372f8d312ab25cc4595223"

vLLM (rdna_extras)

uv pip install -e . --no-build-isolation is what Dockerfile.vllm runs. It does not install requirements/rocm.txt or requirements/dev.txt (dev.txt pulls the CUDA test set).

On 7.2.0, append --torch-backend=rocm7.2. On 7.14.0, leave that flag off.

git clone -b rdna_extras https://github.com/opengfx1030/vllm-rdna.git
cd vllm-rdna
VLLM_USE_PRECOMPILED=0 \
ENABLE_CK=0 \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
ROCM_HOME="${ROCM_HOME:-/opt/rocm}" \
PYTORCH_ROCM_ARCH=gfx1030 \
  uv pip install -e . --no-build-isolation

Smoke test, from Getting started:

python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
python -c "import vllm, vllm._rocm_C; print('vllm', vllm.__version__)"

Once import vllm works, serve from that checkout with tools/rdna/serve_rdna.sh. Pass MODEL and VENV yourself; the recipes do not bake in paths.

Related

• Serve scripts — RECIPE=flashnext-mtp2, 27b-awq, 27b-exl3, …
• Running (Docker) — prefer this unless you need a host venv.
• Building images — the bake graph this page mirrors.
• vLLM forks — kernel behavior after the install.

Fork serve scripts

WIP / fork-source. See Verification status. Knobs below are the in-tree launcher on opengfx1030/vllm-rdna (tools/rdna/serve_rdna.sh, tools/rdna/recipes/*.env, tunableop/README.md). Same path on rdna_extras (v0.31.0 as of 11 Oct 2026). The old 0.28 line is rdna_extra/v0.28.0. Build the venv first: Host venv. First-time V620 path: Onboard. Card-count picks stay on Recipes. Flash-Next graph history stays on Serve lines.

The fork ships one serve entry point under tools/rdna/ (moved from the old scripts/ tree). A recipe file holds only that config’s deltas. The script owns the venv, ROCm library path, TunableOp profile, GPU selection, cache roots, stale-worker teardown, and the vllm serve argv.

Run it from a checkout of the branch you built (rdna_extras is the v0.31.0 line). Hub v0.31.0-extras is the matching Docker bake and already includes this launcher’s PLE / P2P knobs:

MODEL=/path/to/checkpoint VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=<name> [KEY=value …]

MODEL and VENV (or an already-activated $VIRTUAL_ENV) are required. Nothing in the script or the recipes bakes in a home directory, host, or IP.

RECIPE is a script argument (RECIPE=name on the command line). MODEL and VENV are environment variables; a trailing KEY=value can set them too. Precedence, lowest to highest: recipe file → caller environment → trailing KEY=value.

The launcher defaults HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1. Pass a local checkpoint in MODEL. To pull a Hub id on this run, export HF_HUB_OFFLINE=0 and TRANSFORMERS_OFFLINE=0 in the caller environment.

2× Flash-Next expert offload is a sibling wrapper, not a RECIPE= row: tools/rdna/mono/serve_offload.sh after #146 (28 Oct). Same MODEL / PLE_QUANT_DIR hygiene — 2× serve.

Recipes

Files live in tools/rdna/recipes/. Copy one and change only the deltas to add a config.

PLE / n-gram on current recipes

#announcements (11 Oct 2026): three PLE modes are tested on 4× V620 (speed at parity under coding-agent load at 1k / 16k / 32k). The launcher knob is PLE=:

PLE_QUANT_DIR (or VLLM_PLE_QUANT_DIR) must point at a ples_int4 directory that contains META.json, from primitive-ai/Qwen3.8-Flash-Next-PLE-quant. Docker: -v /path/to/ples_int4:/ples_int4 -e VLLM_PLE_QUANT_DIR=/ples_int4 and, for disk mode, -e VLLM_PLE_CPU_OFFLOAD=1. First-time path: Onboard. See Flash-Next serve lines.

Do not paste these onto Flash-Next

#vllm-rdna (Oct 10): agent / Unsloth / old overlay pastes still break serves.

Recipes that omit HIP_VISIBLE_DEVICES use the harness default 0,1,2,3 (override with GPUIDS or HIP_VISIBLE_DEVICES).

Dry-run

PRINT=1 (aliases: DRY_RUN=1, --dry-run, -n) prints the recipe path, the TunableOp profile it would select, the export block, and the exact command, then exits. It does not launch vLLM. Print mode still requires MODEL and VENV, and it skips the check that $VENV/bin/python exists.

MODEL=/path/to/checkpoint VENV=/path/to/venv \
  PRINT=1 bash tools/rdna/serve_rdna.sh RECIPE=27b-exl3

Examples

Flash-Next speculative decode (MTP=2). Long context is where this recipe’s own note says MTP=2 wins:

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

No PCIe P2P on the host (#vllm-rdna Oct 11): add P2P=0 (sets VLLM_RDNA_AR=0 and NCCL_P2P_DISABLE=1; RCCL uses host shared memory). Also try VLLM_RDNA_AR=0 by itself when output is garbage on a no-P2P box.

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4 P2P=0

27B AWQ at MTP=2, with port and KV overrides:

MODEL=/path/to/qwen3.8-27b-awq VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=27b-awq MTP=2 KV=8000000000 PORT=18210

EXL3 stays eager unless you clear that flag. CG_MODE=FULL_AND_PIECEWISE is already the recipe default; it applies once EAGER=0:

MODEL=/path/to/qwen3.8-27b-exl3 VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=27b-exl3 EAGER=0 CG_MODE=FULL_AND_PIECEWISE \
  HIP_VISIBLE_DEVICES=0,1,2,3

Disable TunableOp lookups for an A/B:

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 TUNABLEOP=0

Option map

Legacy names the entry point still accepts: MODEL←MODEL_PATH, KV←KVCAP←KV_CACHE_MEMORY, SEQS←MAX_NUM_SEQS, MAXBAT←MAX_BATCHED_TOKENS, MAXLEN←MAX_MODEL_LEN, GMEM←GPU_MEM.

FEATURES tokens the script knows:

An unknown token aborts. vision-cap is how Flash-Next recipes keep the pixel cap that prevents the vision-encoder startup OOM — see Flash-Next serve lines.

TunableOp

Rows are named profiles, one folder per rocBLAS build, registered in tunableop/profiles.json. Solver IDs are build-specific. The helper (tools/rdna2_028/tunableop_env.sh) selects the folder whose lib_sha256[:12] matches the loaded librocblas.so.5.

• Auto-select is the default. No recipe or launcher change is required when the venv’s librocblas.so.5 hash changes. A ROCm 7.14 start logs TunableOp profile rocm7.14-rocblas5.5 auto-selected for rocBLAS build f30bb442e9b5.
• TUNABLEOP_PROFILE=<name> forces a registered profile. A hash mismatch fails the launch. TUNABLEOP_ALLOW_MISMATCH=1 is the experiment-only override (hard warning, then it uses the rows anyway). Solver IDs are build-specific, so a mismatched profile is never applied silently.
• Lookup is read-only (PYTORCH_TUNABLEOP_TUNING=0). The helper does not write rows to /tmp or the process CWD. If no profile matches, it uses a legacy rocblas-<hash>/ folder when that folder exists, then $HOME/.cache/tunableop/tunableop_results.csv.
• TUNABLEOP=0 turns lookups off (PYTORCH_TUNABLEOP_ENABLED=0) for an A/B against the heuristic.
• PRINT=1 reports which profile it would pick, including “NOT registered” if TUNABLEOP_PROFILE is unknown.

ROCm 10 and rocBLAS 5.6

A venv whose torch loads rocBLAS 5.6 (librocblas.so.5 sha256[:12] c27e2252cc7a) selects rocm10-rocblas5.6 by that hash. Same recipe, same script:

MODEL=/path/to/checkpoint VENV=/path/to/rocm10-venv \
  bash tools/rdna/serve_rdna.sh RECIPE=27b-awq

MODEL=/path/to/checkpoint VENV=/path/to/rocm10-venv \
  PRINT=1 bash tools/rdna/serve_rdna.sh RECIPE=27b-awq

Print mode (library present in that venv) reports:

# tunableop: rocm10-rocblas5.6 (auto-selected for librocblas.so.5 c27e2252cc7a)

A real launch logs TunableOp profile rocm10-rocblas5.6 auto-selected for rocBLAS build c27e2252cc7a.

Thin means the shipped CSV covers 70 shapes per rank. Those GEMMs use the recorded solver. Every other shape falls back to the rocBLAS heuristic: safe, and untuned. The validated serving numbers on this wiki were captured on gfx1030 + ROCm 7.14 (rocm7.14-rocblas5.5, 783 rows). Recipe knobs (TP, KV, MTP, graphs) do not depend on the ROCm version. Treat the first ROCm 10 boot as bring-up: run PRINT=1 and confirm the profile line before leaving it up.

#vllm-rdna (Oct 1): hosts not on rocBLAS 5.5 + ROCm 7.14 do not get the canonical 783-row benefit from the serve script — they need a matching profile (the thin ROCm 10 set, a local $HOME/.cache/tunableop/ table, or a contributed CSV). The shipped 7.14 rows are the 4-card serving set; 8× shapes were not in that table yet — generate local rows on that host rather than expecting a silent speedup.

To fill the gap on that machine, tune there and register the result. The in-tree pipeline keeps scratch under <tree>/cache/tunableop-rows (never /tmp) and does not overwrite the repo profile:

VENV=/path/to/rocm10-venv \
  bash tools/rdna2_028/tunableop_rows_pipeline.sh tune

Curate with tools/rdna2_028/curate_tunableop_rows.py, freeze the CSVs into tunableop/<profile-name>/, and add that name to profiles.json with this library’s sha256. Then prove the lookup actually hits:

python tools/rdna2_028/verify_tunableop_lookup.py \
  --rows tunableop/<profile-name>/tunableop_results0.csv \
  --out hits.json

A hit means the solver TunableOp recorded equals the shipped row. Contributing that profile back is how the 70-row set grows. The older qualify flow is still documented under Regenerate for another rocBLAS build.

Guardrails for a ROCm 10 (or any other) library:

Mismatch symptoms and the historical ~25% prefill gap: TunableOp troubleshooting.

Foreground, aliases, stale workers

serve_rdna.sh execs in the foreground. Background it yourself and poll readiness:

MODEL=/path/to/checkpoint VENV=/path/to/venv \
  nohup bash tools/rdna/serve_rdna.sh RECIPE=27b-awq >serve.log 2>&1 &
# port follows the recipe (27b-awq is 18210)
curl -sf http://127.0.0.1:18210/v1/models

Before exec, the harness signals processes whose environment has the same VLLM_CACHE_ROOT (API server, VLLM::Worker, VLLM::EngineCore, PleOffloadWorker). The default cache root is <tree>/cache/vllm. A second serve that shares that root will tear the first one down.

The old filenames are thin aliases. They forward the environment and any trailing KEY=value:

Related

• Host venv — build the tree this script expects
• Recipes — Hub image vs recipe container vs this checkout
• Flash-Next serve lines — graph modes and PLE
• Environment variables
• TunableOp mismatch

Building the Images

The blivioniag/rocm-rdna and blivioniag/vllm-rdna images are produced by opengfx1030/vllm-rdna-docker (#vllm-rdna, Sep 14 2026 — moved from BlivionIaG/vllm-rdna-docker). The bake repo is deliberately small: two Dockerfiles, one bake graph, one CI workflow — no Python layer, no custom linter.

#announcements (11 Oct 2026): Hub v0.31.0-extras is the current published extras bake (ROCm 7.14 / torch 2.12 / Triton 3.7.1; Flash-Next MTP-2 served). The Sep 14–15 note about a pending 0.28 bake is historical — that line shipped as v0.28.0-extras and is now superseded by 0.31. #vllm-rdna (17 Oct): do not bump the image Triton pin to 3.9+ — those releases drop RDNA1/RDNA2. Stay on 3.7.1 until the bake moves.

Layout

The seven RDNA archs gfx1030;gfx1100;gfx1101;gfx1150;gfx1151;gfx1200;gfx1201 are baked into every image via PYTORCH_ROCM_ARCH.

Build locally

Everything runs through stock docker buildx bake:

# Build everything (bases + vLLM images)
docker buildx bake --file docker-bake.hcl all

# Just the base images, or just the vLLM images
docker buildx bake --file docker-bake.hcl all-bases
docker buildx bake --file docker-bake.hcl all-vllm

# A single target
docker buildx bake --file docker-bake.hcl vllm-0271-rocm720

# Print the plan without building
docker buildx bake --file docker-bake.hcl --print all

Targets are named vllm-<source>-<base>, e.g. vllm-0271-rocm720, vllm-0271-rocm720-extras. Groups: all, all-bases, all-vllm.

Key build arguments (Dockerfile.vllm)

Set per-target in docker-bake.hcl: