RDNA2 fork — Serving

Recommended env stack

HSA_OVERRIDE_GFX_VERSION=10.3.0 is required for the tested V620/gfx1030 native profile.

Prefer this short stack (#llamacpp, Aug 2026). Older long lists of GGML_HIP_GFX1030_* knobs are mostly redundant with the override and can clash with the RCCL autotune path:

HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HSA_NO_SCRATCH_RECLAIM=1 \
GGML_HIP_RDNA2_AUTO=1 \
GGML_HIP_SAFE_STATE_IO=1 \
GGML_TP_SHARDED_OUTPUT=1   # TP2+ only

• GGML_HIP_SAFE_STATE_IO=1 — recommended default; mitigates a known ROCm FA crash class.
• GGML_CUDA_ALLREDUCE=nccl — community reports +10% tgen / +20% prefill on Qwen 122B when RCCL is healthy.
• Optional TP4 mode: GGML_HIP_GFX1030_P2P_ALLREDUCE=auto-expanded (topology-gated).
• If you see internal AllReduce init failed (n_devices != 2) or wild PP variance, strip custom GGML_HIP_GFX1030_* / P2P knobs back to the short stack and re-test on ROCm 7.2.0 or 7.14.0 (not mid-7.2.x such as 7.2.4). See Installing ROCm.

Launch

TP2+ (four-GPU example):

HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HSA_NO_SCRATCH_RECLAIM=1 \
GGML_HIP_RDNA2_AUTO=1 \
GGML_HIP_SAFE_STATE_IO=1 \
GGML_TP_SHARDED_OUTPUT=1 \
./build/bin/llama-server \
  -m /path/to/main.gguf \
  -ngl all \
  --split-mode tensor \
  --tensor-split 1,1,1,1 \
  --flash-attn on \
  --host 0.0.0.0 --port 8080

TP1 — omit tensor split and GGML_TP_SHARDED_OUTPUT:

HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HSA_NO_SCRATCH_RECLAIM=1 \
GGML_HIP_SAFE_STATE_IO=1 \
./build/bin/llama-server -m /path/to/main.gguf -ngl all --flash-attn on --host 0.0.0.0 --port 8080

Batch / ubatch tips

For tensor-split prefill, #llamacpp often does better with larger ubatch than the old 2048/256 bench defaults — roughly ~1024 ubatch per GPU (e.g. TP4 → --ubatch-size 4096) while keeping --batch-size ≥ ubatch. Small prompts may regress slightly; long prompts usually win.

Recent long-context community recipes commonly use --batch-size 16384 --ubatch-size 1024 on TP4 (see Benchmarks).

Host tips that affect llama-server

• CPU governor: if any hot path stays on the host (Flash-Next n-gram tables, --override-tensor …=CPU, MoE/KV offload), powersave can lag bursty PP. Community: switching Intel intel_pstate to performance improved Flash-Next PP ~33% while VRAM-resident 27B was unchanged. Check /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor. See General troubleshooting.
• DAX / Optane model store: fast loads are nice for swap-testing; never mmap GGUFs from dax=always mounts into ROCm — use --no-mmap / --load-mode none. See troubleshooting.
• Qwen chat templates: #vllm-rdna (Oct 8) pointed at froggeric/Qwen-Fixed-Chat-Templates as a drop-in --chat-template-file when official Qwen Jinja wastes tokens or stalls agents. #general (Oct 9): community reports tool calls holding up on that Jinja (OG Qwen3.8 + Open WebUI). Needs verify — resources.

Full validated example (4× V620, Qwen3.5-122B-A10B-MTP)

GGML_CUDA_DISABLE_GRAPHS=1 GGML_CUDA_ALLREDUCE=nccl HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HSA_NO_SCRATCH_RECLAIM=1 LD_LIBRARY_PATH=/opt/rocm/core-7.14/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} \
./build/bin/llama-server \
  -m /path/to/Qwen3.5-122B-A10B-MTP-UD-Q4_K_M.gguf \
  -ngl all --split-mode tensor --tensor-split 1,1,1,1 --main-gpu 0 \
  --ctx-size 262144 --cache-type-k f16 --cache-type-v f16 --kv-unified \
  --batch-size 2048 --ubatch-size 256 --parallel 4 \
  --host 0.0.0.0 --port 8080 --temp 0.6 --metrics --jinja --numa distribute \
  --reasoning on --reasoning-format deepseek --chat-template-kwargs '{"preserve_thinking": true}' \
  --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-type-k f16 --spec-draft-type-v f16 \
  --spec-draft-p-min 0.0 --spec-draft-p-split 0.10 --no-spec-draft-backend-sampling --cache-ram 0

Docker example

TP2 + MTP4 from #llamacpp (ROCm 7.14, MXFP4 Qwen3.8-27B):

environment:
  HSA_NO_SCRATCH_RECLAIM: "1"
  GGML_HIP_RDNA2_AUTO: "1"
  GGML_HIP_SAFE_STATE_IO: "1"
  GGML_TP_SHARDED_OUTPUT: "1"
  HSA_OVERRIDE_GFX_VERSION: "10.3.0"
command: >
  llama-server
  -hf quark75/Qwen3.8-27B-MXFP4-GGUF:MXFP4
  -ngl 999 -fa on -sm tensor -ts 1,1 -fit off
  --flash-attn on --ctx-size 32768
  --cache-type-k f16 --cache-type-v f16
  --batch-size 4096 --ubatch-size 4096 --parallel 1 --cont-batching
  --host 0.0.0.0 --port 8091 --temp 0.7 --jinja --metrics -kvu
  --reasoning-preserve
  --chat-template-kwargs '{"preserve_thinking": true}'
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-ngl 999
  --ctx-checkpoints 0

Build with ./scripts/build-rdna2-portable.sh. RCCL needs working PCIe P2P. If P2P is enabled but slower, set NCCL_P2P_DISABLE=1. If all-reduce fails, try GGML_HIP_GFX1030_P2P_ALLREDUCE=off or GGML_CUDA_ALLREDUCE=none.

LocalAI YAML (2× Flash-Next, #llamacpp Sep 30)

Community LocalAI backend rocm-llama-cpp-development for Navin-Models/Qwen3.8-Flash-Next-Uncensored-AD-4.27-GGUF on 2× V620. Paths generalized. Needs verify — not a fork profile.

backend: rocm-llama-cpp-development
batch: 4096
ubatch: 1024
cache_type_k: q8_0
cache_type_v: q8_0
context_size: 229376   # 32k under 262144 so a second user can grow KV
fit_params: false
flash_attention: true
main_gpu: 0
mmap: true
mmproj: /path/to/mmproj-Qwen3.8-Flash-Next-Uncensored-F16.gguf
split_mode: tensor
tensor_split: 1,1.1
options:
  - use_jinja:true
  - parallel:1
  - cache_reuse:256
  - context_shift:true
  - cache_ram:8192
  - kv_unified:true
parameters:
  min_p: 0.05
  model: /path/to/Qwen3.8-Flash-Next-Uncensored-AD-4.27-00001-of-00033.gguf
  temperature: 1
  top_k: 40
  top_p: 0.8
template:
  use_tokenizer_template: true

Same thread: --batch-size 4096 --ubatch-size 1024 also helped Swift GSQ-RCO PP a little (speculative table).

Swift IQ3_XXS + MTP (2× V620, #llamacpp Oct 2)

Community llama-server line for ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_XXS on 2× V620. Paths generalized. Needs verify — VRAM was called the bottleneck.

HIP_VISIBLE_DEVICES=0,1 ./build/bin/llama-server \
  -m /path/to/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf \
  --mmproj /path/to/mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf \
  -ngl all -sm tensor -ts 1,1.1 \
  -ot 'per_layer_token_embd\\.weight=CPU' \
  --lazy-mode off --load-mode none \
  -c 71680 -fa on -ctk f16 -ctv f16 \
  -b 2048 -ub 1024 \
  -np 1 --cache-ram 8192 \
  --jinja --reasoning-preserve \
  -md /path/to/mtp-Qwen3.8-Flash-Next-Q4_0.gguf \
  --spec-type draft-mtp --spec-draft-ngl all --spec-draft-n-max 3 \
  --temp 1.0 --top-k 40 --top-p 0.8 --min-p 0.05 \
  --host 127.0.0.1 --port 8085

Community snapshot: PP starts around ~800 t/s and falls to ~650 by ~30k context; decode ~35 → ~20 t/s. The Sep 30 Swift row used 4k/1k batch and reported ~400 / ~40 — treat both as single-host numbers, not a recipe bake-off.

The RDNA2 fork’s recent HIP/RCCL work is validated primarily against ROCm 7.14 (#llamacpp, Sep 2026). Mid-7.2.x (e.g. 7.2.4) is not a confident target — upgrade or pin per Installing ROCm.

Single-GPU community ballpark (Qwen3.8-27B)

#llamacpp (Sep 2026), 1× V620, RDNA2 fork, short env stack — treat as single-host snapshots:

On 4× V620, long-context MTP benches (community suite) saw decode hold ~60–65 t/s at 32k on coding prompts but drop harder by 64k — ngram / sidecar MTP work aims to lift acceptance; A/B with llm-context-bench.

Idle power with embedded servers

Running an embedding server (nomic, etc.) alongside chat on the same cards can add ~40 W per GPU even when the embedder looks idle (#llamacpp). Park embedders on a spare card or stop them when not needed if you care about power.

Concurrent slots / multi-agent (V620)

#llamacpp (Sep 2026): on V620 (no matrix cores), --parallel > 1 can crush per-stream decode — community report of ~35–50 t/s → ~5–9 t/s when two generations overlap, and prefill of a second request can stall the first. Prefer one stream per process and scale out with separate instances (different GPUs / aliases) if you need multi-agent throughput. KV-aware load balancers (e.g. paddler) and shared KV layers (LMCache on the vLLM side) are discussed in #llamacpp / #lmcache but are not wiki-validated recipes yet.

Faster GGUF loads (upstream WIP)

Community (#vllm-rdna / #llamacpp, Sep 2026): upstream llama.cpp PR #22466 (async pinned upload for -sm tensor loads) dropped a 122B-Q4 load from >2 min to <35 s on one host. Watch that PR / fork merges if cold-start time matters.

Notable limits

• Validated primarily on 4× V620 gfx1030, ROCm 7.14; other systems use conservative fallbacks. TP4 does work on known-good server boards (e.g. Gigabyte MC62-G40) and on some dual-socket Broadwell-EP hosts. Most successful TP4 + P2P reports are on AMD CPUs.
• Multi-socket hosts can hurt TP — single-socket EPYC preferred. Dual-Xeon UPI traffic has halved prefill in #general even when theoretical cross-socket bandwidth looked fine.
• KV checkpoints crash on tensor split — use --ctx-checkpoints 0.
• GGML_TP_SHARDED_OUTPUT and GGML_TP_VOCAB_SHARDED_OUTPUT are incompatible modes.
• Tensor-split prefill spikes current — layer split can look fine while TP shuts the PSU off at prefill start (transient on the +12 V rail, classic with old miner PSUs). Try 160 W (~2–4% slower) or 140 W (~8–10% slower vs unlocked) before blaming the fork. See Power Tuning.
• TP3 (three cards) has caused driver crashes; stick to 2 or 4 when possible. On boards with three full CPU x16 slots, community still prefers trying tensor split first if the links are PCIe 4.0 x16 — see Host topology.
• Most GGML_HIP_GFX1030_* flags are redundant with HSA_OVERRIDE_GFX_VERSION=10.3.0 unless A/B testing — prefer the short env stack.
• FA / q8 KV: quantized V cache needs FA on; FA occupancy asserts on some head-256 models — see troubleshooting.
• GPU sampling (--spec-draft-backend-sampling and related): install hipcub-devel (or distro equivalent) at build time. Without it, expect device 'Meta()' does not have support for op TOP_K and fall back to slower CPU sampling.
• Long-context quant quality: community (#llamacpp): below Q6 often looks fine on short benches then loops / stalls as context grows; Q8 is noticeably more stable. Pair with sidecar gotchas.

For stock builds see Building & running. Speculative decoding configs: RDNA2 speculative decoding.