Fork serve scripts

WIP / fork-source. See Verification status. Knobs below are the in-tree launcher on opengfx1030/vllm-rdna (tools/rdna/serve_rdna.sh, tools/rdna/recipes/*.env, tunableop/README.md). Same path on rdna_extras (v0.31.0 as of 11 Oct 2026). The old 0.28 line is rdna_extra/v0.28.0. Build the venv first: Host venv. First-time V620 path: Onboard. Card-count picks stay on Recipes. Flash-Next graph history stays on Serve lines.

The fork ships one serve entry point under tools/rdna/ (moved from the old scripts/ tree). A recipe file holds only that config’s deltas. The script owns the venv, ROCm library path, TunableOp profile, GPU selection, cache roots, stale-worker teardown, and the vllm serve argv.

Run it from a checkout of the branch you built (rdna_extras is the v0.31.0 line). Hub v0.31.0-extras is the matching Docker bake and already includes this launcher’s PLE / P2P knobs:

MODEL=/path/to/checkpoint VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=<name> [KEY=value …]

MODEL and VENV (or an already-activated $VIRTUAL_ENV) are required. Nothing in the script or the recipes bakes in a home directory, host, or IP.

RECIPE is a script argument (RECIPE=name on the command line). MODEL and VENV are environment variables; a trailing KEY=value can set them too. Precedence, lowest to highest: recipe file → caller environment → trailing KEY=value.

The launcher defaults HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1. Pass a local checkpoint in MODEL. To pull a Hub id on this run, export HF_HUB_OFFLINE=0 and TRANSFORMERS_OFFLINE=0 in the caller environment.

2× Flash-Next expert offload is a sibling wrapper, not a RECIPE= row: tools/rdna/mono/serve_offload.sh after #146 (28 Oct). Same MODEL / PLE_QUANT_DIR hygiene — 2× serve.

Recipes

Files live in tools/rdna/recipes/. Copy one and change only the deltas to add a config.

PLE / n-gram on current recipes

#announcements (11 Oct 2026): three PLE modes are tested on 4× V620 (speed at parity under coding-agent load at 1k / 16k / 32k). The launcher knob is PLE=:

PLE_QUANT_DIR (or VLLM_PLE_QUANT_DIR) must point at a ples_int4 directory that contains META.json, from primitive-ai/Qwen3.8-Flash-Next-PLE-quant. Docker: -v /path/to/ples_int4:/ples_int4 -e VLLM_PLE_QUANT_DIR=/ples_int4 and, for disk mode, -e VLLM_PLE_CPU_OFFLOAD=1. First-time path: Onboard. See Flash-Next serve lines.

Do not paste these onto Flash-Next

#vllm-rdna (Oct 10): agent / Unsloth / old overlay pastes still break serves.

Recipes that omit HIP_VISIBLE_DEVICES use the harness default 0,1,2,3 (override with GPUIDS or HIP_VISIBLE_DEVICES).

Dry-run

PRINT=1 (aliases: DRY_RUN=1, --dry-run, -n) prints the recipe path, the TunableOp profile it would select, the export block, and the exact command, then exits. It does not launch vLLM. Print mode still requires MODEL and VENV, and it skips the check that $VENV/bin/python exists.

MODEL=/path/to/checkpoint VENV=/path/to/venv \
  PRINT=1 bash tools/rdna/serve_rdna.sh RECIPE=27b-exl3

Examples

Flash-Next speculative decode (MTP=2). Long context is where this recipe’s own note says MTP=2 wins:

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4

No PCIe P2P on the host (#vllm-rdna Oct 11): add P2P=0 (sets VLLM_RDNA_AR=0 and NCCL_P2P_DISABLE=1; RCCL uses host shared memory). Also try VLLM_RDNA_AR=0 by itself when output is garbage on a no-P2P box.

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 \
  PLE=int4 PLE_QUANT_DIR=/path/to/ples_int4 P2P=0

27B AWQ at MTP=2, with port and KV overrides:

MODEL=/path/to/qwen3.8-27b-awq VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=27b-awq MTP=2 KV=8000000000 PORT=18210

EXL3 stays eager unless you clear that flag. CG_MODE=FULL_AND_PIECEWISE is already the recipe default; it applies once EAGER=0:

MODEL=/path/to/qwen3.8-27b-exl3 VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=27b-exl3 EAGER=0 CG_MODE=FULL_AND_PIECEWISE \
  HIP_VISIBLE_DEVICES=0,1,2,3

Disable TunableOp lookups for an A/B:

MODEL=/path/to/flash-next VENV=/path/to/venv \
  bash tools/rdna/serve_rdna.sh RECIPE=flashnext-mtp2 TUNABLEOP=0

Option map

Legacy names the entry point still accepts: MODEL←MODEL_PATH, KV←KVCAP←KV_CACHE_MEMORY, SEQS←MAX_NUM_SEQS, MAXBAT←MAX_BATCHED_TOKENS, MAXLEN←MAX_MODEL_LEN, GMEM←GPU_MEM.

FEATURES tokens the script knows:

An unknown token aborts. vision-cap is how Flash-Next recipes keep the pixel cap that prevents the vision-encoder startup OOM — see Flash-Next serve lines.

TunableOp

Rows are named profiles, one folder per rocBLAS build, registered in tunableop/profiles.json. Solver IDs are build-specific. The helper (tools/rdna2_028/tunableop_env.sh) selects the folder whose lib_sha256[:12] matches the loaded librocblas.so.5.

• Auto-select is the default. No recipe or launcher change is required when the venv’s librocblas.so.5 hash changes. A ROCm 7.14 start logs TunableOp profile rocm7.14-rocblas5.5 auto-selected for rocBLAS build f30bb442e9b5.
• TUNABLEOP_PROFILE=<name> forces a registered profile. A hash mismatch fails the launch. TUNABLEOP_ALLOW_MISMATCH=1 is the experiment-only override (hard warning, then it uses the rows anyway). Solver IDs are build-specific, so a mismatched profile is never applied silently.
• Lookup is read-only (PYTORCH_TUNABLEOP_TUNING=0). The helper does not write rows to /tmp or the process CWD. If no profile matches, it uses a legacy rocblas-<hash>/ folder when that folder exists, then $HOME/.cache/tunableop/tunableop_results.csv.
• TUNABLEOP=0 turns lookups off (PYTORCH_TUNABLEOP_ENABLED=0) for an A/B against the heuristic.
• PRINT=1 reports which profile it would pick, including “NOT registered” if TUNABLEOP_PROFILE is unknown.

ROCm 10 and rocBLAS 5.6

A venv whose torch loads rocBLAS 5.6 (librocblas.so.5 sha256[:12] c27e2252cc7a) selects rocm10-rocblas5.6 by that hash. Same recipe, same script:

MODEL=/path/to/checkpoint VENV=/path/to/rocm10-venv \
  bash tools/rdna/serve_rdna.sh RECIPE=27b-awq

MODEL=/path/to/checkpoint VENV=/path/to/rocm10-venv \
  PRINT=1 bash tools/rdna/serve_rdna.sh RECIPE=27b-awq

Print mode (library present in that venv) reports:

# tunableop: rocm10-rocblas5.6 (auto-selected for librocblas.so.5 c27e2252cc7a)

A real launch logs TunableOp profile rocm10-rocblas5.6 auto-selected for rocBLAS build c27e2252cc7a.

Thin means the shipped CSV covers 70 shapes per rank. Those GEMMs use the recorded solver. Every other shape falls back to the rocBLAS heuristic: safe, and untuned. The validated serving numbers on this wiki were captured on gfx1030 + ROCm 7.14 (rocm7.14-rocblas5.5, 783 rows). Recipe knobs (TP, KV, MTP, graphs) do not depend on the ROCm version. Treat the first ROCm 10 boot as bring-up: run PRINT=1 and confirm the profile line before leaving it up.

#vllm-rdna (Oct 1): hosts not on rocBLAS 5.5 + ROCm 7.14 do not get the canonical 783-row benefit from the serve script — they need a matching profile (the thin ROCm 10 set, a local $HOME/.cache/tunableop/ table, or a contributed CSV). The shipped 7.14 rows are the 4-card serving set; 8× shapes were not in that table yet — generate local rows on that host rather than expecting a silent speedup.

To fill the gap on that machine, tune there and register the result. The in-tree pipeline keeps scratch under <tree>/cache/tunableop-rows (never /tmp) and does not overwrite the repo profile:

VENV=/path/to/rocm10-venv \
  bash tools/rdna2_028/tunableop_rows_pipeline.sh tune

Curate with tools/rdna2_028/curate_tunableop_rows.py, freeze the CSVs into tunableop/<profile-name>/, and add that name to profiles.json with this library’s sha256. Then prove the lookup actually hits:

python tools/rdna2_028/verify_tunableop_lookup.py \
  --rows tunableop/<profile-name>/tunableop_results0.csv \
  --out hits.json

A hit means the solver TunableOp recorded equals the shipped row. Contributing that profile back is how the 70-row set grows. The older qualify flow is still documented under Regenerate for another rocBLAS build.

Guardrails for a ROCm 10 (or any other) library:

Mismatch symptoms and the historical ~25% prefill gap: TunableOp troubleshooting.

Foreground, aliases, stale workers

serve_rdna.sh execs in the foreground. Background it yourself and poll readiness:

MODEL=/path/to/checkpoint VENV=/path/to/venv \
  nohup bash tools/rdna/serve_rdna.sh RECIPE=27b-awq >serve.log 2>&1 &
# port follows the recipe (27b-awq is 18210)
curl -sf http://127.0.0.1:18210/v1/models

Before exec, the harness signals processes whose environment has the same VLLM_CACHE_ROOT (API server, VLLM::Worker, VLLM::EngineCore, PleOffloadWorker). The default cache root is <tree>/cache/vllm. A second serve that shares that root will tear the first one down.

The old filenames are thin aliases. They forward the environment and any trailing KEY=value:

Related

• Host venv — build the tree this script expects
• Recipes — Hub image vs recipe container vs this checkout
• Flash-Next serve lines — graph modes and PLE
• Environment variables
• TunableOp mismatch