Multi-GPU PCIe P2P

For multi-card V620 rigs, enabling GPU↔GPU PCIe Peer-to-Peer (P2P) lets the GPUs DMA directly to each other’s VRAM instead of bouncing through system memory — important for tensor/pipeline parallel serving. This page summarizes the pcie_p2p/ feature of the v620_toolbox repo.

Validated end-to-end on Fedora 43 + AMD CPU (EPYC 7452), 4× Radeon PRO V620. Result: rocminfo enumerates 5 HSA agents and amd-smi topology shows 12/12 GPU↔GPU P2P ENABLED.

Power-cap tuning on Ubuntu 26.04 is supported separately via ubuntu_powertuning/ — P2P recipes here remain Fedora-validated for now. Community Intel notes (Ice Lake) are in When P2P helps — and when it does not.

Two validated kernel paths

Both kernels can be installed on the same host simultaneously; toggle the default with grubby --set-default /boot/vmlinuz-<evr>. Both boot with the 180 W power cap from Power Tuning.

#vllm-rdna / #general (Sep 24): Ubuntu 26 may ship a 7.0-class kernel that sits in the gap between these two validated paths. Do not assume the Fedora Post-7 recipe applies unchanged — rebuild with the toolbox options (or stay on a Pre-7 / ≥7.1 build), then add iommu=pt, confirm ACS off (bit-level caveat), ReBAR / Above 4G on, CSM off. Power-cap helpers for Ubuntu 26 live under ubuntu_powertuning/; P2P itself remains Fedora-validated.

Key prerequisites

• AMD CPU — the KFD P2P path is wiki-validated on AMD (EPYC 7452). Intel Ice Lake can enumerate P2P (community) but often does not speed up inference — see below.
• ≥ 2 Radeon PRO V620 for a meaningful P2P topology (bench validated on 4).
• No pre-gfx1030 AMD GPU installed alongside the V620. ROCm 7.x dropped support for older ASIC families (gfx8xx Polaris, gfx900 Vega, …). If any pre-gfx1030 AMD GPU is present at boot, KFD registration fails and rocminfo bails with Failed to map remapped mmio page on gpu_mem 0. This only matters if your host actually has older AMD cards — remove them first.

Identify your V620 reference boards:

lspci -nn | grep '1002:73a1'    # one line per V620, subsystem 1002:0e34

The full 4-tuple 1002:73a1:1002:0e34 uniquely identifies the V620 reference board and is what the toolbox scripts match on. Other gfx1030 boards (RX 6900 XT / 6800) use device 0x73bf and different subsystem IDs — they’d need the identity match adjusted.

Readiness & verification

The toolbox ships diagnostics that gate on kernel config, hardware, ACS/IOMMU, and runtime state:

# Four-gate readiness check (kernel + hardware + ACS/IOMMU + runtime)
sudo ./powertuning/scripts/v620-p2p-readiness.sh

# End-to-end P2P verification
sudo ./pcie_p2p/scripts/verify-p2p.sh

A healthy system shows:

rocminfo | grep -c '^  Name:.*gfx1030'     # one per V620
amd-smi topology                            # all GPU<->GPU pairs: P2P ENABLED

For the full recipe (kernel config deltas, BIOS/IOMMU settings, what can go wrong), read pcie_p2p/README.md and the knowledge base powertuning/docs/AMD_P2P.md.

ACS is several bits, not one switch

Wiki and toolbox prose still say “ACS off”. #general (25 Oct 2026): that is not a single enable bit. PCIe Access Control Services is a per-port control word (ACSCtl). Clearing every bit is a different recipe from “P2P Request Redirect is off.”

Community host that passed V620↔V620 after the toolbox readiness script still reported ACSCtl = 0x0019 on the relevant switch / root ports (nine ports on that machine):

Same thread, two recipes — do not pick a winner without a second host + verify-p2p.sh:

Inspect without copying a personal BDF. Replace 00:00.0 with your switch or CPU root port (not the GPU):

sudo setpci -s 00:00.0 ECAP_ACS+0x06.w
# 0019 = the community-passing mask above

Then run v620-p2p-readiness.sh and verify-p2p.sh. Treat a readiness “ACS off” line as incomplete until you have amd-smi topology P2P ENABLED and a working TP serve. Needs verify on a second topology.

When P2P helps — and when it does not

amd-smi topology saying P2P ENABLED is not the same as faster tokens. Always A/B with NCCL_P2P_DISABLE=1. Community reports (#general, Ice Lake 4× V620 host):

If bandwidth tests pass but inference regresses:

export NCCL_P2P_DISABLE=1          # llama.cpp / RCCL tensor parallel

On the RDNA2 fork, the README also documents a flag to disable P2P all-reduce fusion (GGML_HIP_GFX1030_P2P_ALLREDUCE=off). Verified ~25 GB/s P2P between V620 pairs does not guarantee faster token generation if links are narrow or the host is a single monolithic PCIe root.

Intel: do not disable IOMMU to “help” P2P

On Ice Lake, disabling VT-d / IOMMU broke P2P even though some generic docs say IOMMU-off is more permissive. Leave IOMMU on, then run the toolbox readiness script and ACS checks. Fedora Server 44 + kernel 6.19 + the powerfix has been reported working on 4× V620 (community; not the Fedora 43 AMD validation).

PLX / PCIe switches

• Intra-switch P2P can stay full-width (e.g. 4× Gen4 x16 behind one PLX 88096). The host↔switch uplink (typically one x16) is the bottleneck. #general (Sep 21): a single gen4 x16 uplink to a PEX88096 is still “full speed to the switch”; cards talk to each other on the switch bus. Consumer X570 boards that expose two CPU x16 (bifurcated x8x8 gen4 when both slots are filled) were called a cheaper host than a workstation board if you only need that uplink.
• #general (Sep 19): one gen4 x16 on the host → community suggestion is a PEX88096 broken out to SlimSAS and 8 risers. Alternative board: dual-SP3 Supermicro H12D-8D, bifurcate all 4 PCIe ports to x8x8 gen4. 6 GPUs is still awkward (not a clean TP size; the open question was TP=8 vs TP=2 × PP=3). One camp says buy 2/4/8 only; another says any even count can work if you plan the split — do not add a third card assuming TP=3.
• 4-slot vs 5-slot switch boards (#general, Sep 2026): a common 5-slot PLX only fits double-width cards; a 4-slot board can take triple-width GPUs (future cooler upgrades). Pick the board for card thickness, not just GPU count.
• 5 GPUs: community prefers TP=4 plus a standalone fifth card (draft / embedder / second model). Pipeline-parallel on 5 can work but is “not optimal.” Odd counts still have TP3 crash notes.
• Without tensor parallel, a PLX box is usually slower than native CPU lanes: higher latency and less aggregate host bandwidth.
• Do not tensor-split across a daisy-chained pair of switches — that single inter-switch link is a TP bottleneck. Prefer TP inside one switch and pipeline-parallel between switches (vLLM can do TP+PP that way; llama.cpp generally cannot).
• Community: ACS often needs extra kernel cmdline fiddling; pcie surprise link down crashes were fixed by putting a small fan on the PLX heatsink (these boards often ship with no airflow notes).
• Workstation hosts that do not POST with V620 + PLX (Dell Precision MMIO High / SR-IOV; Supermicro H12D POST loop): Host firmware.
• External PCIe expansion (#general, Sep 2026 — e.g. Cubix Xpander–class enclosures): each group of cards may get full-width slots behind the switch, while only one x16 (often PCIe 3.0) returns to the host. Community: 8× GPU on that pattern was fine at low context but fell to ~3 t/s around 50–60k context. Prefer 4 GPUs on one switch + P2P / TP inside the switch over stretching TP across a narrow host uplink.

Host topology

Community reports, not wiki-benched:

Layer split vs tensor split (quick)

Community explainer (#benchmarks thread, Sep 2026):

• Layer / pipeline split — VRAM grows with each card, but only one GPU works the model at a time (sequential stages). Less GPU↔GPU traffic.
• Tensor (row) split — all GPUs work together each step; needs much more interconnect bandwidth and often becomes PCIe-limited on narrow links.

On fat CPU roots (gen4 x16 × N), try tensor first. Behind a narrow switch uplink, prefer smaller TP domains or layer/PP splits.

Cabling / risers (community)

Not wiki-benched — common #general notes:

• SlimSAS / SFF-8654 cables: PCIe gen3 is usually fine at ~70 cm; gen4 needs testing per cable/insulation. Prefer known-good gen4 kits over the cheapest Amazon/eBay passive ribbon risers.
• Passive PCIe risers: several community reports of timeouts / inability to hold gen4 x16 with cheap passive risers under multi-GPU load, while the same slots work with cards seated directly. Brand-name gen4 risers (e.g. ADT-class) are often the next step — A/B one card first.
• V620 + blower shroud length: community measure ~37 cm with a common EFH-08E12W-style fan shroud installed — plan chassis / PLX slot spacing accordingly.
• 5-slot PCIe switch → ATX tray: #other (Sep 30) published a Printables adapter (PCIe switch to ATX mounting adapter) for a 5-slot PCIe 4.0 switch on an ATX hole pattern. Community: it matched one common PLX board even when the listing photos differed. Mechanical only — not a P2P or POST fix.