Multi-GPU PCIe P2P

For multi-card V620 rigs, enabling GPU↔GPU PCIe Peer-to-Peer (P2P) lets the GPUs DMA directly to each other’s VRAM instead of bouncing through system memory — important for tensor/pipeline parallel serving. This page summarizes the pcie_p2p/ feature of the v620_toolbox repo.

Validated end-to-end on Fedora 43 + AMD CPU (EPYC 7452), 4× Radeon PRO V620. Result: rocminfo enumerates 5 HSA agents and amd-smi topology shows 12/12 GPU↔GPU P2P ENABLED.

Power-cap tuning on Ubuntu 26.04 is supported separately via ubuntu_powertuning/ — P2P recipes here remain Fedora-validated for now. Community Intel notes (Ice Lake) are in When P2P helps — and when it does not.

Two validated kernel paths

Both kernels can be installed on the same host simultaneously; toggle the default with grubby --set-default /boot/vmlinuz-<evr>. Both boot with the 180 W power cap from Power Tuning.

Key prerequisites

• AMD CPU — the KFD P2P path is wiki-validated on AMD (EPYC 7452). Intel Ice Lake can enumerate P2P (community) but often does not speed up inference — see below.
• ≥ 2 Radeon PRO V620 for a meaningful P2P topology (bench validated on 4).
• No pre-gfx1030 AMD GPU installed alongside the V620. ROCm 7.x dropped support for older ASIC families (gfx8xx Polaris, gfx900 Vega, …). If any pre-gfx1030 AMD GPU is present at boot, KFD registration fails and rocminfo bails with Failed to map remapped mmio page on gpu_mem 0. This only matters if your host actually has older AMD cards — remove them first.

Identify your V620 reference boards:

lspci -nn | grep '1002:73a1'    # one line per V620, subsystem 1002:0e34

The full 4-tuple 1002:73a1:1002:0e34 uniquely identifies the V620 reference board and is what the toolbox scripts match on. Other gfx1030 boards (RX 6900 XT / 6800) use device 0x73bf and different subsystem IDs — they’d need the identity match adjusted.

Readiness & verification

The toolbox ships diagnostics that gate on kernel config, hardware, ACS/IOMMU, and runtime state:

# Four-gate readiness check (kernel + hardware + ACS/IOMMU + runtime)
sudo ./powertuning/scripts/v620-p2p-readiness.sh

# End-to-end P2P verification
sudo ./pcie_p2p/scripts/verify-p2p.sh

A healthy system shows:

rocminfo | grep -c '^  Name:.*gfx1030'     # one per V620
amd-smi topology                            # all GPU<->GPU pairs: P2P ENABLED

For the full recipe (kernel config deltas, BIOS/IOMMU settings, what can go wrong), read pcie_p2p/README.md and the knowledge base powertuning/docs/AMD_P2P.md.

When P2P helps — and when it does not

amd-smi topology saying P2P ENABLED is not the same as faster tokens. Always A/B with NCCL_P2P_DISABLE=1. Community reports (#general, Ice Lake 4× V620 host):

If bandwidth tests pass but inference regresses:

export NCCL_P2P_DISABLE=1          # llama.cpp / RCCL tensor parallel

On the RDNA2 fork, the README also documents a flag to disable P2P all-reduce fusion (GGML_HIP_GFX1030_P2P_ALLREDUCE=off). Verified ~25 GB/s P2P between V620 pairs does not guarantee faster token generation if links are narrow or the host is a single monolithic PCIe root.

Intel: do not disable IOMMU to “help” P2P

On Ice Lake, disabling VT-d / IOMMU broke P2P even though some generic docs say IOMMU-off is more permissive. Leave IOMMU on, then run the toolbox readiness script and ACS checks. Fedora Server 44 + kernel 6.19 + the powerfix has been reported working on 4× V620 (community; not the Fedora 43 AMD validation).

PLX / PCIe switches

• Intra-switch P2P can stay full-width (e.g. 4× Gen4 x16 behind one PLX 88096). The host↔switch uplink (typically one x16) is the bottleneck.
• Without tensor parallel, a PLX box is usually slower than native CPU lanes: higher latency and less aggregate host bandwidth.
• Do not tensor-split across a daisy-chained pair of switches — that single inter-switch link is a TP bottleneck. Prefer TP inside one switch and pipeline-parallel between switches (vLLM can do TP+PP that way; llama.cpp generally cannot).
• Community: ACS often needs extra kernel cmdline fiddling; pcie surprise link down crashes were fixed by putting a small fan on the PLX heatsink (these boards often ship with no airflow notes).
• External PCIe expansion (#general, Sep 2026 — e.g. Cubix Xpander–class enclosures): each group of cards may get full-width slots behind the switch, while only one x16 (often PCIe 3.0) returns to the host. Community: 8× GPU on that pattern was fine at low context but fell to ~3 t/s around 50–60k context. Prefer 4 GPUs on one switch + P2P / TP inside the switch over stretching TP across a narrow host uplink.

Host topology

Community reports, not wiki-benched:

Layer split vs tensor split (quick)

Community explainer (#benchmarks thread, Sep 2026):

• Layer / pipeline split — VRAM grows with each card, but only one GPU works the model at a time (sequential stages). Less GPU↔GPU traffic.
• Tensor (row) split — all GPUs work together each step; needs much more interconnect bandwidth and often becomes PCIe-limited on narrow links.

On fat CPU roots (gen4 x16 × N), try tensor first. Behind a narrow switch uplink, prefer smaller TP domains or layer/PP splits.

Cabling / risers (community)

Not wiki-benched — common #general notes:

• SlimSAS / SFF-8654 cables: PCIe gen3 is usually fine at ~70 cm; gen4 needs testing per cable/insulation. Prefer known-good gen4 kits over the cheapest Amazon/eBay passive ribbon risers.
• Passive PCIe risers: several community reports of timeouts / inability to hold gen4 x16 with cheap passive risers under multi-GPU load, while the same slots work with cards seated directly. Brand-name gen4 risers (e.g. ADT-class) are often the next step — A/B one card first.
• V620 + blower shroud length: community measure ~37 cm with a common EFH-08E12W-style fan shroud installed — plan chassis / PLX slot spacing accordingly.