Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux

A hardware enthusiast details the technical process of configuring four 32 GB Resizable BARs on a Linux system using a PCIe switch. The article explains the memory mapping challenges and the custom kernel module developed to manage address space allocation for multiple high-end GPUs.
Why it matters
This demonstrates advanced hardware hacking and kernel-level optimization for high-performance computing setups that exceed standard consumer BIOS limitations.
My PCIe switch is a Broadcom PEX88096 in the board's one x16 slot, with four AMD Radeon AI PRO R9700s and the two RTX 3090s from my PCIe splitter post behind it, and the build post has the whole box.
The switch gives each card eight lanes and lets the cards copy to each other without going through the CPU. AMD's ROCm calls that peer-to-peer (P2P), and it only turns it on when each card's whole 32 GB is mapped. On Linux, that means a 32 GB Resizable BAR on every card. A base address register, or BAR, maps a range of the CPU's address space onto a device's memory. Resizable BAR is the BIOS option that lets a card's BAR be the full 32 GB instead of the default 256 MB.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in