I went side by side for my setup.
My Pi 5 NVMe build - Discussion - Pimoroni Buccaneers
The NVMe base ends up backside up though. Mine is stock, no cooler, so no big deal for me.
If your no making use of the Pi’s GPIO Header you could mount the NVME Base on top of the Pi.