Working Ok here for me, Pi 4B 4 gig running Buster. My Fan Shim is soldered to a booster header. I don’t know if it matters but I didn’t install the fan shim software.
EDIT: I guess I should also mention that I currently don’t have any i2c devices connected.
The BCM 3, pin 5 wake connection is to the button. My understanding is a ground is placed on pin 5 when, and only while, the button is pressed. If you don’t touch the button on the fan shim nothing should be connected to pin 5 other than a trace on the circuit board going to an open circuit.
If you power down, and leave the Pi powered up. Momentarily grounding GPIO 3 / pin 5 will boot it up. It can also be used to do a proper shut down when running with a config.txt edit, which I have done. dtoverlay=gpio-shutdown