Fan shim random speed?

I haven’t seen that. I have one fan that can be noisy, I can’t say for sure its spinning slower or faster when that happens though. I’m not saying its not happening, just that my tired old eyes aren’t seeing it. I put two small rubber pads under that fan, wedged between the fan and my Pibow case and its quiet.

Mine is quiet and no sign of speeding up and down

How are you using your CPU?
My idea is that the load can have a slight impact on the 5V input of the fan.

Currently more often than not I’m watching one of my Motion Eye camera feeds in a browser Window. Sitting idle at the desktop it runs at about 30c with the fan off. Open the browser to that live feed and it creeps up to 45c my trip point. Once the fan comes on it goes back down to around 40c. Close the browser and it will go back down to 30c and the fan stops. It turns off at 35c.
Not heavy use I don’t think, maybe moderate?
EDIT: This is a Pi 3B+, not a 4B. It just happens to be the Pi I use more often, and has a Fan Shim on it. When I get a chance I’ll plug my 4b in and take some notes.

Ok, my Pi 4B runs at about 40c idle at the desktop. Once I open my camera feed in a browser it creeps up over 45c and the fan turns on. It then runs around 37c until I close the browser window. At 35c the fan turns off and it will slowly go back to about 40c.
This Pi is mounted vertically in a SmartiPi case. The stock fan and cover that comes with this case is removed and the Fan Shim attached.
The 3B+ above is laying flat in a Pibow Coupe case.