I just stumbled across this forum post:
Unicorn pHAT - incompatible with Retropie?
My raspbian install that doesn’t work is a RetroPie image. I double-checked the analog jack and there is no sound coming out, it’s only going over HDMI… Let’s see if I can close in on a solution.