It shouldn’t matter which side you solder the header on. They are through hole plated.
Is the display soldered to the perf board? And is there a blue jumper under it linking the ground over to the other row of 5?
Your soldering may be an issue, you might want to reflow some of those connctions, especially the ones on the top of the board.
A solderless breadboard is what you want for that kind of setup.
Ok, was just double checking to make sure you didn’t accidentally miss a jumper. It happens to the best of us.
I use Perma Proto Hats and Proto Zero’s quit a lot in projects.
With a stacking header I can plug another Hat or pHat in on top of them,
I have one Pi with a huge solderless breadboard I use for proto typing.
I’m a bit spoiled I guess, but its lots of fun tinkering with this stuff.