I’m using Pan Tilt Hat from Pimoroni to control to servo motors. I found here (https://pinout.xyz/pinout/pan_tilt_hat) that Pan Tilt Hat uses 3 GPIO pins and 5V power to work. To my understanding it uses pins no. 2 (5V), 3,5 (I2C) and 36 (GPIO). For my pan tilt hat connection I use wires, as I need some stand-offs to fit my other components. Also I cannot afford to use all 40 pins by pan tilt hat. So I manually connect pins 2,3,5 and 36 to the hat. However, in such case, I get an error: IOError: Failed to write byte. It seems that it fails to communicate over I2C. When I connect my pan tilt hat directly to 40 pin header it works fine.
Yes, you need to wire all the grounds in the pinout with the black horseshoe boarders around them.
On the Pi all the grounds are one common connection. On a Hat or pHat its not easy or may not even be possible to link them all together. Not with out adding more layers (and cost) to the circuit board. Plugging the pHat or Hat into the Pi will link them all together for you. Assuming the user plugs it directly into the Pi. ;)
I had a similar issue on my Fan Shim because I missed the one extra ground connection it used.
It will make life easier and let you screw one stand off on top of the other. Proto hat in between the two.
You stack it between your Pi and Hat and then get access to the GPIO pins.