the SPI and UART pins are broken out at the bottom of the Pan-Tilt HAT, those would be perfectly fine as GPIO.
... in an ideal world all unused pins would remain accessible - that's what pHATs aim to provide, without complication resulting from adding traces that would otherwise somehow have to be routed without compromising the board primary functionality and/or affordability.
HATs do have another goal which is to lower the number of potential hurdles for beginners, which is pretty much at the opposite of the spectrum in that respect, and yet keep cost as friendly as possible.
regardless, I would wholeheartedly recommend the mini-black HAT for a 'best of both worlds' solution. It won't fit a case, but then again there is no real solution to (practical) access and enclosure.