Yep - I’ve done some benchmarking; setting up a flipbook in the Pi’s memory and triggering transfers at 60 fps uses almost no user time (a bit of kernel time, but not much). It would be slick to use the ARM in the hat, but, unless there’s another SPI device needs the bus, the benefit to running on the hat would be minimal.
But it would still be slick!