Janky Shairplay-Sync

So I built a home speaker running Shairplay sync. It’s powered by a pi-zero with a USB wireless adapter and a Just Boom Amp Phat. Initially the software works just fine but after a while (a minute or so) the track starts to judder with bits getting missed and then eventually the sound will stop for several seconds at a time, then resume.

Does anyone have any idea how I can begin to debug this? Is it the software, the hardware or the wifi? Is there a way to log debug information from Shairplay? Are there any settings I could tweak that would make it more efficient for a Pi Zero? I would appreciate any help anyone can offer!

So far I’ve tried doing a fresh install of Raspbian-lite with only shairplay-sync installed using the Pimoroni one-liner but the problem persists.