I think I was able to solve it for now, based on some linked articles about other DACs, like Issue: PhatDac on Pi Zero with pigpio and moode 6.3 - #14 by alphanumeric (I can’t add more than two links, so…)
For anyone running into this, it was pigpiod interfering, I added -t 0 as described in pigpio library , not yet sure if this is really the best solution and if this leads to other challenges with GPIOs, but at least: audio plays perfect that way.