I haven’t come across this problem before, but it could be a problem with advmame 3.5 handling of audio output device.
It might be worth reverting to regular audio temporarily, and seeing if advmame 3.5 works over the Pi’s built-in 3.5mm jack (assuming you’re not using a Pi Zero).
You can do this by commenting out the line: “dtoverlay=hifiberry-dac” in /boot/config.txt
And making sure the line dtparam=audio=on is present and uncommented.
Ensured tparam=audio=on was there and uncommented
Commented out hifiberry
System, booted. Mame1.4 played sound out of the speaker still, not out of headphone
Advmame 3.5 no sound at all.
I can still see sound bouncing in the EQ in the TAB GUI of Sound - Details.
This must be some hack/tweak/hardcoded madness in Advmame that’s missing in 3.5 I had a look through the source/issues on GitHub and can’t find any mention of it, though.
Is there any way you can launch Advmame from the command-line and see what output it gives you? There may be useful messages/errors relating to audio.