It wont do them, or you, any harm, but it is, on paper, bad for the Pi. That said, I’ve sat here and mashed the buttons constantly with no ill effect. It’s technically driving two pins out of their tolerances, but only if you press the button that corresponds to an LED as it lights up.
I measured 34mA going across the pins, which is somewhat higher than the 16mA designed and recommended maximum.
Moving the Blue LED to the far right seems to work, but my crazy alternate layout seems slightly more reliable. I could easily be imagining that though!