This works as designed (see the schematic of the Badger2040). Without VBUS, the enable-pin keeps the Badger2040 without current until you press a button.
You can find a solution here if you don’t need all buttons: Badger2040 does not wake up after deep-sleep