Yeah, that Virtual Environment stuff is IMHO a PITA. But as of Bookworm, it is something you have to deal with. A workaround, as you are using a Pi Zero, is to use a previous version of Pi OS
Or have a look see here.
Virtual Environments, Libgpiod, gpiodevice and the current mess I’m making of our Python libraries - Support - Pimoroni Buccaneers