That’s an interesting problem? I have one and tinkered with it trying to detect if the ambient light was Red, Green, or Blue. I didn’t do anything dealing with an objects reflected light etc. The code I used is here,
Want to use a BH1745 for Ambient Lighting - Support - Pimoroni Buccaneers
If you don’t mind posting your code I’ll play around with it.