pHat DAC connecting to an amplifier

I have gotten sound on the pHat DAC working but I need more volume for my 4" speakers. When I try to hook the unit up to a Lepai lP-2020TI amplifier I do not get sound. I am using red and white rca plugs on the amplifier.

Could you give us a bit more information about how you have it set up? Are there any differences between the two setups?