It sounds like you have a problem with balancing/unbalancing. If you connect an unbalanced line to either the input or output XLR, all three pins of the XLR must be connected or you will get a lot of hum and buzz. Typically, you connect pins 1,3 to ground and pin 2 to signal.
Beyond this, be aware that the XLR input on the unit is a line input rather than a microphone input, so you will not get best results unless you use a mike preamp.
Most people use the effects loop….