Sorry, it is not quite clear what is happening with your setup.
If Kill Dry is turned on, when you turn the mix of the Program or Preset down, you should get no signal from the H90 output. Can you confirm that is happening?
The Program Out Gain is at the end of the signal path and should lower the overall volume. Can you try a simple setup where you remove the preamp / insert configuration and just go guitar > H90 > amp input and try adjust the program output level?
