Sorry for the issue you are having. I am not familiar with TotalMix so I can’t suggest anything there, but you should be able to route your audio directly from Ableton to the H9000 without the TotalMix software, or at least using the software at some sort of default setting.
Did you view this video that was shared before? https://www.youtube.com/watch?v=j6gyzOjjA88&t=72s
The “External Audio Effect” plug-in in Ableton makes this very simple. Leave your Ableton master output at channels 1-2 (or whatever you normally use for the master output channels). Use channels 3-4 for the inputs/outputs of the Ableton external audio effect. Connect those to the H9000. Now you have an external hardware insert within Ableton.
It sounds like you are creating a feedback loop with the setup you are trying because you are sending the master bus of Ableton to the H9000 and then trying to record that back.
