Sorry, I’m not sure why Cubase is not measuring the latency correctly for you. I don’t have a copy of Cubase to try this with at the moment, but I just tested a similar feature in Logic with a Dante card and it worked correctly. I bypassed the algorithms in the FX Chain, pinged the external FX plug-in, and got a latency measurement. I then removed an algorithm and pinged again, and I got a smaller latency measurement.
Do you have another piece of hardware you can use as an insert in Cubase to try and measure the latency to see if it works there? If you can get audio to pass through the MADI card, then it should work when you ping it for latency.
Is the analog output on the PCIe card a TRS 1/4 stereo jack? If that is all you are missing to connect the card directly to your monitors, I would suggest purchasing a cable or adapter for that rather than routing the audio back to the H9000 and then to your monitors via SPDIF.
Once a Direct I/O routing has been made, that output may not have another input source routed to it. So I’m not sure what you mean by having parallel paths where both the analog inputs and outputs of Cubase via MADI are routed to the H9000’s SPDIF outputs simultaneously.