Your problem may be caused by the fact that ADAT really only works at 48K or below – this is limited by the capability of the optic fibre.
At higher speeds, ADAT can be converted to something known as S-MUX, where each ADAT channel occupies two of the 8 channels. Sounds like what you are seeing.
