This is, as you say, very weird. Changing the ADC/DAC settings should have no effect on the AES XLRs. Could you send us your session so we can look at the routing?
The setting for the digital audio mode is there because some resources are shared between the various digital audio interfaces, so it seemed simplest for users to decide on one they want to use. Specifically, receivers are shared between the AES, XLR and SPDIF inputs so you can't run all of those at the same time. However, more flexibility is possible than we currently offer the user. For example, ADAT is routed separately from the other digital IO, so you could use all 8 ADAT channels and all 8 AES channels. Or you could use XLRs and some, but not all, of the DB25 channels. We are planning to re-think this in a software update.