Unfortunately, could be many things. A few questions:
1) If you run your Protools without connecting to the H8000, do you get the distortion ? If no, it is not the Avid I/O. If yes, it is not the H8000.
2) Have you installed the latest beta drivers ? These are also needed for Lion.
3) You say you are using the AES routing. What have you set as the H8000 sync source ?
4) Are the connected AES lines showing as "locked" on the H8000 ?
5) Does the Firewire show as "locked" on the H8000 ?
5) What are the AES lines connected to ? Are there any other digital inputs connected to the H8000 ?
You should not rush to down grade, both because V5.6 is probably not the problem, and also because it is needed for Lion.