People are different. Some notice latency at around 4-5 ms, others at 20+ ms, and I talked to a guy who was okay with 50 (he played ambient stuff with tons of effects and swells).
I think the psychoacoustic aspect also plays a role. If you see a speaker or sound source at a distance, your brain will expect the sound to be delayed and it will feel natural. Not so much If the sound is artificially delayed with a speaker close up or headphones.
