Here's how I'm seeing this: in any room, there are reflections from walls, furniture, whatever. The reflected waves can reflect again (higher order reflections), allthough attenuated. These reflections are non uniform over frequency, hard objects absorb low frequencies, soft objects absorb high frequencies. To make things even messier, high frequencies are more directional.
So, what gets in the end to your ears: direct wave from speaker, add delayed reflected waves with frequency content modifiend by reflecting object, add waves that reflect from more than one object with longer delay because of higher travelled distance, and frequency content even more affected. Put simply, this is an acoustic mess.
What is my question: why do speakers provide though better stereo imaging than headphones that do not suffer from such acoustic problems ?