Yeah, you have to simulate the attenuation from different directions - HRTFs (head related transfer functions)[1] are used to do this. They're already supported by some games, and you have to use them for VR audio, since the user is in control of the camera position and angle.