Maybe we can use a neural net to fill in the shadowed regions. I have only looked at 2D inpainting algorithms and not sure what the state-of-the-art in 3D inpainting is though.
Right, the shadowed region does make the scene look less realistic, since there is no such thing in the real world. While the lazy part of me says it's okay to leave it this way, 3D inpainting is definitely something I should look into.