I think I recall reading a paper a while back on free divers that showed there was damage being done to the brain even though it wasn't immediately apparent. Similar to how lots of small brain impacts add up in football players.
I'm almost certain any amount of oxygen deprivation is damaging to the brain. If there were studies that showed conclusions to the contrary, I'd be very curious to read them.