If you can't access IPv6 enabled sites, you can't access Google, Facebook, Wikipedia, or Netflix, who all enabled IPv6 permanently in 2012[1]. According to [2], around a third of US has IPv6 connectivity. If AAAA records constitute some big problem, it's not well known.
I'm not sure what you're referring to when you say "Thats not how it works".
I was referring to people with broken IPv6 setups, mostly Teredo and 6to4 (old Mac OS X versions), for whom IPv4 worked but IPv6 was not actually routable. At the time, publishing AAAA records would lock out all of those users.
Oh, I see now. You're reading "Having IPv6 access is a necessary for accessing Google". What I meant was "If the presence of AAAA records means you can't access a site, then you can't access Google".
A user who only has IPv4 connectivity relies on their software recognizing this when accessing an IPv6 enabled site. Software's failure to do so is one of the main reasons for the slow rollout of IPv6.