[Grimburger]

> There is a a difference between a human being able to access content vs a search engine indexing it

Much of the problem with search today arises from websites showing googlebot what it wants to see and showing real users. I have to manually remove entire domains from google search as they often appear 1st yet don't show any content without me signing up for an account. Clearly that's not what they are showing to google.

There should be no differentiation between a crawler and a human being with regards to what is being served.