[petepete]

But it's frustrating that search engines are able to index the whole page, so things I don't have access to are listed in search results and I can only find out when I've clicked.

[mdre]

Well, you could change your user agent string.

[Brybry]

This hasn't worked well in a decade or more.

I used to pretend to be Googlebot a lot to bypass paywalls but I think these days there is either IP range filtering or more advanced fingerprinting.