Sometimes they use user agents for this, but those are easily faked, so it's only done by websites that don't have comprehensive auth walls.
A more comprehensive method is based on ip ranges, say whitelisting traffic from Google and Bing. This gives you > 95% of search traffic as Google alone has >90%, and many various smaller search engines like Yahoo or DDG are Bing resellers.
On the other hand, a pure ip based check can be circumvented too. Sometimes you can view how search engines see a website through google translate. But places like LinkedIn have countermeasures for all of these circumventions.
A more comprehensive method is based on ip ranges, say whitelisting traffic from Google and Bing. This gives you > 95% of search traffic as Google alone has >90%, and many various smaller search engines like Yahoo or DDG are Bing resellers.
On the other hand, a pure ip based check can be circumvented too. Sometimes you can view how search engines see a website through google translate. But places like LinkedIn have countermeasures for all of these circumventions.