Similarly, when I think of ChatGPT as a really cool and advanced search engine frontend, its behavior - including its limitations and its failures - make the most sense to me.
It's a language model, not a search engine. It doesn't work well as one unless integrated into an actual search engine, like Bing does. Without such integration, it's much closer to human memory than search engine - it will recall stuff it has seen many times pretty well and completely fail at stuff it just glanced over once, filling any gaps with made up stuff like a kid on an exam hoping to get at least a few points with their wild guesses.
Yeah, I think we're talking about different things (and per my comment, I didn't say that it was a search engine). I'm reasonably well aware of what it is and what it's made of; I'm talking about a mental model for understanding and predicting when and why it works well vs when it doesn't.
And what I've found so far is that when I place it in the same mental bucket as the interface to a modern search engine (not the search engine itself, but the interface for both input and output), it actually fits in pretty well there in many ways. Not in every way, of course, but things like the nuances of crafting prompts and how a scarcity or abundance of reference material affects its output.
> I'm talking about a mental model for understanding and predicting when and why it works well vs when it doesn't.
I'm talking about it too. If I enter a specific phrase into a search engine that can be only found on a handful of websites, I expect it to return those results to me. Like, typing the VAT ID of my company will return bunch of information about it on various sites. This is absolutely not going to work with a LLM - instead, at best it may notice that what you typed looks like a VAT ID and will then proceed to give you information about a company it completely made up. The mental model of understanding what works with LLMs and doesn't is drastically different from a search engine. Human memory on steroids is a much better (though of course still not perfect) model.
Again, we seem to be talking past each other, sorry. I'm really, really, really not talking about the search engine itself. I'm talking about the hunk of tech that makes up the interface layer between the human and the search engine, and the fact that that hunk of tech can be hooked up to a search engine is interesting but not entirely germane.
If using the analogy of human memory works for you - that's great! To me, it's not as good a fit, but that's ok.
> The mental model of understanding what works with LLMs and doesn't is drastically different from a search engine
Agreed! But again, that's not what I'm talking about. :)
> I'm talking about a mental model for understanding and predicting when and why it works well vs when it doesn't.
That's what you said earlier you were talking about, and that's what I replied to. Now you're saying that you're in fact not talking about "the mental model of understanding what works with LLMs and doesn't" at all. Seems you have to improve your communication skills mate ;]
What I'm saying is that using LLMs while imagining them to be kinda like search engines is just a way to get burned by hallucinations and disappointed with poor results. They don't work even remotely similar to search engines, neither internally nor for an external observer. For some kinds of input they may trick you into believing they actually do, but that impression will fall apart pretty quickly once you try to actually exercise it. That's how you get people who are genuinely shocked that ChatGPT gave them references to papers that were completely made up, for example - which is something that shouldn't surprise anyone using this tech at all, as that's just how it works.
> I found that as search engines emulated natural language, their results got steadily worse
I would wager that that has not been the experience for the general population (read: non-technical people) and/or that degradation of results has not been because of emulating natural language but because of other factors (like advertising dollars).
Search engines have become incredibly more accessible for non-techies during the past 3 decades. Sure, even today a techie is usually able to coax higher quality results out of a search engine, but it's still a pretty recent advancement that an average Joe can just announce their question out loud and a device on the shelf will not only figure out what they are asking with a decent degree of accuracy, but it will also go search for something relevant, extract an answer, and then speak it back to the user in a pretty sensible way.
It is in this senses in particular that ChatGPT feels like a natural progression for search engines.
I completely agree that my experience has not been the same as the the general population's. But that doesn't really help me. My searches are still worse. Just find me pages that match the text I specify please. Add some boolean operators and I'm happy.
And because the majority of people have a better experience, I dismiss your second option of other factors being at play.
> And because the majority of people have a better experience, I dismiss your second option of other factors being at play.
That's fine, though the point I was (clumsily?) trying to make was that there are different factors here that allow multiple things to be true at the same time: power users routinely feel like search result quality is going down, and I think you can pretty objectively show that to be true in many cases.
Simultaneously, though, the barriers for "normal" people to do decent searches have dropped dramatically - there was an accessibility hurdle that was previously challenging for a lot of people and it's incredibly better now vs just a few years ago. This too, I believe, can be shown to be objectively true in many cases. (anecdotally as well - just last week I watched a number of very un-technical senior citizens get what they wanted out of Google and I didn't see much evidence that it was because of their skill at crafting good search queries).