[dimatura]

I eventually ended up switching fields to ML (and that's my current day job), but I started out as an undergrad studying psychology/cognitive science. During those years I started a research project on what is called the "feeling of knowing", or FOK for short - a subtopic in the broader topic of "metacognition" in cognitive science. The FOK is closely related to what is colloquially knows as tip-of-the-tongue phenomenon - basically, that subjective feeling that we know something, whether or not we can actually recall it. There's some interesting aspects to it. For example, the FOK is generally pretty accurate (but not perfect, of course). And it tends to be more robust than actual memory; as we age, for example, we tend to be better at judging than we know an actor's name (and confirming it once we look it up) than recalling it. It seems like LLMs have very little in the way of metacognition, and just confabulate if they don't know something, as we've seen. I'm sure we'll be seeing some efforts to give some analog of a FOK to LLMs in the near future.