The thing I find most interesting about LLMs is the emergent skills that happen almost instantly(based on model size) that weren't inherently trained for.
Picking movie names from emojis seems like something other than interpolation.
Not really IMO. At scale the model learns that the rainbow emoji is literally a synonym for “rainbow”. Computers are great at memorizing giant lists of facts such as this. It feels way more amazing for an LLM to guess the Carebear Movie from the rainbow emoji and bear emoji but it’s literally the same as asking the LLM to guess a movie that involves rainbows and bears with words.
It’s just an embedding space buried within the bowls of the LLM.
It’s just an embedding space buried within the bowls of the LLM.