Evaluating the quality of language models is a challenge in itself. This post just presents another way to see how much your model can understand alongside a new model. This is typical when presenting new tech for which old evaluation methods might not tell you anything.
It's not really about getting the answer to that question as it is about figuring out how much information your model can glean from text.
That's a heck of a question to expect an AI to answer. It's weird (we almost never discuss half animals), oddly phrased ("on the moon" is much more standard), and runs afoul of an imprecision in English (we normally given masses when asked about weight, this requires an answer in force, or else by analogy a mass with equivalent Earth-weight).
It's because the name of the moon is "The Moon", not "Moon". I think SF writers sometimes pretend it's called "Luna" so it'll have a more interesting name.
Thanks! I'm indeed not native :)
In my own defense it "felt" like it was missing something. However I am confident there are many moons and just one Moon, our own.
Also, the answer is 1.62kN to 4.8kN, I know Wolfram doesn't employ a language model or anything of a sorts, but with all the other NLP magic I've seen I sort of expected a valid answer.
Bleh.. .English and it's weird grammar .. Sometimes I find it such a messed up language compared to my native tongue..... I guess it's the separation of modifiers and nouns that's part of the problem.
It shouldn't really be that difficult. If it's capable of structuring "How heavy is _ on the moon?" The answer would be start out the same way looking for weight of _.
It seems to me that these kinds of questions could be handled but require some adjustments or tuning more than we need fundamentally different approaches. We can't do everything at once so there's going to be simple things that don't work for quite a while.
Well, yes. A B+ high-school student can also reliably recognise the number 710. If you think that AI should be able to do something just because a high-school student can, I think it's your expectations which are out of touch.
It is a tough question, but we keep hearing from AI-boosters about how the Singularity is right around the corner and have you seen this AI that can make plausible-looking text?
Needs a supervisor for cheeky monkey curve ball playful question to interpret answer on gradient of seriousness from thousands of mind model with access to knowledge library and language as commonly used. The part before "on" doesn't sound out of place from a template for language exchange in a butcher's shop and "on Moon" is a playful reframing adjusting the physics model. A child lucky to have a parent with training in physics would get an answer easy to this question.
Or it simply needs to be augmented with something other than GPT. The form of my question is very easily solvable in a programmatic way, just not by a GPT based neural net.
Someone mentioned multi model training, that sounds awesome!
Nice answer, but it actually supports the point being made here. NLP is an attempt to see how far we can go in processing natural language, without doing this sort of translation, and without structured knowledge bases. As (I'm pretty sure) Wolfram Alpha contains a structured knowledge base and draws on it for this sort of question (though not without help in this case), its abilities are orthogonal to the issues addressed in the article.
Well even if you take the most symmetric halves by volume, the density of an ideal elephant is not uniform. Some organs don’t come in pairs and all that
I guess it depends on how many hairs you want to split and how precise you want the answer
However most unpaired organs don’t contribute to a mass asymmetry. It’s thought this has evolved to aid with locomotion as any asymmetry will negative impact on efficiency
It's not really about getting the answer to that question as it is about figuring out how much information your model can glean from text.