If there was someone I knew who I could ask a question, with a 97% chance they were correct, I'd consider them to be a phenomenon, and an extremely valuable resource. For some definition of correct, I'd be happy to be 60% accurate.
Yeah but what if you got to choose between 2 people that knew the answer 50% of the time but one of the them enjoyed making up answers when they didn't know and the other just told you they didn't know?
This is not about "a grain of salt". Imagine you're a ten year old kid and you're looking for answers on some subject, say the Holocaust. What if all the "answers" you're getting are from far-right hate groups? How do you think that will affect that person?
Likewise, if you're looking for medical advice and you're getting nothing but anti-science woo, can we really be surprised that anti-vaxxers are becoming more numerous? These outbreaks of deadly, yet preventable diseases have serious fatal consequences for many.
"Grain of salt" means fuck all when people are dying from bad answers. You can't learn from your mistakes when you're dead.
At least Google could frame these with the idea of keeping a skeptical mind, but they should probably stop surfacing things as "the answer" and instead as "top search result".
I'm not sure why you feel the need to comment on how credulous I might be.
Anyway, the 97% is an impressive result, it just isn't so impressive as to be beyond criticism. My comment is a little over the top, in response to the other comment that sets the bar at effusive praise being the only proper analysis of the system.
There's no margin of error for the test they did, just potential sampling bias.
Given the 5000 questions they asked, the system will provide the wrong answer 2.6% of the time. Every time, until they improve it. There's a chance that they managed to ask the only 150 questions that it doesn't know the answer to, but not a very big one.
I agree -- given the sample space, there is not effective way to calculate the margin of error. Even so, I don't think that there are many examples of non-deterministic mechanisms that produce the correct answer 97% of the time.
A parser (assuming you are talking about a programming language parser) has the luxury of having highly structured and deterministic inputs, and to be able to refuse giving an "answer" if they are not.