[impendia]

I was in a research math lecture the other day, and the speaker used some obscure technical terminology I didn't know. So I dug out my phone and googled it.

The AI summary at the top was surprisingly good! Of course, the AI isn't doing anything original; instead, it created a summary of whatever written material is already out there. Which is exactly what I wanted.

[Arisaka1]

My counterpoint to this is, if someone cannot verify the validity of the summary then is it truly a summary? And what would the end result be if the vast majority of people opted to adopt or deny a position based on the summary written by a third party?

This isn't strictly a case against AI, just a case that we have a contradiction on the definition of "well informed". We value over-consumption, to the point where we see learning 3 things in 5 minutes as better than learning 1 thing in 5 minutes, even if that means being fully unable to defend or counterpoint what we just read.

I'm speficially referring to what you said: "the speaker used some obscure technical terminology I didn't know" this is due to lack of assumed background knowledge, which makes it hard to verify a summary on your own.

[gosub100]

At least with pre-AI search, the info is provided with a source. So there is a small level of reputation that can be considered. With AI, it's a black box that someone decides what to train it on, and as someone said elsewhere, there's no way to police its sources. To get the best results, you have to turn it loose on everything.

So someone who wants a war or wants Tweedledum to get more votes than Tweedledee has incentives to poison the well and disseminate fake content that makes it into the training set. Then there's a whole department of "safety" that has to manually untrain it to not be politically incorrect, racist etc. Because the whole thesis is don't think for yourself, let the AI think for you.

[impendia]

If I needed something verifiable, or wanted to learn the material in any depth, I would certainly not rely on an AI summary. However, the summary contained links to source material by known experts, and I would cheerfully rely on those.

The same is true if I imagined there would be misleading bullshit out there. In this case, it's hard to imagine that any nonexpert would bother writing about the topic. ("Universal torsor method" in case you're curious.)

I skimmed the AI summary in ten seconds, gained a rough idea of what the speaker was referring to, and then went back to following the lecture.

[auntienomen]

A lot of the time, the definitions peculiar to a subfield of science _don't_ require much or any additional technical background to understand. They're just abbreviations for special cases that frequently occur in the subfield.

Looking this sort of thing up on the fly in lecture is a great use for LLMs. You'll lose track of the lecture if you go off to find the definition in a reference text. And you can check your understanding against the material discussed in the lecture.

[lazide]

The issue is even deeper - the 1 thing in 5 minutes was probably already surface knowledge. We don’t usually really ‘know’ the thing that quickly. But we might have a chance.

The 3 things in 5 minutes is even worse - it’s like taking Google Maps everywhere without even thinking about how to get from point A to point B - the odds of knowing anything at all from that are near zero.

And since it summarizes the original content, it’s an even bigger issue - we never even have contact with the thing we’re putatively learning from, so it’s even harder to tell bullshit from reality.

It’s like we never even drove the directions Google Maps was giving us.

We’re going to end up with a huge number of extremely disconnected and useless people, who all absolutely insist they know things and can do stuff. :s

[sam_goody]

I have a counterpoint from yesterday.

I looked up a medical term, that is frequently misused (eg. "retarded"), and asked the Gemini to compare it with similar conditions.

Because I have enough of a background in the subject matter, I could tell what it had construed by its mixing the many incorrect references with the much fewer correct references in the training data.

I asked it for sources, and it failed to provide anything useful. But once I am looking at sources, I would be MUCH better off searching and only reading the sources might actually be useful.

I was sitting with a medical professional at the time (who is not also a programmer) and he completely swallowed what Gemini was feeding him. He commented that he appreciates that these summaries let him know when he is not up to date with the latest advances, and he learnt alot from the response.

As an aside, I am not sure I appreciate that Google's profile would now associate me with that particular condition.

Scary!

[kenjackson]

This is just garbage in, garbage out. Would you better off if I gave you an incorrect source? What about three incorrect ones? And a search engine would also associate you with this term now. Nothing you describe here seems specific to AU.

[apothegm]

The issue is how terrible the LLM is at determining which sources are relevant. Whereas a somewhat informed human can be excellent at it. And unfortunately, the way search engines work these days, a more specific search query is often unable to filter out the bad results. And it’s worst for terms that have multiple meanings within a single field.

[kenjackson]

That word "somewhat" in "somewhat informed" is doing a lot of lifting here. That said, I do think that having a little curation in the training data probably would help. Get rid of the worst content farms and misinformation sites. But it'll never be perfect, in the same way that getting any content in the world today isn't perfect (and never has been).

[apothegm]

It’s not even about content farms and misinformation. It’s about which of the results even are talking about the same topic at all. You should have seen what came up when I searched info about doses for a medication that comes in multiple forms and is used for multiple purposes. Even though I specified the form and purpose I was interested in, the SERP was 95% about other forms and purposes with only two that were topical to mine in the first two pages. (And yes, I tried the query in several forms with different techniques annd got substantially the the same results.) The AI summary, of course, didn’t distinguish which of those results were or were not relevant to the query, and thus was useless at best, and dangerously misleading at worst.

[solumunus]

Try the same with Perplexity?

[FridayoLeary]

I have to agree. People moan that the ai summary is rubbish but that misses the point. If i need a quick overview of a subject i don't necessarily need anything more then a low quality summary. It's easier then wading through a bunch of blogs of unknown quality.

[rsynnott]

> If i need a quick overview of a subject i don't necessarily need anything more then a low quality summary

It's true. I previously had no idea of the proper number of rocks to eat, but thanks to a notorious summary (https://www.bbc.com/news/articles/cd11gzejgz4o) I have all the rock-eating knowledge I need.

[jennyholzer2]

In my experience Google's AI summaries are consistently accurate when retrieving technical information. In particular, documentation for long-lived, infrequently changing software packages tends to be accurate.

If you ask Google about news, world history, pop culture, current events, places of interest, etc., it will lie to you frequently and confidently. In these cases, the "low quality summary" is very often a completely idiotic and inane fabrication.