[crazysim]

ChatGPT can't access the external internet. The example ChatGPT instruction where they "teach it" just has the user giving it a URL to their docs and ChatGPT is like "oh ok got it" (Figure 3). I'm not sure if the blog post writers are aware it's hallucinating what the JSON implementation is and hasn't actually read the documentation at all.

[paxys]

There's this internet phenomenon (common on Reddit and other such forums) where you find high-quality discussions on a variety of topics and everyone seems so clever and informative, until you come across a thread on a topic that you are actually an expert on and find that every top-voted comment is objectively incorrect. ChatGPT is essentially that but on steroids. Its primary ability is being able to sound extremely confident in all its responses.

[danielvaughn]

I think a lot of overly-eager and under-informed people are jumping on the bandwagon and shouting half-baked proclamations from the rooftops to get clicks and views.

But underneath all the bullshit is something truly useful, so I wouldn't necessarily say that Confidently Incorrect™ is ChatGPT's primary ability.

Coincidentally, over the past few weeks of this ChatGPT craze, I've needed to create a lot of fake data to seed a database. Normally not a big deal, but these records need to have foreign keys pointing to one another. I wondered whether ChatGPT could help me out, so I briefly described the fields I needed, their format and data type, the type of information they contained, etc etc. It did it almost perfectly, and fixing the little errors was trivial. I was dreading having to do this because it's such a pain in the ass.

To me that kind of thing is going to be the most useful application. Everyone's freaking out (in both the "scared" and "excited" sense) over AI's ability to replace creativity, but I'm focused on it's ability to replace toil.

[paxys]

While I agree with your premise, the "toil" that you mention is a very small use case for a tiny minority of people. And automating it away has always been trivial. What would have taken 30 minutes to write a Python script now takes 5 minutes to come up with the perfect prompt. And then you have to test and be sure that it does what you want. Neat, but not world-changing.

Meanwhile ChatGPT is already being hailed as the Google killer. People are using it to write essays and form opinions. Plenty of online and offline debates contains phrases from ChatGPT repeated verbatim. It's being adopted in newsrooms around the world. So what it is good for in theory and what people are actually using it for are worlds apart. IMO the technology is going to have its "self driving car kills pedestrian" moment very soon because of over-eager and careless users, and that is going to set the entire field back.

[turtlebits]

I disagree. Toil is part of everyone’s job and what creates procrastinators. I used it recently to help me write a browser extension, where I find writing vanilla js and xpath expressions very tedious (just a bunch of syntax lookup) and would not have even bothered in the first place.

If you approach it with the right mindset, it’s extremely valuable.

[SamPatt]

Increasing software development speed 6x would be fairly world changing already, wouldn't it?

Add in all the automation of customer service via actually good Chatbots and lots of very cheap (and fairly high quality) copywriting and it seems world changing to me.

[vkou]

> Meanwhile ChatGPT is already being hailed as the Google killer.

The only Google-killing aspect of it is a better understanding of user queries.

Although given how often I have to remind it of the details of my questions, maybe it's not even good at that.

[fock]

Given it has probably "hardcoded" a lot of questions by the usage of "finetuning" via RL, probably the latter statement is true. Also it has one way only to understand your query. If that way is wrong: welcome synonyms replaced by the tokenizer, welcome hallucinations by raising the temperature. Or you can introduce "context" (8000 tokens) or retrain.

[hungryforcodes]

> ...a better understanding of user queries.

No, lack of ads. Before content marketing, Google was pretty useful

[vkou]

How do you think it's going to be monetized, if not by ads?

[soco]

Like said somewhere on the internet, we don't need AI to create art, we need AI to do the dishes and free our time to create art.

[theonlybutlet]

But art's the easy part, I wonder if that's why it's fun...

[hungryforcodes]

...not done well though. Dishes are much easier to do well.

[prox]

When I want a quick insight to launch me into further inquiry, chatgpt has been so much more useful than regular search with the first 10 hits of AI generated advertisement spam. I use one that provide its source as well, so I can check.

Reddit tends to be a second best bet, and the most upvoted is not always the best, on a busy thread some ‘real’ experts are usually a bit down with lengthy descriptions of their analysis.

Search is third, it’s quick but I do often need to append stackoverflow and reddit to get meaningful answers. I forgo Google completely more and more since the quality is so low, perhaps 2% of my queries go there tops.

[ikekkdcjkfke]

Absolutely. I know what i want, i know how to do it, but i don't want to type it out.

[Felminor]

Just imagine how many little code snippets you used (at least I did) to make my life easier and faster

A script which parses jaon for a specific key or transforming a table from y to z.

Now everyone has this available to them.

It's great already!

[harperlee]

But those are the low hanging fruit that help me avoid rusting skills… and this is a common phenomenon on automation: you destroy the first rungs on the experience ladder, that no one wants to do but are essential to develop foundational skills.

[danielvaughn]

That's a pretty good point. I'm looking at this from the perspective of having done this for over a decade. All that stuff feels like a waste of time to me now. Even as I was using ChatGPT for my fake data, I was thinking to myself "this is like having a junior dev assisting me". So yeah, those little micro tasks that build muscle memory are going to vanish.

[kristopolous]

That's almost every reddit thread I've been involved in.

Oratory prowess is way more important than accuracy. If you ever want to "lose a debate" on reddit, pick something that's counter-intuitive or widely misunderstood and use nuance and citations in your defense. You'll get cyberbullied every time - sometimes even banned from the sub.

[smogcutter]

From those examples, I think you’re maybe also running up against people generally having a low threshold for giving a stranger on the internet the benefit of the doubt.

It’s hard to be surprised that “eugenics, but the good kind, just hear me out!” and “state’s rights, but not the disingenuous lost cause kind, just hear me out!” are a tough sell with complete strangers who have no prior reason to trust you or be interested in your opinions.

[kristopolous]

correct!

Btw, I removed them from my examples, didn't want to "discuss" that here.

I'm kind of a history nerd and it's like a fishing hook with delicious bait when I see stuff like that. "Oh wait! You think it was just the south that argued states rights?! Actually (brings up fugitive slave clause jurisprudence)"

Or how manner and conduct books of the 1800s, eugenics texts, and jordan peterson's 12 rules for life are basically dots in a straight line. That stuff is just catnip.

I just need to stop being distracted and go write actual books or something.

[nqrac]

Peterson an academic who is clueless about the real world. His best work was discussing politics in connection with liberty, especially the Trucker protests.

His biggest flaw is that (like the other Daily Wire pundits) he never goes into economics and has absurd ideas that we live in a competence hierarchy.

The whole "intellectual dark web" imploded after the start of the Ukraine war, when they first went full neocon, then realized that their audience does not like that and now start backpedaling massively.

[kristopolous]

I'd like to recommend "decoding the gurus" podcast which is probably the best treatment of this material I've seen.

It's still lacking but I should "put up or shut up". They're doing a fine job

[pfannkuchen]

I’m no fan of Peterson, but I’m curious— how are intentionally altering the gene pool of a group and cleaning your room colinear?

[kristopolous]

Look at the table of contents of this book titled "Eugenics" https://archive.org/details/naturessecretsre00shan/page/6/mo...

There were multiple points of interest for the Eugenicists. One of them was for the general public and how to get them to have a proscribed conduct and behavior.

Maybe you've seen those diagrams where they have 2 life paths depicted from the Victorian era. That's from Eugenics. Example: https://ia800301.us.archive.org/BookReader/BookReaderImages....

Here's "The Road to Success" for "The Young Man" https://archive.org/details/naturessecretsre00shan/page/244/...

Again, this is part of Eugenics. We focus on the part that was removed (race science) and leave mostly unexamined the parts that were reinvented or repurposed as say, "The Power of Positive Thinking".

Eugenics books, for the general public, were a different beast. Probably the best succinct description for how they thought of themselves can be summarized in this 1921 diagram: https://images.prismic.io/wellcomecollection/efc97918-8073-4...

It is a dangerous psuedoscientific epistemology - a practice with a bunch of faulty logic and ways of knowing that can lead to disasters and it's alive and well - for instance, assuming a social structure in lobsters applies to humans. That these adherents tend to be revealed as secretly racist and thus subscribe to the part which was removed, should be of no shock.

We've mischaracterized Eugenics[1] and allowed it to fester by other names or as Maurice Bardèche, the french neo-fascist observed about fascism in the 60s, "With another name, another face, and with nothing which betrays the projection from the past, with the form of a child we do not recognize and the head of a young Medusa, the Order of Sparta will be reborn"

The point is that if you don't draw the perimeter right around an idea, you can't reign it in. It'll just escape from your hands and re-emerge as something else. So, for instance, the slave plantation becomes the prison farm (https://en.wikipedia.org/wiki/Prison_farm).

We tend to, for good reason, ignore societal trainwrecks and blights on our past. But trainwrecks are the most important thing to study if we strive to build safer trains and seek to avoid similar disasters in the future.

Things have to be accurately tackled with in order to defeat them.

---

[1] it's worth noting it's a big topic and some "real" science has also come out of it, such as biosocial theory and biostatistics studies. Peer reviewed journals such as https://en.wikipedia.org/wiki/Journal_of_Heredity used to be eugenics journals. This is, however, a very small minority.

[jiggawatts]

I post on both Reddit and YC News regularly, and it is fascinating to see the differences in biases, counter-arguments, or outright rejections of reality.

YC News people generally seem to prefer to hear the truth, especially from sources that appear knowledgeable. I've had people here comment in reply to me saying I've "changed their mind". That never happens on Reddit.

Reddit has cliques, quasi-religions, and fads. Anything contrary to the group-think is mercilessly voted down. Intelligent, thought-out counter arguments that aren't entertaining are especially likely to garner lots of downvotes. Not only do the readers feel "personally attacked" by contrary opinions, there is also that ever present undercurrent of anti-intellectualism of typical Americans that makes intelligent discourse distasteful to them.

[avgcorrection]

It’s amazing to me in all my years on the internet how many communities I insert myself into, then in private feel like a pretender, and yet how relatively uncommon it is that people call me an idiot.

[smcl]

Good afternoon, fellow Imposter Syndrome Haver :) Just jumping in to say that you are most likely not a pretender or an idiot, you're probably just very self-conscious, hold yourself to high standards and give yourself lofty goals.

[alophawen]

Can confirm. Was banned from /r/golang for speaking well of Rust. I use both.

[pyuser583]

I’ve been quizzing ChatGPT about it’s knowledge of the oil industry - it has no idea what’s talking about.

For example, I ask it to simulate a discussion between a flow engineer and a geologist, and it doesn’t seem to understand they work together regularly.

Not like there’a a stack overflow for this sort of thing.

[stevenhuang]

you need to prompt it with more specific knowledge; if you're generic in your prompts it will be generic in its responses. inject some field-specific terminology and be esoteric in your queries, etc

its latent representation is vast, I find you can get much higher quality+interesting responses if the prompts are tuned appropriately

[IIAOPSW]

This is at best going to slightly improve the output. I asked about something hyper specific and also extremely trivial: transit wayfinding type queries specific to the NYC subway. All the semantics are services and stations. A service is nothing more than a list of stations it stops at and a station is nothing more than a list of services which stop at it. Eg if I tell it "I'm on the F, how do I get to Time Sq", it should cross reference the service/station lists, find the commonality, and tell me "switch to the A at W 4th".

No context about anything in the outside world or even about the physical nature and purpose of commuting is needed to comprehend and answer these questions. There is no external context or sophisticated logic beyond just matching those sorts of lists to each other. Its a wholly closed system of highly standardized tokens.

I ask it what stops the F and A have in common. 125th St is on the list. I know that's wrong. I ask it to list all the services that go to 125th. The F is not on the list (correctly so). I point out the inconsistency between the two outputs. It says sorry and does nothing. I tell it to remake the list of stops the F and A have in common. It is now missing a few stations from before, and has added new incorrect results.

This is just a snippet. I went in circles with it for probably an hour playing wack a mole with its inability to correctly recall more than 10 true details in a row. These were not obtuse, esoteric, or even logically complicated queries. Nor was it ambiguous stuff open to interpretation. Nor was it even something that you'd need a real meatspace body to comprehend, like the feeling of the sun on a summer day. This should have been a language models bread and butter.

[stevenhuang]

You're right, it seems particularly bad in navigation. I lead with some general queries on the NYC system, asked it for the common stops for service lines F and A, and it also hallucinated. The wiki for the service lines have pretty complete representations here, so this data should be suitably represented in its training corpus.

Would be interested in how a multimodal LLM such as PaLM-E (trained on maps, etc) fair in these sorts of queries.

https://www.reddit.com/r/MachineLearning/comments/11krgp4/r_...

[IIAOPSW]

I'm not hopeful that "seeing the map" (multimodal training) will make the difference everyone is hoping for. The transit map is going to be the exact same information as the lists, but coded in the learned design language of colored lines and dots. The words-only version should work just as well or better because it shortcuts the implicit OCR problem of trying to make it learn off the map. Indeed transit maps other than NY are often abstracted and have nothing to do with the underlying geography. So abstract representations (such as lists of words) should be fit for purpose.

Here's another one that fails spectacularly. The digits 0-9 drawn as an ASCII 7 segment display. It gets it mostly correct, but it throws in a few erroneous non-numbers and repeated/disordered/forgotten numbers. Asking it for ASCII drawings of simple objects can really go off the rails quickly.

The fault mode is very consistent. When a prompt forces it to be specific and accurate on an unambiguous topic for 10 or more line items, it will virtually always hallucinate at least one or two. Especially if the topic is too simple to hide behind a complex answer. Even if its learned not to hallucinate 90% of the time, and even if that's good enough to pass at first glance, within a list of 10 things it only has a 35% chance of not hallucinating any of them.

For what its worth, it did very well on law questions. Try as I might, it refused to accept that there's a legal category known as "praiseworthy homicide". Though, I suspect this has less to do with the underlying model and more to do with openAI paying special attention to profitable classes of queries.

I'm sorry to say, I think the problem maybe more intrinsic to the current approach to AI. Frankly, LLM works unreasonably beyond my expectations, but its making up for a lack of a real first principles theory of cognition with absurd amounts of parameters and training.

[ChatGTP]

Here’s the thing though, when you keep having to “prompt it”, are you really doing the work yourself by coaxing a good enough bullshit answer that you’ll believe it? It could still be useful, not sure.

[stevenhuang]

A minimum sort of knowledge of the field would be helpful.

But I feel that if one is inquisitive enough, they'd know when exploring a topic to ask broad questions of the sort:

- what are the common types of problems encountered in X field

- what are the main roles and responsibilities of personnel in X field

- list some open questions in the field of X

After these open-ended questions one can dive into each of them with:

- what are some common solutions to X, compare and contrast X vs Y

The risk of seeing hallucinated info is always there, but just for an example when I was exploring the types of space-time metrics and asking chatgpt which tensors are most suited for which scenario (and also which coordinates are best for which scenario, such as moving from Schwarzchild to Eddington Finkelstein coordinates to deal with the event horizons near black holes), their individual wiki pages said generally the same things.

I was able to ask ChatGPT in very naïve "idk please explain like im 5" ways and it was able to very patiently rephrase dense jargon into surprisingly understandable explanations.

I'm sure if an expert in GR were to drill down deeper into the details they will find inaccuracies in ChatGPT's responses, but for the inquisitive information sponge user, a dive through ChatGPT to get a general feel of a topic is insanely useful. After that, one could dive into the appropriate books/papers/primary sources to start the real learning once they find what they're interested in.

[aliasxneo]

Can you give an example? I see this advice a lot but usually devoid of practical tips.

[stevenhuang]

Here's an attempt. Looks pretty good to me.

It's sort of like, how would you imagine an expert would respond if all you said was "simulate a discussion between a flow engineer and geologist". Without providing any further context or hints into what you want, I also wouldn't be surprised if I get a generic response. But instead if you sort of know what you want to ask, they'd have more to work with and be able to better answer your query.

Note I know nothing about this field, I just looked at https://en.wikipedia.org/wiki/Engineering_geology#Soil_and_r... and understood this was for oil and gas, so:

> Simulate a discussion between a flow engineer and a geologist in the planning phases of a drill operation. Use field-specific terminology and aim for accuracy, the scenario will be used to teach a petroleum engineering class focused on deformation monitoring.

Flow Engineer: Good morning, I'm the flow engineer for this drill operation. Nice to meet you.

Geologist: Hi, I'm the geologist in charge of the geological analysis for this project.

Flow Engineer: Great, I understand we need to work together to ensure the success of this drill operation. Can you tell me more about the geological features of the area we're drilling in?

Geologist: Sure. We're drilling in an area with significant deformation, caused by tectonic activity. There are a variety of rock formations, including sandstone, shale, and limestone. There's also a possibility of encountering fault zones, which could pose a risk to the drilling operation.

Flow Engineer: Thanks for the information. We'll need to take into account the deformation and rock properties when designing the wellbore trajectory. Have you conducted any geomechanical analysis?

Geologist: Yes, we've conducted a rock mechanics study to determine the stress and strain characteristics of the formations in the area. Based on the results, we recommend using a multilateral well design with a horizontal section to avoid encountering the fault zones and minimize the impact of deformation.

Flow Engineer: That's helpful. We'll need to ensure that our drilling fluid properties are appropriate for the formations we'll be encountering. Can you tell me more about the porosity and permeability of the rock formations?

Geologist: The porosity and permeability of the sandstone formations vary, but in general, they're relatively high, which means we'll need to be careful when selecting our drilling fluid to avoid fluid loss into the formation. The shale formations are less porous and permeable, which means we'll need to use a different type of fluid to avoid damaging the formation.

Flow Engineer: Thanks for the information. We'll take that into account when selecting our drilling fluid. We'll also need to monitor deformation during the drilling process. Do you have any recommendations for deformation monitoring tools we should use?

Geologist: Yes, we recommend using acoustic emission sensors to monitor rock fracturing and deformation, as well as tilt sensors to monitor any changes in the wellbore inclination. We also recommend conducting periodic borehole imaging to ensure that the wellbore is remaining stable.

Flow Engineer: Excellent. We'll ensure that we incorporate those tools into our drilling plan. Thank you for your time and expertise. I'm looking forward to working with you on this project.

[fock]

Oh yeah - and now I change this and suddenly the thing is fun and slightly unreal:

"Simulate a discussion between a project manager and a public health officer in the planning phases of a drill operation. Use field-specific terminology and aim for accuracy, the scenario will be used to teach a petroleum engineering class focused on public relations."

GPT: > Project Manager: Good morning, I'm glad to have you here to discuss the planning phases of our upcoming drill operation. As a project manager, my priority is to ensure that all aspects of the drill operation are conducted in a safe and efficient manner.

> Public Health Officer: Good morning. As a public health officer, my role is to ensure that the drill operation does not pose a risk to public health and the environment. I'll be happy to discuss our strategies for achieving that goal.

> Project Manager: Great. One of the first things we need to do is to identify potential health hazards associated with the drill operation. As you know, drilling involves the use of various chemicals, which can pose a health risk if not handled properly.

> Public Health Officer: Absolutely. We'll need to carefully review the chemicals being used and ensure that they are being handled in accordance with proper safety protocols. We'll also need to develop emergency response plans in case of accidental spills or leaks.

> Project Manager: Agreed. Another concern is air quality. The drilling process can generate dust and emissions, which can have adverse effects on air quality.

> Public Health Officer: That's right. We'll need to monitor air quality throughout the operation to ensure that it meets the standards set by the Environmental Protection Agency. We'll also need to take measures to minimize dust and emissions, such as using dust control equipment and properly maintaining all equipment.

> Project Manager: And what about water quality? We'll be drilling in close proximity to bodies of water, and we need to ensure that we don't contaminate them.

> Public Health Officer: Yes, that's a critical concern. We'll need to make sure that all wastewater and other fluids are properly contained and disposed of. We'll also need to monitor water quality regularly to ensure that there are no leaks or spills.

> Project Manager: It sounds like we have a lot of work to do to ensure that this drill operation is safe and environmentally responsible.

> Public Health Officer: Indeed. But by working together and following proper protocols and procedures, we can ensure that the operation is successful without posing a risk to public health or the environment.

I guess there was some confusion about roles/at the existing temperature roles swap (if you rerun it, you get different results, essentially swapping the lines...)

EDIT: and third try actually arrives at really nice hallucinations:

> Project Manager: Good morning, Dr. Smith. Thank you for taking the time to meet with me today. We are in the planning phases of a drill operation and we want to ensure that we are following all necessary public health protocols. Can you provide some guidance on what we should consider?

> Public Health Officer: Absolutely, it's my pleasure to help. The first thing you should consider is ensuring that all workers on-site are up-to-date with their immunizations, particularly for diseases such as tetanus and hepatitis.

> Project Manager: Understood. We require all our workers to have up-to-date immunizations before starting work, but we will double-check to ensure that all workers have the necessary immunizations.

> Public Health Officer: That's great to hear. In addition to immunizations, you should also consider providing training on proper hygiene practices, such as hand-washing and avoiding contact with contaminated materials.

[oefrha]

In the pre-LLM world we already had rumors/myths/lies/propaganda becoming "facts" after being reinforced through repetition without verification. (Relevant xkcd: Citogenesis https://xkcd.com/978/.)

Now, instead of authoritative sounding humans who are possibly wrong or with an agenda, we have an "infallible", "impartial" oracle capable of inventing whatever it wants to. Yes, I have seen ChatGPT treated as infallible on this very forum: "you are wrong because ChatGPT says <insert nonsense here>". Can't wait to see LLMs taking "post-truth" to a whole new level. The propaganda potential is immense, to point out just one application.

[theonlybutlet]

Perhaps it's a good thing, everyone will have to become experts in discerning the truth. Much like Video Game graphics looked "real life" until the next gen came out, perhaps society will start to see through any rumours/myths/lies/propaganda. Either that or we descend into a state of anomie.

[oefrha]

Doubt it. I can hardly verify things happening in my city even if I try, let alone e.g. things happening on the other side of the globe which the media keeps telling me about in order to drum up the next cold/hot war. I used to be able to tell Photoshop jobs by non-experts, but soon enough everyone and their mum will probably be able to produce completely fake videos indistinguishable from real ones, so even "video evidence" will mean little.

[brewtide]

100% on how it "takes over the world". We treat it like an oracle, hand it over willingly in mindset alone.

[nuclearnice3]

Before the had the internet, back when we still had newspapers, we had a similar phenomenon referred to as Gell-Mann Amnesia.

> Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues.

> In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.

https://www.epsilontheory.com/gell-mann-amnesia/

[tshaddox]

I think the effect is a bit overblown. Yes, I have noticed coverage of topics I know about that are oversimplified or straight up mistaken. But these are technical topics that I don’t expect journalists to get right (and in many cases I’d prefer they didn’t try to cover them at all). And I also won’t implicitly trust coverage about other technical subjects that I’m unfamiliar with.

I would argue that national politics and international affairs as they are covered in mass news media is a different type of coverage. It’s not about learning technical details and explaining them. It’s more about observing actions and broad sentiment and literally reporting those observations. I wouldn’t trust a reporter to get scientific details correct about some chemical process that’s the subject of some debated EPA regulation, but I could trust a reporter to summarize the debate and report on the sentiment of politicians, public sentiment based on opinion polls, etc. It would be weird to say “that newspaper screwed up the difference between a dominant 7th chord and a major 7th chord in a story about a famous songwriter, therefore I can’t trust them to report which politicians gave the most heated arguments in the debate about the EPA bill.”

[zoklet-enjoyer]

Check out this excerpt from a local news story I saw yesterday. You know what else acetic anhydride is used in? Production of aspirin. This is basic scare tactics to manipulate people and I see it everyday. A combination of ignorance, laziness, and being the voice of those in power. Fake news and manipulation of the public.

https://www.valleynewslive.com/2023/03/09/fargo-cbd-store-ow...

"“Semi-synthetic derivatives, or chemically derived cannabinoids, refer to certain types of substances that are produced by converting a cannabis extract into a different substance through chemical reaction,” said forensic scientist with the North Dakota State Crime Laboratory Charlene Rittenbach at the March 3 committee hearing of the bill.

For example, according to the Addiction Prevention Coalition, to make THC-O: you need to extract CBD, extract Delta 8 THC from the CBD and then add acetic anhydride.

The National Library of Medicine says acetic anhydride is corrosive to metals and tissue, and it’s used to make fibers, plastics, dyes and explosives."

[zoklet-enjoyer]

This is why I've said Trump is right about fake news. When I say that people think I'm a weirdo Trumper.

[nuclearnice3]

One nuance might be that those are two difference senses of fake news. In one case, we have news that is wrong. In another case, we have an insult aimed at reporting that we don't like, regardless of accuracy.

[zoklet-enjoyer]

I haven't trusted the news since the time I participated in manipulating local news stations to create stories about jenkem. More modern equivalent is Momo. It's all fake

[kragen]

This sounds interesting! What did you do?

[remarkEon]

Gell-Mann Amnesia Effect but at scale. I find this all the time when reading the NYT, or Washington Post, or whatever, about something I'm intimately familiar with. ChatGPT just does this as a sort of carnival trick, and people seem to think it's funny for some reason.

https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes...

>Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.

>In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.

[LordAtlas]

Reddit version of Gell-Man Amnesia: https://www.johndcook.com/blog/2021/01/18/gell-mann-amnesia/

[WelcomeShorty]

> There's this internet phenomenon (common on Reddit and other such forums) where you find high-quality discussions on a variety of topics and everyone seems so clever and informative, until you come across a thread on a topic that you are actually an expert on and find that every top-voted comment is objectively incorrect.

Sounds like the old days to me, where we would meet up , discuss the world and what have you, without any means to fact check anything online.

Changes there was a real expert on the subject in your circle was at best 20% So we just enjoyed / continue our life. No harm done as long as nobody took the discussion as gospel.

[squarefoot]

That was also common on TV. Everyone seems credible, until we spot errors when someone in our field starts talking.

edit: minor typo

[helsinkiandrew]

When I've asked ChatGPT about things I know a little about I find its response is often wrong. So much so that I always respond with "Are you sure that's correct?" it then often responds "I'm sorry you are right, I should have said... " and then says something a little closer to the truth.

I wouldn't rely on ChatGPT or any system that relies on it, except perhaps for summarising/rephrasing a single, known document. It worries me that people are using it for software development - creating stubs, bootstrap code. It's perhaps OK if you know the language/platform well enough to fix issues, but if you then use it on something that your not an expert in, what is the outcome.

For example:

Finding the source of literary quotes - miss attributing them - took 5 or 6 QA responses for it to get close.

Explaining how a p-channel enhancement mode MOSFET works - explained n-channel

Explain regex - often gets the technical parts right but can make very bad suggestions on what the regex aim is.

Name some classic cocktails created in Europe - suggested Manhattan, which it said was acceptable because it was popular in Europe

Suggest some gin based cocktails that don't include citrus, suggested those with lemon juice.

[chrstphrknwtn]

This generalises very broadly.

I have some expertise and knowledge is one or two specialised fields, and when I read Wikipedia, news articles and blogs about those fields, it's maddening how incorrect many people are, whilst sounding perfectly knowledgeable.

[nonethewiser]

Therefor you have to verify the answer or accept that it may be wrong.

I’m finding that it’s fairly good for code snippets. You can test that pretty quickly. But asking it something you don’t know and can’t verify is a huge risk.

[Kiro]

> common on Reddit and other such forums

It's especially common on Hacker News.

[Pxtl]

"mansplaining as a service" is the term I've seen used. Or Artificial Cliff Calvin.

[dragonelite]

I compared ChatGPT to a middle tech manager that just heard he has to create a last minute presentation. He knows all the current buzz words to sounds smart.

[alextheparrot]

My first recollection of this sentiment pertained to the New York Times, for what it’s worth.

[gonzo41]

Same effect happens with newspapers. Check your sources.

[lordnacho]

Sounds like internet version of Gell-Mann effect

[taskforcegemini]

it is not limited to the internet. this is also applicable to any news company

[_0ffh]

I don't think it's an internet phenomenon. It's a human phenomenon. It doesn't matter if it's a conversation or a piece in a newspaper (c.f. the Gell-Mann amnesia effect).

[ceejayoz]

> The example ChatGPT instruction where they "teach it" just has the user giving it a URL to their docs and ChatGPT is like "oh ok got it".

Yep. I asked it to summarize a URL. It happily invented something completely fabricated and convincing sounding (until you go to the URL) based of the domain name and article slug.

[micromacrofoot]

we’ve made the startling discovery that we’re rather predictable

[taberiand]

My hypothesis is that we will soon see the emergence of a super-human artificial intelligence, not because of some astounding breakthroughs in our understanding of intelligence, but because of irrefutable proof that humans just aren't all that complicated to surpass.

[oceanplexian]

Agreed, artificial intelligence doesn’t need to turn into Skynet to be “super human”, just better than the average 85-90 IQ human.

[tayo42]

That's funny, someone must have given these ai bots an iq test by now. I wonder how it did

[avgcorrection]

Right. But that tech will be created by humans, somehow.

These misanthropist slogans run up to contradictions so easily.

[micromacrofoot]

I don't think it's so contradictory, we have a long history of stumbling into technological advancement. We have a tendency to get ahead of ourselves before we even understand what we're doing.

[rhaway84773]

I think this is right.

Setting aside the creativity aspect (ie where do brand new ideas come from), which is a minuscule fraction of human thought, I suspect we are good pattern matching machines with the ability of extrapolation.

I think with sufficient data that computers can understand, this is eminently replicable.

[pyuser583]

People used to say “AI wont surpass humans, we’ll meet them in the middle.”

[jahsome]

I asked it to generate me documentation for a handful Bicep definitions and include links to the official Microsoft Docs. It generated about 50% erroneous links, but once I updated my prompt and fed it the "root" of the MS docs I wanted it to use, it was able to correct all the urls.

At the time it _seemed_ like it wasn't just doing a mindless string replace, because different parts of the url changed on most of the erroneous links. I figured it wasn't going out and spidering the link I gave, but it led me to believe maybe there was _some_ form of index that it simply hadn't prioritized prior, but was able to do with the additional context.

[jimbokun]

Wouldn’t it be wild if it just guessed the URLs correctly, because it had seen so many URLs from Microsoft URLs it was able to correctly predict the form of URLs it hadn’t seen yet?

[colechristensen]

These models have probably been trained on a large portion of the Internet.

The training data however can be a few years old.

I have noticed this when trying to get it to generate some code, it was clearly using considerably old versions.

[codebje]

> I have noticed this when trying to get it to generate some code, it was clearly using considerably old versions.

I have noticed the same thing when trying to use junior devs to generate some code, usually correlating with how old the most popular Stack Overflow question on the subject is.

[colechristensen]

This is how we decided to describe chatgpt’s programming ability the other day. A very fast but somewhat inadequate jr developer. Is amazing really but you also have to ask for the same thing a dozen times and still spend quite a while figuring out what’s wrong with it.

[Godel_unicode]

What I have been doing is asking it for code and then asking it to find bugs in that code. The bug finding works relatively well.

[teaearlgraycold]

copy.ai's chat bot can read URLs now

[swatcoder]

Hah. Of course the authors are not aware.

OpenAI handed this tool to people with effectively no guidance or instructions so that they can trade on the glow of misconceptions and harvest user data. The whole point of giving the thing away for free these last few months is to let people make fools of themselves like this while pumping up OpenAI’s value as a technology firm. It’s no accident that they don’t engage to clarify all these persistent misunderstandings.

[goatlover]

The concern is the increasing number of anecdotes where people are taking ChatGPT's output at face value, even though everyone knows by now that it does sometimes get stuff wrong and does hallucinate. I thought it would take longer.

[HideousKojima]

A very annoying trend I've already started noticing on some forums (including here unfortunately) is someone asking a question and one of the replies is someone saying "I put your question into ChatGPT and it said..."

[Lewton]

I downvote and flag every chatgpt generated comment, even if they’re up front about it

If I want to know what chatgpt thinks about something, I can ask it myself

[breakingcups]

Case in point: https://news.ycombinator.com/item?id=35031737

[rideontime]

Apparently everyone doesn't!

[dcow]

The problem is that people also sometimes get stuff wrong and hallucinate. So ChatGPT is just mimicking how we behave and it seems completely normal.

[ehnto]

While it's true humans are fallible, it is more clear what someome should and should not get wrong, and we can make safer assumptions about the knowledge we get from certain individuals. ChatGPT doesn't "know" anything, you can get it to be right and wrong about the same detail in the same conversation. It's not clear what it should be correct about, and so it's a pretty poor source of information.

[dcow]

And then there are trolls.

[renewiltord]

> OpenAI handed this tool to people with effectively no guidance or instructions so that they can trade on the glow of misconceptions and harvest user data.

When one logs on to Chat GPT, they get the following notices:

> This is a free research preview.

> Our goal is to get external feedback in order to improve our systems and make them safer.

> While we have safeguards in place, the system may occasionally generate incorrect or misleading information and produce offensive or biased content. It is not intended to give advice.

> Conversations may be reviewed by our AI trainers to improve our systems.

> Please don't share any sensitive information in your conversations.

Effectively no guidance or instructions indeed. Perhaps ChatGPT is sometimes confidently incorrect because it learned from online commenters.

[swatcoder]

Did you read those statements?

None of them say what the product is, what it does, or how to use it — which are the kind of thing one might call guidance and instructions.

Those are just disclaimers to make sure you don’t hold OpenAI responsible for anything you do with this thing, or blame them for anything they do with your attempts.

[reacweb]

They say what it is: "This is a free research preview. Our goal is to get external feedback in order to improve our systems and make them safer."

It is not a product. They open it so that public can contribute to help them.

They do not say the aim of this research.

[creatonez]

This is baseless speculation

[derefr]

It can't access the internet now. But it was fed a snapshot of the Internet during training. So if you tell it a URL, it can potentially remember what the page looked like, if it happened to see it during training.

[yencabulator]

In either scenario, it's hallucinating. Either JSON queries were supported in that snapshot of the documentation, or they weren't.

[simonw]

That's very unlikely. ChatGPT doesn't contain a full copy of everything it saw when it was trained - it has model weights that were updated by that training process.

It's very unlikely the model weights would contain enough detail to let it match content to the URL that the content originally came from.

[grugagag]

You’re saying ChatGPT has access to a an older snapshot of the internet with URLs and everything?

[polishdude20]

Is it possible that it "remembers" that url from it's training and the content tied to it?

[Retr0id]

Assuming the earlier statements from ChatGPT were correct, the feature did not exist before the training data cutoff, so that's impossible.

[visarga]

I don't think they prefix each document with the URL in the training set, but there might be URLs in the clear text.

[nextaccountic]

> ChatGPT can't access the external internet.

ChatGPT by itself can't but if you build a product on top of its API, you can fetch the url itself and send the data in the ChatGPT's prompt

[honeybadger1]

It also can be prompted to read the contents of a webpage, I have used it to help me craft a regex expression for removing junk utilizing invoke-webrequest to parse out CPU models compatible with 22H2 upgrades from Microsoft(that page did not exist in 2021).

[IanCal]

You mean it prompted you to go and get the contents of the page? Chatgpt does not access the internet when running. Bing chat does though.

[honeybadger1]

No, I do not mean that. I provided it the URL and then asked it to use the contents of the CPU table from the invoke-request to strip away everything after the X and it not only did it, it even provided the example output of the array it modified.

Here is an excerpt from my chat:

" can you please modify the current array using the Microsoft listed CPUs in totality from the URL https://learn.microsoft.com/en-us/windows-hardware/design/mi...

Certainly! Here's the updated $compatibleCPUs array based on the list of supported Intel processors for Windows 11 version 21H2 from Microsoft's documentation:

"

[simonw]

It hallucinated the list of processors based on the URL.

Here's an experiment I just did to prove that it hallucinates content based on URLs: https://fedi.simonwillison.net/@simon/109998914078891574

Or try it yourself: Try these two prompts in separate ChatGPT sessions:

> List the processors on this page: https://bit.ly/422t51T0

Then:

> List the processors on this page: https://learn.microsoft.com/en-us/windows-hardware/design/mi...

That Bitly link is one I just created that redirects to the Microsoft page.

[honeybadger1]

Interesting, and certainly convincing. I have a paid account, I wonder if this makes any difference? It would make sense to me that it is basing it's 'reality' off of the prevous 21h1 page(which would have existed) and it understands the goal I am trying to achieve.

[IanCal]

It's not accessing that page live (or recently cached). That's not part of how it works. It's possible the model was updated and has learned from that, but it's also very possible it's just made up the answer. The old model cutoff was 2021 I think, I don't know if/what newer training has been done.

[simonw]

Even Bing can't make outbound HTTP requests - it can only access cached snapshots of pages held in the Bing search index. I confirmed this on Twitter with one of the Bing execs: https://twitter.com/mparakhin/status/1628646262890237952

[IanCal]

Cool, I really hope they open this up. It feels like being able to restrict it to either a subset of public pages or private data would immediately be extremely useful and only a small conceptual change from what's there.

I just built this locally over our RFCs and it's good but a properly integrated thing would be fantastic.

[lumost]

ChatGPT is also highly familiar with what keywords to search for.

[IIAOPSW]

Jokes on you. The entire blog post you just read, ChatGPT. There never was a database.

[password1]

> ChatGPT can't access the external internet.

This might not be entirely true. People were able to fetch urls and data from the internet that couldn't possibly be in any training dataset (in eg posted the same day for the first time).

https://medium.com/@neonforge/i-knew-it-chatgpt-has-access-t...

[svane]

This actually shows perfectly how convincing ChatGTP can be. The author is too eager to believe the output from ChatGTP, even though this should be a clear indicator something else is happening:

> But ChatGPT has rewritten my article in a different style, what a interesting finding…

The URL conveniently contains the topic of the blog-article, meaning ChatGTP can 'hallucinate' the contents, even without internet connection.

To show this, I tried to follow the same steps as @neonforge, but with a non-existing URL. Unfortunately `lynx` 'didn't work', but curl showed some interesting information.

`curl https://medium.com/@peter/learn-how-to-use-the-magic-of-terr...` returned some html. Instead of a 404 and a <title> element containing just 'Medium' it hallucinated the following title tag: `<title>Learn How to Use the Magic of Terraform Locals (Step-by-Step Guide) | by Peter | Medium</title>`

I'm not saying that ChatGTP definitely has no access to internet, but so far I have not seen any proof or indication that it does.

[clementneo]

I don't think this shows at all that ChatGPT can access the internet? It looks like ChatGPT was given a URL with a title and hallucinated a blog post based on that title. The author then brushes it off as 'written in a different style', when it just looks like a totally different article overall (or maybe it's testament to the fact that too many Medium articles are low quality and indistinguishable from the output of a LLM!)

If anything, this blog post is a perfect example about how people can put in whatever they want as an input and take the output as truth, without any rigorous approach about what would count as a true fact about the model.

[chrismorgan]

That article is bunkum. There’s not the slightest reason to believe that it “rewrote his article in a different style”: it hallucinated a completely different article based on the URL. The other “proof” is all clearly wildly inconsistent and hallucinated: it’s a desktop machine, a server, inside Google in I think France, in the USA, in AWS, in Germany, and I think the last couple of “tests” don’t make sense anyway.

It convinced him of what he wanted to believe. He didn’t do a very good job of testing.

[simonw]

That was all hallucination.

If you want to prove this to yourself, tail a server log and paste a URL into ChatGPT and see if you get a hit.

[pazimzadeh]

How is it getting information partially correct, but not able to verify the correctness to at least be internally consistent with itself?

[Groxx]

Because it has essentially no logic system whatsoever. It doesn't have any way to "know" something is correct because when it produces a word for word copy of a factual statement, it is internally identical to a completely made up (hallucinated) sentence... because both are hallucinated. The deriding "it's fancy autocomplete" remarks are, in a lot of ways, technically correct.

You can still do useful things with it, that much is extremely clear. But it doesn't think.

[jchanimal]

It’s been worth it to me to feed it code. Here a session log from an encryption discussion https://docs.google.com/document/d/1e2RV-ytnwTq1LRY0BRPgNWCr...

[andrethegiant]

It absolutely will be able to access the external internet

[andrethegiant]

Told ya