[agent_turtle]

There was a study recently that showed how not only did devs overestimate the time saved using AI, but that they were net negative compared to the control group.

Anyway, that about sums up my experience with AI. It may save some time here and there, but on net, you’re better off without it.

[keeda]

That study gets mentioned all the time, somehow this one and many of the others it cites don't get much airtime: https://www.stlouisfed.org/on-the-economy/2024/sep/rapid-ado...

>This implies that each hour spent using genAI increases the worker’s productivity for that hour by 33%. This is similar in magnitude to the average productivity gain of 27% from several randomized experiments of genAI usage (Cui et al., 2024; Dell’Acqua et al., 2023; Noy and Zhang, 2023; Peng et al., 2023)

Our estimated aggregate productivity gain from genAI (1.1%) exceeds the 0.7% estimate by Acemoglu (2024) based on a similar framework.

To be clear, they are surmising that GenAI is already having a productivity gain.

[agent_turtle]

The article you gave is derived from a poll, not a study.

As for the quote, I can’t find it in the article. Can you point me to it? I did click on one of the studies and it indicated productivity gains specifically on writing tasks. Which reminded me of this recent BBC article about a copywriter making bank fixing expensive mistakes caused by AI: https://www.bbc.com/news/articles/cyvm1dyp9v2o

[keeda]

The quote is from the paper linked in the article: https://s3.amazonaws.com/real.stlouisfed.org/wp/2024/2024-02...

It's actually based on the results of three surveys conducted by two different parties. While surveys are subject to all kinds of biases and the gains are self-reported, their findings of 25% - 33% producitivity do match the gains shown by at least 3 other randomized studies, one of which was specifically about programming. Those studies are worth looking at as well.

[foolswisdom]

It's worth noting that the METR paper that found decreased productivity also found that many of the developers thought the work was being sped up.

[keeda]

Yes, self-reporting has biases and estimating tasks is still a fool's errand, which is why I noted that the estimates from these surveys matched the findings from other RTC studies.

However, what doesn't get discussed enough about the METR study is that there was a spike in overall idle time as they waited for the AI to finish. I haven't run the numbers so I don't know how much of the increased completion time it accounts for, but if your cognitive load drops almost to 0, it will of course feel like your work is sped up, even though calendar time has increased. I wonder if that is the more important finding of that paper.

[rockemsockem]

I'm not talking about time saving. AI seems to speed up my searching a bit since I can get results quicker without having to find the right query then find a site that actually answers my question, but that's minor, as nice as it is.

I use AI in my personal life to learn about things I never would have without it because it makes the cost of finding any basic knowledge basically 0. Diet improvement ideas based on several quick questions about gut functioning, etc, recently learning how to gauge tsunami severity, and tons of other things. Once you have several fundamental terms and phrases for new topics it's easy to then validate the information with some quick googling too.

How much have you actually tried using LLMs and did you just use normal chat or some big grand complex tool? I mostly just use chat and prefer to enter my code in artisanally.

[lisbbb]

How much of that is junk knowledge, though? I mean, sure, I love looking up obscure information, particularly about cosmology and astronomy, but in reality, it's not making me better or smarter, it's just kind of "science junk food." It feels good, though. I feel smarter. I don't think I am, though, because the things I really need to work on about myself are getting pushed aside.

[rockemsockem]

For me it's pretty much all knowledge that I'm immediately operationalizing. I occasionally use it to look up actors and stuff too, but most of the time it's information that provides direct value to me

[flkiwi]

This is kind of how I use it:

1. To work through a question I'm not sure how to ask yet 2. To give me a starting point/framework when I have zero experience with an issue 3. To automate incredibly stupid monkey-level tasks that I have to do but are not particularly valuable

It's a remarkable accomplishment that has the potential to change a lot of things very quickly but, right now, it's (by which I mean publicly available models) only revolutionary for people who (a) have a vested interest in its success, (b) are easily swayed by salespeople, (c) have quite simple needs (which, incidentally, can relate to incredible work!), or (d) never really bothered to check their work anyway.

[fzeroracer]

Why not just look up the information directly instead of asking a machine that you can never truly validate?

If I need information, I can just keyword search wikipedia, then follow the chain there and then validate the sources along with outside information. An LLM would actually cost me time because I would still need to do all of the above anyways, making it a meaningless step.

If you don't do the above then it's 'cheaper' but you're implicitly trusting the lying machine to not lie to you.

[rockemsockem]

See my previous statement

> Once you have several fundamental terms and phrases for new topics it's easy to then validate the information with some quick googling too.

You're practically saying that looking at an index in the back of a book is a meaningless step.

It is significantly faster, so much so that I am able to ask it things that would have taken an indeterminate amount of time to research before, for just simple information, not deep understanding.

Edit:

Also I can truly validate literally any piece of information it gives me. Like I said previously, it makes it very easy to validate via Wikipedia or other places with the right terms, which I may not have known ahead of time.

[fzeroracer]

Again, why would you just not use Wikipedia as your index? I'm saying why would you use the index that lies and hallucinates to you instead of another perfectly good index elsewhere.

You're using the machine that ingests and regurgitates stuff like Wikipedia to you. Why not skip the middleman entirely?

[rockemsockem]

Because the middleman is faster and practically never lies/hallucinates for simple queries, the middleman can handle vague queries that Google and Wikipedia cannot.

The same reasons you use Wikipedia instead of reading all the citations on Wikipedia.

[fzeroracer]

> Because the middleman is faster and practically never lies/hallucinates for simple queries

How do you KNOW it doesn't lie/hallucinate? In order to know that, you have to verify what it says. And in order to verify what it says, you need to check other outside sources, like Wikipedia. So what I'm saying is: Why bother wasting time with the middle man? 'Vague queries' can be distilled into simple keyword searches: If I want to know what a 'Tsunami' is I can simply just plug that keyword into a Wikipedia search and skim through the page or ctrl-f for the information I want instantly.

If you assume that it doesn't lie/hallucinate because it was right on previous requests then you fall into the exact trap that blows your foot off eventually, because sometimes it can and will hallucinate over even benign things.

[lisbbb]

A lot of formerly useful search tools, particularly Google, are just trash now, absolute trash.

[agent_turtle]

OpenAI is currently being evaluated in terms of hundreds of billions. That’s an insane number for creating a product that “speeds up searching a bit”.

[rockemsockem]

Did you miss the part where I said it helps me find new knowledge that I wouldn't have otherwise?

That is pretty significant in my book.

[agent_turtle]

I don't know how you use search but I often find incredible information that I didn't explicitly search for.

How do you quantify such things? How can you say with a straight face that this magic box gives you more relevant information (which may be wrong!) and that will revolutionize the workforce?

[rockemsockem]

I am searching for the incredible information and I can't find it without the LLM because I don't know the proper terminology ahead of time and search isn't that good unless you know exactly what you want.

[ares623]

Is the "here and there" tasks that were previously so little value that they are always stuck in the backlog? i.e. the parts where it helps have very little value in the first place.