I have to say that it's been bizarre watching people say how much of a "game changer" LLMs are. There is clearly an advance there, no doubt, but it hasn't really been able to solve any actual problems that I have.
They won't replace anyone yet, but you're sleeping on AI if you think it's not super helpful. For programming it has pretty much replaced Google for me, as I can ask very detailed questions that are specific to my code. Of course you should always double check the work, but it has definitely made me more productive.
I did have a bit longer comment typed out but didn't follow through on it. That is to say that yes, for common code (or as a developer told me, stuff you've written before and just can't be bothered to find again), it can knock stuff out in seconds that might take me a few hours, so in that sense it is very impressive. But that's just writing code that's been written many times already, and I'm not a developer anyway. I think it shows that in truth, few of the issues we solve are novel.
You're not supposed to ask it to implement something novel or large scale. You ask it how to use an api, or to refactor some code with a new class you made. You stack these up, and you can be way more productive than the person doing the equivalent of manual labor.
It's a free text box, you can ask it anything you want, but yes; the answers to large scale questions have been quite lacking in my experience. But as a systems guy rather than a developer, those tend to be the actual problems that I have.
Yeah the "check the work" thing is key. I think the reason these work so well as coding assistants is that we already have really mature tools for checking the work: compilers, static analyzers, test suites, etc.
It's not like writing a history paper where you can write "Napoleon was a French general who conquered Europe in 1987" and just be wrong in a way that requires cross-checking facts. Computers are less forgiving of things that are wrong.
But I do feel like I see incorrect documentation proliferating a bit more, and I'm curious whether we'll see an up-tick in more subtle bugs that is wrong but not so wrong that it is rejected early on. I haven't personally seen that in my work yet. I do also see more code that I don't like during code reviews, that I'm sure was spat out by an AI, but often I conclude that my preference isn't really "right" or even better, but just a preference that doesn't really matter.
There’s a lot of marketing around llms, with related spam everywhere. For a while i tried using chatgpt for instance as a replacement for google search, but it fails even at that. I do still use it for basic queries but can never be trusted to be reliable.
Same. Heck, I went to SO just to find an obscure piece of info that help me solve my issue and saved me 1 or 2 days of asking chatGPT the "right" question. ChatGPT is good but you need a "Jarvis" type AI to be Tony Stark and even then Tony must point it in the right direction.
It's a common argument here: "LLMs are overhyped, because they could not solve my super niche issue".
Their current status is basically "very junior programmer with a photographic memory, who has read a sizable chunk of the internet". They are excellent if you regularly need to implement code that has been done before.
Well, given that a good chunk of the audience is treating LLMs as the next step in human intelligence, you tend to be very critical, trying to prove to the other side this is not what they're claiming to be.
Every now and then it's helpful! I hate how it tends to "guess" for me, though. And it's not hard questions, it's exactly the kind of boilerplate it's supposed to be good at. I'll ask things like how to connect to the Amazon API in Python for some quick boilerplate, or about a built in library in Python, and it'll just make stuff up. I often have to yell at it to not make stuff up 1 or 2 times before it gives me "real" code.
If I'm doing something obscure or niche forget it, it'll just make up a bunch of nonsense. But I wish it was easier to get it to say "I think it should work like x but I'm not sure about y part" instead of just going all in and making stuff up.
We need to train it on old LessWrong posts so it learns about degrees of belief and such lol
Have you tried the paid version of ChatGPT? GPT 4 is way more capable than the free 3.5 version you get without paying.
I use it to work through architecture problems that are over my head or find answers to things that aren’t obvious in documentation. These are things I used to have to ask staff engineers for advice with. I’ve even had it write large sections of code for me. At this point most of the code it puts out works.
You can also do things like ask ChatGPT to use multiple specific technology stacks to implement your solution, which is really useful for when you’re trying to come up with a solution that fits within existing infrastructure.
It’s like a mega-brain that’s an expert in everything and understands your specific prompt without giving you the junk you don’t want that has similar words.
I feel like it's a question of what "game changer" means. I find it incredibly useful! It helps me a lot in doing my job every day. I wouldn't personally say it has "changed the game" for me, but I think others might say that, just because the actions I take each day (my "game") is different now (has been "changed").
But the way those actions have changed just also isn't that big a deal. I have much better "intellisense" by using copilot, and I have much better stack overflow via interactive chat that will tell me answers immediately rather than linking to a kinda-sorta-the-same duplicate question after an hour. But using code completion and searching stack overflow weren't activities that took up much of my time. They took up some of my time, and it's really awesome that I'm using that time much more efficiently now, but it's still a small proportion.
I still spend most of my time reading code and docs, talking to my colleagues about things, and thinking about how to structure interfaces and larger components in order for them to be more useful and to evolve more easily as we hire more people and adapt to different business initiatives. The AI assistants help me some with these tasks, sometimes I ask them to summarize stuff I'm reading, or (honestly hardly ever) to help me understand unfamiliar code, but these are quite small value adds. They aren't anywhere near the point where they can replace my work on these tasks. They just can't store enough context on the business or do enough different kinds of things or take enough initiative to do this part of my work.
All of this is to say: I'm bullish on "LLMs as coding assistants" but - for the moment - remain bearish on "LLMs as replacement for a software engineer".
If your job gets replaced by an LLM, then you are probably not active on a level that has you reading Hacker News.
I'm sure more evolved LLMs will begin to replace jobs soon, but it will mostly be the low hanging fruit. By the time it gets to what I'm doing we'll be in Wall-E floating chair [1] ubi territory.
Your problems might be more complex, but it helped me, a noob programmer with 6 months of experience, launch a full fledged product that I now charge people real money for. Practically all the code was written by GPT-4.
Can this product scale to 10M users? No clue. Does it have all the features that I wanted? Not yet. But is it live, usable, useful and profitable?
Would be cool to detail what “full fledged product” chatgpt wrote for you. Usually these statements are vague and never reveal what exactly the product or code are, and when they aren’t it turns out that the author is not exactly a “noob” and the code wasn’t entirely written by chatgpt and is not more than the equivalent of a google search.
What LLM are you using? I use ChatGPT and StableDiffusion semiregularly. Maybe I just suck at prompts, I find them very hit and miss, even for very simple tasks.