Here's my concern with this. Is that the AI is good at generating "mostly correct" code, that will have subtle errors, so less skilled coders just take the output of whatever the LLM spits and assume it's correct, later it turns out there is a bug. The person who submitted the code doesn't understand the code, because they didn't bother learning it just that it was mostly correct. Instead of discovering and fixing the source of the error they just go in and discover where the error is output and slap some more AI generated code over it to squash the bug.
Now you may be saying "what's the big deal with that? That happens all the time all over the place."
The issue is that the biggest thing AI brings is it allows you to generate lots of code, very fast. So we have more and more code, that is less and less understood by any human being.
Eventually no one has any idea what the code is doing in the first place, and what ends up happening is the Cult Mechanicus of Warhammer 40k we become tech priests running around performing arbitrary and inane rituals to appease "the machine spirit" because no one anywhere has any idea how the dang program actually works.
Like we've all heard horror stories of the Oracle DB code base, imagine if that started to become common. I mean people talk about how AI helps them generate code faster, but to be honest I find most issues are the result of too much code, rather than too little.
My point is I grow concerned about the ability of AI to generate code and the effect that is going to have on increasing the amount of code while reducing understanding.
Here is an anecdote to illustrate how "The Machine TM" tried to get me fired.
For context, Microsoft introduced a "AI revamped Intellisense" in Visual Studio (not Code) a few months ago. It's not Copilot, it just makes suggestions but only for your next line of code.
It's Okish most of the suggestions are good, it saves time on repetitive tasks.
Yesterday I was writing some code to disable some customers.
I had a list of all customers and a list of customers to disable.
Intellisense offered to pass the list of all customers to the method that will disable them. If I wasn't paying attention, next Monday (and the rest of the week really) would have been quite "entertaining".
> Is that the AI is good at generating "mostly correct" code, that will have subtle errors, so less skilled coders just take the output of whatever the LLM spits and assume it's correct, later it turns out there is a bug.
Not unlike the code generated by the fresh out of bootcamp, careless rookie, who isn't trying to make things work so-much-as collect a paycheck. But hell, at this point I need a job too, maybe I should swallow my morals and submit AI solutions as well?
Sure, but I think the imagined hellscape is one where the entire stack of developers don't know what they're doing.
It's like the ostensible toaster company that has outsourced design, engineering, manufacturing, marketing, sales, HR, everything I’ll the way to the janitor. The question then is: what does the toaster company even know about making toasters? If you play this game long enough, then you arrive at a point where entire sectors operate this way. Keep going and the entire economy works this way. At some point we have to stop outsourcing, or we'll lose a lot of knowledge when the original creators of that knowledge kick the bucket. At this point we can't rely on that knowledge being encoded in language models.
My hope is this will mostly translate into being able to distinguish between who really knows what they're doing (the AI is a tool to be used for some tasks and its work is to be scrutinized like anything I'd find on GitHub or elsewhere) and those who are barely hacking it in the field, but I also worry about what you're describing.
Like color/exposure auto-balancing tools in video editing. They can be really useful and can often produce solid color/exposure results! But most of the time they only get you 80% of the way or sometimes just spit out really ugly results. Ultimately auto-balancing didn't ruin editing - mediocre editors can't get by on it - so hopefully we don't see that here!
yes that's true, and sincerely keep going with that maybe another pass could do code review
maybe an elaborate workflow, but each individual step in that workflow isn't insurmountable
if you cannot get the correct answer in one shot, instead a workflow is required, we should just write off this whole thing as useless or setup that workflow? will let the reader decide
> so less skilled coders just take the output of whatever the LLM spits and assume it's correct, later it turns out there is a bug. The person who submitted the code doesn't understand the code, because they didn't bother learning it just that it was mostly correct. Instead of discovering and fixing the source of the error they just go in and discover where the error is output and slap some more AI generated code over it to squash the bug.
Ever worked with offshored programmers (won't name country or companies)? The things I've seen. Getting code via email that doesn't compile. Then seeing they replied-all, cc'd some random employee not on the project and 2-3 managers about "getting the urgent fix". Having a teenager with a 3 months bootcamp, an AI and a few post-it with git commands would actually be an improvement over this.
> The issue is that the biggest thing AI brings is it allows you to generate lots of code, very fast. So we have more and more code, that is less and less understood by any human being.
I know guys whose whole contracting careers is re-writing (completely from scratch) huge codebase that were offshored. 10x reduction in line count is pretty much standard. I guess these AIs means their business model will keep working for the next decade!
What you're describing will primarily happen at incompetent orgs without good development practices, code review, good management, good hiring, etc.
If anything, a tool that reveals and punishes incompetence is good for everyone. I'm not the world's most enthusiastic capitalist, but in this case, I think we can let the market sort it out.
Without expert filtering of the suggestions, yes, disaster. Experts will benefit the most from this because they can filter incorrect suggestions and reinterpret. Novices will be more susceptible.
There is certainly ongoing research on this, expert systems have been an area of research for a while. Here is recent research relevant to this post: Do Users Write More Insecure Code with AI Assistants? (https://arxiv.org/pdf/2211.03622.pdf)
Indeed. Copilot is like productivity rocket fuel for me. More importantly, less typing means less strain on my hands/wrists and a longer career.
For junior developers I wouldn't recommend it, because as you say, they don't have the pattern matching to find and fix the hidden errors that copilot generates. Also, it's harder to conceptualise code that you haven't written.
I would compare it to the way that chess players use engines. Grandmasters can have an engine turned on without it being distracting, and they can tell when the engine is suggesting a good move vs a weird move, and explain what the move accomplishes.
Beginners, are generally advised to play with the engine turned off, and to instead analyse their moves afterward, because otherwise they will just play the engine moves and not learn anything.
But most joiners will be the ones that will use it the most, trying to leapfrog their lack of experience with AI - which will provide the biggest issue imho
This was the point that drove me to write the article in the first place. I think that there should be some kind of knowledge threshold before you can use assistants. I think the same point can be applied to why students need to know how to calculate some equation without a calculator.
“Automobiles: a convenient crutch for people who can’t ride horses, or a recipe for disaster?”
“Word processors: a convenient crutch for people who can’t type, or a recipe for disaster?”
God these headlines are going to be hilarious in ten years.
Yes, AI code is primitive today. The first implementations sometimes get basic stuff wrong and often get complicated stuff wrong.
But the state of the art is evolving daily. It’s somewhere between ignorance and gatekeeping to act like the issues we see today (which are already much fewer than we say a year ago) are so endemic to the concept of generated code that the whole thing is a terrible idea.
Most code will be written by AI. You can choose to adapt and leverage that, or you can choose to be that guy who insists that your field is the one true place that we should resist automation (for the good of the world, of course, not any self-interest, that’s just a coincidence).
>God these headlines are going to be hilarious in ten years.
We'll see. I think you're pulling an "apples to oranges" here, comparing the AI situation to the advent of automobiles and computers/word processing.
The big difference is the typical application of the technology. By comparison, AI is absolutely RIFE with fraud & lazy get-rich-quick schemes - on an absolutely unprecedented scale & scope. Just like cryptocurrency: art generation/NFTs, SEO arms race, generated blogs, generated articles, etc. If you look at ChatGPT prompt repositories, these type of prompts are virtually always filling the top several pages when sorting by ratings/views.
I'm more than a little concerned with where things are headed as the internet gets more and more crapped up by AI generated hot-air, riddled with factual inaccuracies. Another concern is, as more companies adopt & rely these tools for regular use, they often downsize their workforces in the process and lose valuable subject matter experts (knowledge) along the way.
There's also the fact that this is going to compound in later training models. I'm not sure where exactly the demarcation point will ultimately lie, but I think we're going to end up thinking of the Internet as two main epochs: "pre-AI" and "post-AI".
IMO: All AI-generated or AI-assisted content should be clearly tagged as such.
I understand the concerns, but I think you’re assuming that we are frozen in time. I think this is a brief moment where text is either AI generated or not.
In a year or two, leveling content as AI assisted would be like labeling something “electricity assisted” today; you’d just label everything (California prop 65 style).
Everything will have some level of AI involved. I mean even today we have spell checkers and grammar checkers and automatic summarizations that tons of people use. The gradient from that to text expansion and whole document creation is going to be blurry and gradual.
With you on pre-AI and post-AI epochs. It will be sort of like the way the internet itself caused pre-security and post-security epochs, and we’re still dealing with that.
But I think it’s great. Yes, we have to grow up. Yes, our tools have to get better. But just like the internet’s benefits far outweigh the security pain, I feel pretty good about AI cost/benefit ratios.
Sure, just as soon as we get those “contains code from new college hires” label I’ve been asking for.
Relying on perfect coding skills is a losing game. Way better to have healthy code review and testing processes that assume your developers are fallible.
The idea of AI assistants is good. The hype around current AI assistants is bad. You may comfortably assume everyone critical of LLMs' current performance believes this. Begone, straw men.
I hate to even reply to this because it's falling into the author's trap. But, it's the typical new-tech clickbait. There's no data to back up their alarmist, knee-jerk hand wringing, just like when people thought novels were bad for people [1], or when writing would ruin our memories [2]. This article will be looked at similarly in a few years.
> There's no data to back up their alarmist, knee-jerk hand wringing, just like when people thought novels were bad for people [1], or when writing would ruin our memories [2].
Before we knew about germs the "data" suggested that we could bleed out evil tumors to cure illness, or similarly that the sun revolved around the earth. Would you also have labeled such conjecture as alarmist, knee-jerk hand wringing? A lack of data by no means invalidates a hypothesis, on the contrary it encourages further investigation, which leads to scientific progress.
I found it ironic that you referenced Socrates as an example of such alarmist claims, because the way I interpreted what he said is that without truly understanding a thing, and instead relying upon rote recitation, one could only continue to repeat the conclusions that others had painstakingly worked toward, but they would not be able to explain why or how, only that it is.
You need look no further than the embarrassing circus that was "fact checking" during recent political cycles to see that the concerns Socrates held have in fact come to fruition. Without any evidence except for a tweet hordes of people would confidently and voraciously defend a stance they knew nothing about, often times being completely wrong.
SOCRATES: "The same is true of written words. You’d think they were speaking as if they had some understanding, but if you question anything that has been said because you want to learn more, it continues to signify just that very same thing forever. When it has once been written down, every discourse roams about everywhere, reaching indiscriminately those with understanding no less than those who have no business with it, and it doesn’t know to whom it should speak and to whom it should not."
> just like when people thought novels were bad for people, or when writing would ruin our memories
In fairness, there are also lots of counterexamples where the long-term consequences of a new technology were not adequately foreseen, and ended up posing huge problems in the long run. So I don't think it's unreasonable for people to look skeptically on these sorts of tools.
Agree, but in general those unforeseen consequences differed from the moral panic at the time. Saying “this will have consequences and we should, IDK, know that” is different from prophesying specific dooms without evidence.
By definition, there's never any data to judge innovations. Obviously some opinions turns out to be wrong, but if everybody just waits for data before doing anything, nothing will ever happen. So if you don't like what they are saying, at least try to provide a counterargument.
The difference is that one side has put actual thought into their argument, while the other side is attempting to invalidate the argument based on the fact that people other than themselves have not put in the work necessary to make a claim one way or the other.
Scientific progress doesn't happen by referencing the vast stores of data that we haven't collected.
That is a pretty common logical fallacy. It is actually a well-eatablished fact that "absence of evidence is not evidence of absence", i.e., maybe the reason why there is no evidence is that nobody looked for it, or, as is the case here, because something is too new to find any evidence in either direction
I think you’re mistaking the fallacy. That fallacy is best applied when someone asserts certainty in absence, as in the case of God.
If someone just asserts some random thing, it’s reasonable to ask for evidence. While their lack of evidence does not disprove their point, it’s certainly an argument for being skeptical about their point.
Otherwise I get to assert that every atom is a superintelligence, and your inability to provide contrary evidence is a point in my favor.
as long as these systems are not let near certain industries like aviation software, health stuff i dont mind however people write their software. maybe at this point it will only improve the js frontend mess :)
Not gonna lie, this is kind of the answer I was aiming for when I wrote this. Never did I say that AI is bad or shouldn't be used ever. I just think that it's value must be understood fully, whether you're using it for coding or just need to rephrase 2 sentences for an essay.
At some point, I realized most developers don't need to be very good. The ones with talent will get good, the mediocre ones can use crutches, which will probably make them better. We can all get on with our lives.
I find it at least mildly amusing that it seems like we spend a considerable amount of time trying to make things easier for ourselves as developers, but if you succeed too hard, people begin to get nervous or angry. Kinda like the negative reactions you'll see to no-code tools.
Some of it must be rational in some way, but I have to believe a lot of it is insecurity. To be fair, if AI could do what I do better than me, I'd be bummed out because I enjoy doing it and also I'd probably be out of a job. On the other hand, if it's physically possible, then it's inevitable.
There are people who once worried about autocomplete ruining programmers. I'll give you that I don't enjoy programming without intellisense, but it's not because I can't.
Let's hope chatgpt&co will not end up like no-code which overpromised for decades and didn't deliver. But surely I'm only having a negative outburst here. ;)
To be optimistic in the medium term, the spread of AI code generators among novices will put pressure on languages and tooling to become even more idiot proof. That's not a bad thing!
You AI assistant may not be able to get everything right within its training data and resource allocation, and you might be too inexperienced to know what it did wrong, but languages and toolchains can adapt to better spot/describe/resolve the characteristic errors of using them. It starts an arms race that ends up with better compilers, linters, debuggers, analyzers, "debugging assistants", etc.
That change ends up being a positive one for everybody, but will take long enough that we will absolutely see plenty of new ugly messes in the meantime.
I'm not so sure about that. Idiot-proofing (while being impossible -- ingenious idiots will always find a way around it) comes with tradeoffs that can easily reduce software quality.
I think their is a fair amount of Javascript and before that PHP floating around indicates we should not be making it easier for idiots to write software.
I think there's a secret third thing, domain specific fundamentals/boilerplate.
I've had good results with code generation for minimal proof-of concept things, like 'show me code for [maze construction algorithm]' or 'visualize this platonic solid in mathplotlib, with numeric labels on every vertex' or 'fetch data from https://hacker-news.firebaseio.com/v0/item/34741898.json and build a pandas dataframe from it'.
I've come to prefer it to visiting Stack Overflow when I need to get up to speed on a new thing, so that I don't have to read people's petty ego trips or irrelevant answers. I also find it responsive when I give carefully detailed prompts to develop an algorithm, as it saves me a lot of typing mistakes and syntactical screwups, eg mixing up dimensions in 3d or flipping [x:] and [:x]. Debugging sometimes feels faster than on my own code, because I didn't mentally commit to the error and create a blind spot for myself.
It's much easier to get good results by specifying incremental (across multiple responses) or stepwise instructions in pseudocode, rather than big bang problem statements. Although the latter occasionally kick out magically-working complete programs, they're much more likely to yield catapults: https://www.youtube.com/watch?v=5aCgSwmm5Ho
> I've come to prefer it to visiting Stack Overflow when I need to get up to speed on a new thing,
I'm certainly guilty of this very same mistake, key word being "mistake." I'd hope for a world where the better way would be to read some good docs which gave an accurate mental model of the thing I want to get up to speed with. You know, organized and structured knowledge.
vs a quick answer to a single point without any context.
I don't regard this as a mistake. Tools and docs don't necessarily flow from the same source, and even good docs can't anticipate every need or blind spot of every reader - if we have perfect custom documentation, then it too would have to be AI generated. If good mental models were sufficient, there would be no ned for experiment and all great programs would flow from perfectly written specifications and never contain any bugs.
I like to go back and forth between the docs and hacking together little things that do one job to maintain a feel for what I'm doing (also in many other contexts besides programming).
This is exactly the same format of argument that convinced me as a learner to use notepad++ for years, which I now believe to be a garbage idea. IDEs provide so much important contextual information - a copilot program to show you different techniques would be helpful in a similar vein.
Caveat: chatGPT is a chat program, that it can produce working code in places is a coincidence, it would not be appropriate for this use
> the same format of argument that convinced me as a learner to use notepad++ for years
Offtopic, but there was a time when that wasn't an entirely specious argument: back when most major IDEs were very heavyweight and bloated, think Eclipse or Netbeans.
They've improved an enormous amount since then, both in their performance and general UX, as well as in the important assist features they provide, to the point that if you're not using those today, you're definitely hampering your productivity.
There's some relevance to the current situation, which is that these AIs are certainly going to improve, probably very rapidly.
> Caveat: chatGPT is a chat program, that it can produce working code in places is a coincidence
> The Codex models are descendants of our GPT-3 models that can understand and generate code. Their training data contains both natural language and billions of lines of public code from GitHub.
...but I don't know how/if those might be integrated into ChatGPT.
We are trying to build a workflow and it goes like this:
- we sketch the general idea for the article in Notion
- we use ChatGPT to expand on the topic and explore contrarian perspectives to validate the idea
- I do the final edit and use Grammarly and Quilbot to ensure clarity and fluency
- result goes onto our 400k subs newsletter (from a 1mil list of our users who opened any of our newsletters at least once)
- feedback is used to improve on the idea which goes on our blog
From the content creator's perspective, AI tools are of immense help. Finally, we as a small team can allow ourselves to find time and create content instead of being stuck in a constant development loop.
Let me give you some direct feedback on this then - it reads as 'content' and not an article. ChatGPT seems to blow out word count in a fluent way but the resultant text seems to have much lower information density, the sort of thing you might expect from a student learning to write paragraphs. Genuinely I asked if it was AI generated because it felt like filler.
I've been trying Github copilot. It's especially good for suggestions based on localized context. For example, if you have this:
def do_thing():
print('doing a thing')
def do_thing_2():
print('doing a thing 2')
def do_thing_3():
print('doing a thing 3')
(Which, btw, do_thing 2 & 3 were autogenerated). If you type:
def do_all_the_things():
it will auto generate:
do_thing()
do_thing_2()
do_thing_3()
which is great if you have localized patterns in your code. It has also perfectly generated tests for things like a wrapper around a publicly known api.
Where it starts to get a little suspect is in some of its inferences where it doesn't have the localized context or a well known api as its corpus. It can spit out disastrously incorrect code on occasion. Still a time saver, but you have to remain alert.
FUD. I used ChatGPT yesterday for some bash scripting. I find ChatGPT useful in situations where "I know what I don't know" e.g., I can never quite remember the array syntax in bash, I forget what flags I want for grep or curl to do the task at hand. Sure, I can read the man page (and I am a big fan of reading the man page), but the AI assistant gets me to my destination much faster after I interactively iterate to arrive at what I want. Yes, I have learned the hard way that you should not trust ChatGPT too much and you should view its output with a lot of skepticism. But overall it is a huge advancement in developer productivity, and can serve as a learning tool as well.
ChatGPT and Github CoPilot are best viewed as tools with limitations, but there are ways to check their output. How about asking ChatGPT to look for flaws in a snippet of Github CoPilot-generated code, or using the auto-comment function on CoPilot to do the same with some ChatGPT-generated code?
This is a decent way to check machine learning systems. I haven't tried it in a while, but if you take a text in English and run it through about five different languages sequentially on Google Translate, then go back to English, the result can be interesting.
The author's point might apply to people who are learning a computer language for the very first time. You have to learn the low-level intrinsic complexity by hand, it can't just be dropped into your head in a big chunk. To paraphrase Euclid, "There is no royal road to geometry, or to programming."
Where these tools are pretty great is when having to write something in a new language. For example, if you've never used Javascript but are familiar with C, you might ask, "what's the JS equiavalent of a C struct", or if with Python, "how do I constuct the JS struct equivalent, with its fields being the keys from this Python dict" and then "how do I populate the fields correctly using this json object as the source" and so on.
However, this approach could get one into trouble, if you didn't know what object lifetime was, or what scope was all about, or the difference between a shallow copy and a deep copy, why having many references to a single object might result in data corruption, etc, etc.
Also, often the output or recommendation is just wrong, or very poor code style, so it needs further checking with search engines, usually something like "known issues with library x" etc. That's why I'm not really convinced paying for them, especially on a limited budget, is worth it, at least not quite yet.
Incidentally, I really dislike the subscription model all these online services promote, with no option for just purchasing a service for a given block of time and not having to manually halt the payment.
Idle comment: a largely-unexamined subtext in almost every critique is that you can substitute "the average person" for "the AI" and the critique is exactly as valid, to the extent that it is.
Where "average" means, imperfect, and somewhere shy of domain expert.
Easy example: "Freelancers paid below prevailing market rate: Convenient crutch or recipe for disaster"
AI bots based on LLM are living mirrors, reflecting the status quo of humanity, and the average human is pretty average, and many are below average and a lot of things.
New developers should avoid crutches as much as possible to begin with. Dependency on such things are, themselves, a recipe for disaster (or at least poor programs), AI or not.
personally I feel they would turn out to be useful for new developer but its likely new devs will suffer 'developmental disability' (excuse the pun) after using them for significant period. This very akin to our (Gen Xer's) parents telling us not to rely on calculator for math problems. I know there is a stack overflow rebuttal to that and chatGPT could be very valuable there, especially if it avoids sifting through manpages to discover basic functionality.
I think one area AI assistants may be very useful is to do standard code quality and advanced linter style code review. I cant remember how many times I have seen engineers give 'cosmetic' CR feedback and completely miss big structural issues. If this makes everybody focus on deeper issues then I am all for it.
I work in embedded software and often I see the code quality coming out of SoC vendors is just full of bad practices. If AI can be a good code acceptance tool especially for last minute fixes we should jump on it.
"How can we reasonably verify the output and know it is correct? At least when using AI to create code, we can easily test the code to see if it performs according to the desired behavior. However, that is no easy task for most general information. It is unclear at this point how difficult it will be to improve AI in this regard."
This is really one of the great things about Twitter, etc.
Follow a few people who are proven (by their predictive accuracy) to be experts in their fields.
Their comments on the quality of claims in their field of expertise is much more valuable than the reports being made by "experts" presented on corporate media, or by LLMs.
The general populace will use the crippled AIs presented by mainstream corporations and governments.
Anyone who cares about their information will use un-crippled AIs operating without guardrails, which are always available to monitor what they read and hear, and comment on it.
Sort of like when my relatives call me to analyze a potential Phishing email -- I don't do anything difficult; view the raw mail; check the DKIM signing and SPF validity, look at the full URLs of originating servers vs. the claimed From: and the document contents' claimed source, verify all transited MTAs, check the contents for malicious looking payloads, etc.
Any half-decently trained LLM can do all of this, instantly and trivially, and lay out the evidence in an understandable fashion for a non-technical user.
Imagine that, but amplified by an LLM trained to recognize and categorize arguments in a variety of technical fields. Not to tell you what to believe -- but to categorize the claims by its heritage vs. the main groupings of theories in that field, and then give a set of links to highly rated "for" and "against" arguments.
I don't need some LLM trained by a partisan, with guard-rails set up by some corporation's legal and HR departments. I don't think most people will stand for that, when we have laptops that can already run reasonably capable LLMs w/ training on focussed datasets!
I hope that is the case, but I am doubtful of the overall success. My reasoning being looking at the lack of success of decentralized competition of big tech for search and social media. Other than crypto currency, these other web3 attempts just haven't been largely successful.
Mostly due to the fact the most people will opt for features and convenience over privacy and liberty. The people using Brave, Minds etc will likely use non gated AI, but I suspect the most everyone else will not. Hope I'm wrong.
Reminds me of the arguments against 4GLs, Low/No Code, IDEs, Code Coloring, Intellisense, automatic refactoring, and auto-completion.
I used to think you had to be able to work in C to be a "real programmer" and have learned that a lot of good applications have been created by programmers not using C and some that have never used C. Before C it was assembly, and machine code before that.
AI is a tool that all programmers need to learn how to use correctly.
how can you use a tool potentionally generating hidden bugs into its suggestions "correctly"? and/or legally ambigious output? these are not important for everyone, but they are to me..
Write tests? The skill moves from remembering incantations to being able to define a clear spec in the form of prompts and tests. Basically, programming becomes more declarative.
Now you may be saying "what's the big deal with that? That happens all the time all over the place."
The issue is that the biggest thing AI brings is it allows you to generate lots of code, very fast. So we have more and more code, that is less and less understood by any human being.
Eventually no one has any idea what the code is doing in the first place, and what ends up happening is the Cult Mechanicus of Warhammer 40k we become tech priests running around performing arbitrary and inane rituals to appease "the machine spirit" because no one anywhere has any idea how the dang program actually works.
Like we've all heard horror stories of the Oracle DB code base, imagine if that started to become common. I mean people talk about how AI helps them generate code faster, but to be honest I find most issues are the result of too much code, rather than too little.
My point is I grow concerned about the ability of AI to generate code and the effect that is going to have on increasing the amount of code while reducing understanding.