Sounds like exponential growth of crappy software. I'm not saying that before we didn't have mass produced crap in SE, but now it will turn into explosive overflow.
We are living in a ZIRP-like era where builders at the fastest pace layer have misattributed their velocity to exponential gains in model capability. In fact, they are surfing on decades of careful effort to build a robust foundation of highly reusable software libraries.
This strategy will seem to work really well until the economy that enabled that foundation to form is hollowed out. Then, there will be a reckoning (but we will have no choice but to march forth from there).
It's not just software libraries. Specs, applications (the browser!), expectations, device integrations, operating systems, etc. So much that starting from scratch seems impossible.
I'm not agreeing or disagreeing with you, but my brain cannot comprehend how machines can advance such interconnected systems while keeping humans in focus.
Perhaps I shouldn't have watched the Animatrix again.
> This strategy will seem to work really well until the economy that enabled that foundation to form is hollowed out. Then, there will be a reckoning (but we will have no choice but to march forth from there).
There will only be a reckoning if models don't get much better.
If they do get much better you can just have them refactor, fix bugs in, or replace the existing codebase.
The concept of tech debt is sort of meaningless if you anticipate intelligence gains in models to continue.
Its already speeding up human decision processes, and while ethics / alignment may seem unique to humans we also see normative expressions in monkeys or apes (like the experiment where one is given a grapes, the other cucumber).
A lot of ethics is based on symmetry: symmetric relations, equal rights, equal voting power, ... symmetries sound rather mathematical if you ask me, and decision structures have historically been pressed towards democracy (or at least depiction of it). One could say that modeling humanity as an empire with a king, ignores the will of sometimes hungry farmers with pitchforks. To prevent the occasional "implicit democracy" (royaltycide), it turned out in the interest of the king to recognize the powers of those farmers, and to formalize it in the decision making process. Or at least pretend to.
I believe machines will be able predict the preference sentient creatures would prefer in terms of decision structures, but I don't believe it will be able to predict (without human exposition) those novel preferences that stem not from sentience but from being specifically human properties (i.e. irritants which are quasi universal for humans, etc.), some of them humans know how to make predictions for (we can run expensive simulations modeling what happens when protein X is exposed to substance Y, and then make heuristic predictions of the effect on a full human in a realistic environment). So at a fundamental level I agree: machine learning models are not guaranteed to help much in predictions concerning entirely unexplored territory, neither by humans nor by natural selection. But it will definitely be capable of replacing the average human job, which doesn't involve consensual exploration outside of the homeostasis required in the implicit job description, that seems entirely automatable, regardless if its physics, mathematics, (harder than computer science), let alone programming.
It won't be able to magically systematically correctly predict out of distribution datapoints, it could only explore it like humans could by trial and error.
I am more and more inclined into not believing this crappy software theory.
Especially as teams invest in proper agentic harnessing.
We have had a champion in our team that has invested a lot of time into it over the last 4 months, and if anything, quality has improved, not decreased. Architecture is more coherent, codebase has been cleaned up, agents find information quickly, code produced is very solid and my role is more and more checking that the output meets the requirements. But I cannot confidently say that I would've done a better job than AI more often than not I have to admit it does a better job than mine.
The mistakes are less and less technical and merely in the domain mapping. And AI is still not creative as I am for finding solutions quickly to unlock stakeholders' issues. Also, AI is still not creative as I am for finding the proper solutions for advanced technical problems. But it does a better job than me, even on that front, one shotting few solutions in a fraction of a time it would've taken me to test one idea myself.
Mind you, I don't like AI and I think it ruined the job, I don't like working this way, it's exhausting, way more work on one side, way less fun and fiddling with technical parts.
And yet, I have the genuine belief that few years from now we'll be cloning open source repositories that are already optimized/harnessed and tested for agentic loops and best practices left and right with software engineers mostly overseeing the domain translation and putting their 2 cents on the non-boilerplatey parts of the product (which, in general, are a small part of the surface).
I think that the next years of my career will be mostly spent in setting up and writing the harnessing and domain mapping part. Then I will move to another sector, not because I necessarily believe I won't have a job, but because I want to vomit thinking that's going to be my job.
"Watching John with the machine, it was suddenly so clear. The terminator would never stop. It would never leave him, and it would never hurt him, never shout at him, or get drunk and hit him, or say it was too busy to spend time with him. It would always be there. And it would die to protect him. Of all the would-be fathers who came and went over the years, this thing, this machine, was the only one who measured up. In an insane world, it was the sanest choice."
As long as you've indicated what you want, the machine will try to do what you ask of it. It won't get tired because "the codebase is too big", or it has gotten bored of the pattern, or it wants to introduce a new technology.
It just does the thing you asked of it. (note, that yes, I get that as a codebase size increases, it might make it more difficult to fit into context, but that only applies if it needs to read a large percentage of the project to implement the task, which shouldn't be the case.
there are good actors, which are empowered by AI to produce positive impact, but often there are N times more bad actors, which push crappy code to close feature requests fast, increase performance LoC-like metrics, etc.
I still can't tell from the outside whether it sounds like a great time to be in security because of the vulnerable slop being churned out, or a terrible time because the people paying to make it don't care.
"exponential growth of crappy X" applies to every industry that went from being an artisanal craft to being mass produced with little or no human input. and we live much better lives than we did before the industrial revolution.
What counts as “done” has a time component, so I think we’re going to see more of a spectrum where some businesses try to skimp as much as their market will allow but others will recognize that racking up technical debt is a long-term loss. Stuff like brochure sites will certainly be cut down but anything where there’s liability or long-term customer relationship is going to need to factor in quality as well.
You could say the same when higher level languages getting popular.
Previously programming was the domain of Math, Physics, EE doctorates. These days we even have a few months coding bootcamp
This strategy will seem to work really well until the economy that enabled that foundation to form is hollowed out. Then, there will be a reckoning (but we will have no choice but to march forth from there).