[dataviz1000]

Several years ago I thought a good litmus test for mastery of coding is not finding a solution using internet search nor getting well written questions about esoteric coding problems answered on StackOverflow. For a while, I would post a question and answer my own question after I solved the problem for posterity (or AI bots). I always loved getting the "I've been working on this for 3 days and you saved my life" comments.

I've been working on a challenging problem all this week and all the AI copilot models are worthless helping me. Mastery in coding is being alone when nobody else nor AI copilots can help you and you have dig deep into generalization, synthesis, and creativity.

(I thought to myself, at least it will be a little while longer before I'm replaced with AI coding agents.)

[epolanski]

Your post misses the fact that 99% of programming is repetitive plumbing and that the overwhelming majority of developers, even ivy league graduates, suck at coding and problem solving.

Thus, AI is a great productivity tool if you know how to use it for the overwhelming majority of problems out there. And it's a boost even for those that are not even good at the craft as well.

This whole narrative of "okay but it can't replace me in this or that situation" is honestly between an obvious touche (why would you think AI would replace rather than empower those who know their craft) and stale luddism.

[DanHulton]

> 99% of programming is repetitive plumbing

Even IF that were true (and I'd argue that it is NOT, and it's people who believe that and act that way who produce the tangled messes of spiderweb code that are utterly opaque to public searches and AI analysis -- the supposed "1%"), if even as low as 1% of the code I interacted with was the kind of code that required really deep thought and analysis, it could easily balloon to take up as much time as the other "99%".

Oh, and Ned Ludd was right, by the way. Weavers WERE replaced by the powered loom. It is in the interest of capital to replace you if they are able to, not to complement you, and furthermore, the teeth of capital have gotten sharper over time, and its appetite more voracious.

[sitkack]

Capital is also willing to have vastly lower quality and burden the remaining labor with more toil in exchange for even lower costs. Velocity will rise, quality will fall, toil will increase leading to more burnout but there will be more expendable bodies to cycle through the slop cleanup farm.

[thunky]

> Even IF that were true (and I'd argue that it is NOT)

Can you share what these "hard problems" are that > 1% of developers are working on?

[durumu]

Even if most of the code you write is solving repetitive plumbing tasks, today's models are incredibly bad at API design taste. IMO designing software in a way that minimizes side effects and is easy to change and test is more than 1% of software engineering.

Lately most of the code I write has been through LLMs and I find them an enormous productivity booster overall, but despite the benchmarks they're not expert human level quite yet, and they need a LOT of coaxing to produce production quality code.

As far as things LLMs are bad at, I think it's mainly the long tail. I'm not sure there's one singular thing that >1% of programmers work on that LLMs suck at, but I think there are thousands of different weird sub-specialties that almost no one is working on and very little public code exists for, thus LLMs are not good at them yet.

[h4ch1]

Try using any AI tool to write a working realtime GI (global illumination) implementation. I've been working on a novel implementation for 60fps/1080p GI and everytime I use Copilot or Claude to even try fixing a minor bug/troubleshoot it nukes entire functions and rewrites them using garbled shader code, old syntax/methods.

Puts things into stark perspective for me.

PS. no amount of prompt engineering will save you in this endeavour.

[thunky]

> I've been working on a novel implementation for 60fps/1080p GI

I don't think this is the type of work a "normal" developer is working on though, which is what my question was targeting.

I would think 99%+ are going to use an existing implementation like unreal engine, which AI can certainly help with.

[uludag]

I've started to come to the conclusion that only greenfield projects consist of repetitive plumbing. Legacy software is like plumbing if all the pipes were tied into a knot. The edge cases, ambiguous naming, hacky solutions, etc. all make for a miserable experience, both for humans and AIs.

[benreesman]

They're remarkably useless on stuff they've seen but not had up-weighted in the training set. Even the best ones (Opus 4 running hot, Qwen and K2 will surprise you fairly often) are a net liability in some obscure thing.

Probably the starkest example of this is build system stuff: it's really obvious which ones have seen a bunch of `nixpkgs`, and even the best ones seem to really struggle with Bazel and sometimes CMake!

The absolute prestige high-end ones running flat out burning 100+ dollars a day and it's a lift on pre-SEO Google/SO I think... but it's not like a blowout vs. a working search index. Back when all the source, all the docs, and all the troubleshooting for any topic on the whole Internet were all above the fold on Google? It was kinda like this: type a question in the magic box and working-ish code pops out. Same at a glory-days FAANG with the internal mega-grep.

I think there's a whole cohort or two who think that "type in the magic box and code comes out" is new. It's not new, we just didn't have it for 5-10 years.

[layla5alive]

Yeah, I mean the interpolation part is new, but boy do I miss pre-enshitification Google!

[burnte]

I have similar issues with support form companies that heavily push AI and self-serve models and make human support hard. I'm very accomplished and highly capable. If I feel the need to turn to support, the chances the solution is in a KB is very slim, same with AI. It'll be a very specific situation with a very specific need.

[Melatonic]

There are a lot of internal KB's companies keep to themselves in their ticketing systems - would be interesting to estimate how much good data there is in there that could in the future be used to train more advanced (or maybe more niche or specific) AI models.

[burnte]

They COULD be great AI chatbots with good data, but in general what is being deployed is just crap. The cheapest app they can get set up as fast as possible so they can check a box, there's no real concern to make it good. I had a talk with my last CEO in January about a similar project, I gave a plan on how to do it right. They instead tried to half ass it and it completely failed. But they don't care because the board member that wanted it is happy because it exists.

[Melatonic]

This has been my thought for a long time - unless there is some breakthrough in AI algo I feel like we are going to hit a "creativity wall" for coding (and some other tasks).

[red75prime]

Any reason to think that the wall will be under the human level?

[hirako2000]

Off the thousands of responses I have read from the top LLMs in the last couple of years: never seen one that was creative. Throwing writing, coding, problem solving, mathematical questions and what not.

It's somewhat easier to perceive the creativeless aspect with stable diffusion. I'm not talking about the missing limb or extra finger glitches. With a bit of experience looking through generated images our brain eventually perceives the absolute lack of creativity, an artist probably spot it without prior experience with generative AI pieces. With LLMs it takes a bit longer.

Anecdotal, baseless I guess. Papers were published, some researchers in the fields of science couldn't get the best LLMs to solve any unsolved problem. I recently came across a paper stating bluntly that all LLMs tested were unable to conceptualize, nor derive laws that generalize whatsoever. E.g formulas.

We are being duped, it doesn't help selling $200 monthly subscriptions - soon for even more - if marketers admitted there is absolutely zero reasoning going on with these stochastic machines on steroids.

I deeply wish the circus ends soon, so that we can start focusing on what LLMs are excellent, well fitted to do better, faster than humans.

Creative it is not.

[sitkack]

It doesn't need to be creative, it needs to brute force boilerplate.

[red75prime]

An opinion on the current state of the field. The usual stochastic parrot mention. That, I see. Reasons for the existence of the wall? Not so much.

[hirako2000]

That it is usual doesn't mean it's false.

Everyone I talked to, knowledgeable in machine learning and/or deep learning, who had no reason to pretend of course, agreed an LLM is a stochastic machines. That it is couples with very good other NLP techniques doesn't change that.

It is why even the best models today miss the shot by a large margin, then hit a decent match. Again back to the creative issue. If it was done before, a good input and a well trained model on good data will output good data likely to match with the best (matching) answer ever produced. Some NLP to make it sound unique is not equal to creativity.

[zyngaro]

Curious to know what are those challenging programming problems are. Can you share some examples?

[dataviz1000]

Currently I'm porting the Playwright / Puppeteer client API to run in a Chrome extension without using the Chrome DevTools Protocol (CDP). Since there is 100s of Chrome extension AI copilots using Playwright with CDP with all the problems that come with that, I believe my library can be very useful. The vscode copilot chat with any model is always trying to evaluate strings in the content script using chrome.scripting.executeScript with eval('') and new Function('') which violates the CSP policy in MV3. The use case is novel but calling executeScript is common and this policy has been enforced for the last 2.5 years and available for a couple before that. Worse is it will convert my valid function definitions to eval() and new Function('') even though it has nothing to do with prompt.