[jimbo808]

They're actively trying to use lobbying power to make open weight models illegal. So I'm just not going to use their services at all anymore. I don't think they're a net gain if you're a skilled senior, and the hidden cost in terms of technical debt and skill atrophy is just being swept under the rug. I'll be okay without their bullshit generator.

[pmarreck]

> I don't think they're a net gain if you're a skilled senior

I'm a skilled senior (I'm 54 and been coding since I was about 8; I've been 100% AI-generated code for at least 6 months now and have produced a combination of speed and quality that has astonished me; my velocity is apparent at https://github.com/pmarreck/) and this has been a massive net gain, so your claim is now officially in sheer defiance of reality.

In a skilled senior's hands, this is like an expert power tool. In the hands of someone less-skilled, it is likely also... less-skilled. It's a magnifier.

> and the hidden cost in terms of technical debt and skill atrophy is just being swept under the rug.

Nope, no it's not. It's being reviewed, measured, and controlled against. Because... you WILL need more controls to take full advantage. Look, I even invented a whole new control methodology around it called MFIC: https://gist.github.com/pmarreck/b30aa3ca69cb70a5526f8a63ab8...

[jimbo808]

I think I might have written a comment similar to yours maybe 6 months or a year ago. I'm not quite sure to respond to these sorts of replies. I have used LLMs/Claude Code quite extensively professionally and was a very early adopter, have built tooling around LLM/agentic development, and genuinely embraced it. They aren't useless, but the short term gains you think you're getting come at a very steep price that you may not actually account for consciously for quite some time, if ever.

[illiac786]

I think the uncomfortable debate is not about skill atrophy as a general phenomenon (it’s happening anyway, doesn’t matter how much we debate it) but rather, _which_ skills are atrophying and if these skills are now superfluous/worthless or not.

If you don’t use a skill, it’s like a gene a species doesn’t need anymore, it will atrophy.

Is that bad and if yes, why? Skill atrophy is not intrinsically bad. I don’t know how to make tinted glas for church windows and I will never learn it because there are machines doing it now.

But I would for example think that critical thinking would be a catastrophic skill atrophy. As far as I know, there is no proven link though (and one would have to define what is “critical thinking” in the first place). Writing assembler without any autocomplete, I’m not so sure it’s such a problematic skill atrophy.

[simondotau]

One could argue that the cumulative atrophy of skill around writing CPU assembly has been problematic in some respects, but it’s also completely unreasonable to lament what we’ve gained in return.

As far as I’m concerned, so long as we can be happy with AI we can run locally, AI is no different to the rise of scripting languages or the pocket calculator. It’s only problematic if the calculator is rented to you as a service.

[pishpash]

Hence only let your skill atrophy to the extent where if all you had were your local laptop you can still be competent. Relying on paid subscription services for your skill is a fool's errand.

[techpression]

It’s not one single skill being lost, it’s about many and how they interact.

I just did a big refactor with opus, it went ok, some bugs. The normal stuff. One of the bugs was in a part of the code no longer needed, which Opus had just filled with comments more or less. Asking it fix the bug worked, but then I really looked at the code and realized just that, this is pointless now.

I’ve only been coding for 20+ years so I might be more susceptible than the author, but I’m quite terrified about losing skills in writing code, but also designing good structure, coherency and system overview. These are the things people claim you need more of with LLMs, but is what you outsource the most, even if you think you are describing it in detail.

We are all collectively growing the skill of complacency and laziness though, and those are not great ”skills” to have. And I’m just as guilty as anyone.

[DanielHB]

Since compilers became a thing Assembly language knowledge atrophied[1] across the workforce.

Since automatic memory management became a thing memory management and pointers knowledge atrophied[1] across the workforce (although not nearly to the same degree).

I think the pattern here is that compilers almost always output better machine code than humans, automatic memory management doesn't output better machine code than skilled humans can very (especially with modern languages that give you a lot build-time safety checks).

And even then, there is still demand for assembly knowledge in the workforce, it is just very niche.

I don't think LLMs will ever be good enough to "almost always" output better code than humans. But, like automatic memory management, it will likely make some types of programming more niche.

The key thing here is that compilers are deterministic, deterministic tools have way less variance in output quality. Automatic memory management is not as deterministic as a compiler because it happens at runtime. LLMs output build-time code, but the can be drastically different if I sneeze too hard.

[1]: as in % of the workforce, not absolute numbers. Hard to get exact figures on this, but I think we have more experienced people actively using Assembly today than we had before compilers became the default (late 80s). We probably have more active C/C++ programmers today than before Java became popular (early 2000s).

[vidarh]

I might have written a comment similar to yours maybe a year ago.

Yes, some skills will atrophy, but the learning curve for LLMs is also steep and you will acquire new skills that will pay off the costs many times over.

We see this in discussions like these where you have people running the gamut from using them as glorified auto-complete or babysitting them (usually a net loss in speed, though it'll feel less draining) via people running multiple agents in several different tabs (a gain) to people prompting for harnesses rather than tasks, and putting the agent in the resulting harness (where the multipliers come in) and even people at the peak of experience with them today are only scratching the surface.

I'm very aware that just as my assembler skills are not what they once were, my skills in the languages I'm now writing less will not be what they once were a few years down this line.

But I produced far more before I started using LLMs through the force multiplier of modern languages and frameworks than I did in assembler in the 80's and 90's, and I produce far more now with LLMs, and I will produce even more in the future by learning how to take advantage of new capabilities.

I have Claude refining a system that wasn't tractable a few years ago in another terminal as I'm writing this. I don't care if it would take me a bit longer to get back up to speed on a C codebase again if I was stripped of all access to LLMs any more than I care if it'd take me a bit longer to get up to speed on programming assembler on a Commodore 64.

[muse900]

I get what you are saying, but how can we be talking about skill atrophy when our main skill is changing from being able to produce code ourselves to being able to leverage LLMs to write that code.

At the end of the day there are goals achieved with coding. Coding is a tool to reach either your business needs or some personal aspiration.

When it comes to businesses I don't think a business cares if you used the best stack possible, or you've written it in assembly, as long as it works. Judging from the biggest coding drivers out there, most of the code produced globally and the biggest apps out there have had skilled engineers writing code but its not always perfect. As long as it works. Lets not forget that the web is build in php and js.

So again my argument is that, are you atrophying a skill that is going to exist in the next 1 to 2 years, or is everything going to shift towards LLM code writting.

Personally I think that LLM code writing is the winner, whether we like it or not, it accelerates business objectives, which at the end of the day its what is the deciding factor.

And yes I do miss the days I was writing code and I was solving complex problems myself.

[darkwater]

> At the end of the day there are goals achieved with coding. Coding is a tool to reach either your business needs or some personal aspiration.

This is your opinion and I even share it, but there are many people here for whom writing the code was/is the whole deal. You would not have languages and heck - even editors! - holy wars otherwise.

[muse900]

Yes for sure there are people that writing code is the whole deal.

How many of them are driving goals and businesses though?

Like if we take for example an ex coder like Gabe, someone from his team comes up and says to him "we can launch this game in 1 year by using LLMs, codebase will be meh/okish" or "we can launch this game in 4 years we'll hire the top engineers and write the best software piece ever, a technical novelty".

I already know his answer... even if his answer was the 4 years which won't be, his board would disagree.

So yes there are people that love and enjoy writing software, but the truth is that business is leading software, not the other way around.

For me personally nowadays I don't need languages, I am so deep into coding using LLM's, not vibing, that I don't really mind on what is written. Also we are just at the start of this thing.

[DanielHB]

I am consistently underwhelmed by the output, I can't really explain it besides LLMs have no taste about what is easier to read for humans.

I usually start a task with an LLM and then do small refactors using the LLM and then do some manual refactors before I am done. But often for more complex tasks the manual refactors are quite large.

Maybe it is because they can read walls of text so easily, so they output walls of text that are hard to read for humans.

I feel quite sad because a lot of my fellow colleagues are not putting this extra effort in to make things easier to understand by humans. PR review is basically me just doing this extra effort for them and their LLM implementing my comments.

And that is when I can even pinpoint the bad taste in the code structure, sometimes it is not something you can easily describe in a PR comment besides "no human would structure the code like this".

[socalgal2]

It’s hard to know if your experience is relevant from 6 months to year ago. The models are getting better every couple of months. My current experience is mostly like the other senior above. For me the last 4 months I’ve gone from mostly writing code by hand to writing almost no code by hand. I guide the LLM and it’s a force multiplier. I review its code and discuss with it how to test and what needs changing and it does it. I point out things it didn’t handle and it handles them.

[jimbo808]

Uh, I didn't say I stopped using LLM's 6mos to a year ago. I have to for my job, it was just that long ago that I began to understand that they aren't what they seem and definitely aren't a "force multiplier," more like a "debt generator."

[aix1]

> a very steep price that you may not actually account for

Could you elaborate on this steep price that you have in mind? What does it consist of?

[jimbo808]

Technical debt and skill atrophy

Technical debt due to accumulated excessively verbose, badly architected, often redundant, feature-bloated code which always looks good, even upon earnest review, but actually sucks and becomes extremely difficult to maintain in ways which are not obvious in code review. The issue is this: your tooling can help, and can make you feel better, and you might think you wrote all the prompts and made all the tools to mitigate these issues, but you haven't. If you're not consistently seeing it generate code that is very very close to the way a skilled senior dev such as yourself would have done it (with similar line count, etc), that is a red flag even if the code looks great and works.

[mingusrude]

> ...badly architected, often redundant, feature-bloated code which always looks good, even upon earnest review, but actually sucks and becomes extremely difficult to maintain in ways which are not obvious in code review.

I can only judge from my own experience but with or without LLMs, these are the codebases that I have worked with during most of my career. To me, much of the question is whether LLMs produce worse code than the me and my colleagues have done in the past and I don't think that's the case. It is however very common that people hold LLMs to a higher standard than human colleagues and then it's not a useful comparison.

[torginus]

LLMs can produce bad code in ways no human ever would. I remember working on headscratcher problem of walking a tree while gradually building up state, and since LLMs are supposed to be good at algorithms, I asked Claude.

It came up with a correct LC-hard tier solution that involved dynamic programming, and was essentially an unreadable dense mess that was impossible to reason through as a human.

It worked, but it was so bad, that I sat down and realized after a bit that with maintaining a small cache, and being very particular about how the nodes are traversed, I reduced the solution to like a 10 line modified DFS, that I could understand.

[x3ro]

In my experience, even if the code is not worse, there is so much more of it that people quickly stop understanding what is going on. Now you could argue „the LLM knows“, but that comes at an insane token cost, even if it works well.

This is in the interest of big AI companies: if they quasi-monopolize the skills entire sectors of the economy need in order to function, that will be great (for them).

Everyone keeps comparing this to compilers, but I don’t need a multiple-hundred dollar subscription to use LLVM. And people didn’t stop understanding how computers work either, just because they used C. And yeah, maybe local LLMs will become the norm, and I hope so. But market forces (hardware prices) certainly are working against that right now.

[seer]

Yeah but I think going back to hand writing bespoke code is not coming back, the genie is out of the bottle.

But we could build much better tooling around keeping the agents honest. The problems you are describing are absolutely real and I see them every they.

One friend of mine had almost a mental breakdown when he just went ahead and drilled a bug producing Claude to the point that it itself admitted it was “a piece of shit”. He knew that arguing with an LLM agent is more than useless, but it was cathartic for sure.

When I encounter a situation like this I always go down to - have I done everything I could to catch these errors in my automated validation, and update it as needed.

Agents are also more than happy to spend tokens refactoring, once you have such a test harness be good enough, producing successively better and more general abstractions is quite easy.

The old rule of thumb of “make it work, make it fast, make it pretty” still applies , just with much much faster iteration speed.

It seems with agents people have forgotten the last 2 steps since they produce a _working_ solution, and it might be hard to justify spending time “cleaning it up”, but this still remains essential.

[shevis]

The cost is skill atrophy. When was the last time you wrote something entirely from scratch by hand without AI assistance? It’s a skill entirely separate from prompting and reviewing. And it atrophies when you stop using it.

[aix1]

> The cost is skill atrophy

I hear what you're saying but I'm not sure I buy it in the context of this thread (a response to someone who is 54 and has been coding since they were eight).

I am in a similar boat, having been coding full-time for fourty years. The way I use the current tools is that I own all architectural and design decisions but let Claude Code fill in the blanks. I reckon the quality of the output is about 90% of what it would have been had I done everything myself, but I get a lot more done (easily 3-5X).

Will I forget how to write a "for" loop just because I haven't been writing many of them by hand lately? Those skills are so deeply ingrained that I seriously doubt it. I can ride a bike after a multi-year break, or converse in a language I haven't regularly spoken for several decades. Or write using pen & paper even though I hardly ever do it. I don't see why coding would be any different.

[defrost]

I have a greater concern about societal skill atrophy.

I also am not about to forget how to for(;;), that said, as a result of some years invested in aligning old pre WGS84 mapping with modern GPS and improving digital mapping, there are fewer people per capita with the skills to navigate via paper maps in the absence of GPS.

Old farts coding since age 8 (in which I include myself with a decade+ over a sprightly young 54) will retain coding skills for as long as they apply them - the fear is that fewer and fewer others will develop and exercise such skills due to AI.

It remains to be seen if that's a bad thing long term.

[mrbungie]

From what I gather from GPs upper post: Technical debt, skill atrophy, delusions of grandeur about one's own abilities / psychosis.

[pmarreck]

> very steep price

I have yet to see it, but OK

Either measure it or it sounds like a conspiracy theory

[Thanemate]

The skilled seniors better stop downplaying what actually led them to be skilled in the first place, and realize that the conditions to develop that skill has been gone and almost deemed unproductive in today's workplace.

Not disagreeing that LLM's are a force multiplier, but I highly doubt whatever value will end up finding multiplying in the next generation of seniors, at this rate. It's surreal to me that I have to point out that recognition AND recalling are both necessary components of skill acquisition, because humans largely knew this since the dawn of education.

[chewz]

I have been thinking about it - paid apprenticeship is the answer - just as it is in other professions like medicine.

Seniors should be paid to actively introduce juniors to the trade over couple of years. No more bootcamp entry.

And it would be significant $ for senior to agree to expend his time and energy on software engineering apprentices. There would be also very limited number of places with good seniors. Exactly like medicine for a long time now.

In fact it is already happening in some companies I know about - seniors geting their bonuses tied to juniors being under their wings.

[sumedh]

> Seniors should be paid to actively introduce juniors

Instead lets train the contractors of an IT sourcing company and then we don't need you.

[einichi]

This is something for educators to deal with, not a 50+ senior IC, but yes your point is extremely important.

[monk_e_boy]

As an educator (UK) it's very hard to find people with experience to join us. The pay is SO crap and the workload is huge. Trying to teach complex, large systems is impossible, because we don't have them. We can't throw 1,000 files and 20,000 lines of code at a student and teach them how the system was engineered. We can show them toy systems.... which is fine if there is a work route that takes them and nutures them for 5 or so years. But without that pipeline, shrug not sure what we do about it

[vincnetas]

i would rephrase "Force multiplier" as "Force power up".

If your "force" is above 1 then its ok to have AI power up your force. 2.3 to the power of 3 is 12.169.

But if you're a beginner and your "force" is bellow 1 so power upping this makes it worse. 0.2 to power of 3 is 0.008

[integricho]

I believe you are miscalculating the effect of skill atrophy, there is benefit and actual experience gained by doing the work yourself. You are an experienced dev and already have a lot of tools and knowledge under your belt so at the moment it is hard to see the actual issue, as this is just a productivity multiplierfor you. But give it a couple of years working under these conditions, your tech savvy nature will be severely diminished.

[alchemism]

In a couple of years he will be 60, too. Then 65, then 70.

Seniors will be able to stay in the game much longer than before, mark my words.

[aix1]

Indeed. I think it's a much bigger issue for juniors, who haven't yet had a chance to build that systems design muscle.

When an LLM is making a bad design decision but the engineer doesn't have the experience to spot it AND the consequences don't become apparent until much later (which is often the case) -- it's kinda hard to learn.

[throwaway27448]

Have you really found claude to much more more capable than eg deepseek? Anthropic has little to no chance of producing a competitive business model in the long term.

[vidarh]

The cheap models are cost-competitive if you are running them in long-running agentive tasks.

But they take a lot longer to reach the same goal for complex tasks, so the difference is still very real, and the cost-savings are still very much a question of how well you manage to characterise the tasks they will do quickly and pick and choose what to use when.

I kind of agree that I think the cheap models will eat away at the moat very effectively, but if it doesn't seem more capable to you, you're not giving it complex enough tasks to see what they can do.

(FWIW, I've burned billions of tokens on each of Deepseek, Kimi, GLM5.2, GPT, Sonnet, Opus, Haiku using the same harness, and we've kept stats on cost per task)

[jeffyaw]

absolutely, for me the tui, ultracode agentic workflows, and streaming logic are far superior. the closest model is minimax 3.0 imo and i ended up adding a custom tui, agentic workflows, streaming logic and implementing skills to that (in typed) in order to get to an acceptable claude fallback. on their own i haven’t found one model comparable to claude, not even chatgpt.

[hhh]

Yeah, using deepseek feels like shit and I spend hours steering deepseek in a direction versus opus-4.7 or 4.8 where I can just kinda let it ball out on some reverse engineering problems.

[n3storm]

I don't. Using claude code, claude.md etc with deepseek v4 is almost undistinguishable.

[boc]

> Anthropic has little to no chance of producing a competitive business model in the long term.

Extraordinary thing to say about the fastest growing company in the history of capitalism. They will soon have access to public markets, essentially unlimited capital, and can build insanely large models that they don't have to make public... ever. They can just use those models to run their business, train better models, eat competitors, etc.

But maybe it's Anthropic that isn't thinking ahead enough - you clearly think you can see around corners with your proclamation. So why do you think they have "little to no chance" of surviving long term?

[throwaway27448]

We both know that both revenue and especially valuation, especially for something with as little moat as software, can vanish overnight. This is a commodity market and anthropic is serving boutique software, and the market gets more crowded every day.

[boc]

So you're arguing it's the Yogi Berra "Nobody goes there anymore, it's too crowded" of business models?

[jurgenburgen]

> Extraordinary thing to say about the fastest growing company in the history of capitalism. They will soon have access to public markets, essentially unlimited capital

There is no such thing as unlimited capital. The faster they grow the faster they burn capital. Eventually it will run out.

[boc]

That's not how economies work. It's not a fixed pie that everyone shares.

[kerabatsos]

Finally, someone said it. 20+ years in software and my productivity and velocity is wild right now.

[philipwhiuk]

> MFIC

Getting another agent to validate the first agent is a tower of sand.

> my velocity is apparent at https://github.com/pmarreck/)

Forgive me, but the active repos all look like reimplementations of existing good open source code (which of course is ideal training data) - rm_safe has rip for example. Or prototypes. Is there anything that actually has a user base > 1?

[byzantinegene]

you're right, but if skills atrophy so will the efficacy of the tool.

[pmarreck]

Skills atrophying in terms of what? Remembering specific API's that you always had to look up anyway? You don't lose developer intuition, analytical thinking or technical inclination, and those are the things that matter, anyway.

I recently did a fleetwide upgrade to Zig 0.16. Do I remember every single change from 0.15? No. Do I have to? Also no. Both because I can look it up if I need to, but also because the LLM already does.

If I don't look at a codebase that I myself haven't looked at in a year, I will not recognize some things when I return to it. Is this sense of "atrophy" meaningful when this was a problem long before LLMs came on the scene?

[wesleywt]

How do you deal with the lack of cognitive engagement? I think it is the primary driver of developing and maintaining skills as well as generating new ideas. Letting an agent do it for you will just get you average but well formatted code. Not something new.

[bfjvibybd6cuvu6]

I haven't cared about cognitive engagement in over a decade. Neither has my CEO.

[kakacik]

You very conveniently avoided skill atrophy, the biggest issue for existing engineers to deal with (not even going into topic of cancelling whole hiring-junior-and-raise-senior approach which is just shortsighted and retarded to be polite, but general greed has overtaken the field so its not shocking this went out of window). Everybody using llms excessively is measurably doing worse, ie students. I have hard time believing out industry is somehow immune to that and refuse to do the experiment on myself if avoidable, which it sort of is for me.

Its like drug that will give you few years of great high, and ruin rest of your life afterwards. Use it by all means, I don't care about your output, nobody here does, you do you.

I do care about my long term skills, which aren't about piping some llm outputs. My employer ain't dumb fuck who is pushing for llms at all costs as much as possible. Anyway, most of my day work are processes, discussions, pushing things through - llms can't do a shit here, its personal conversations, connections, often psychical contact to get things done on time. Startup world would be different but I am as far from that unstable environment as I am from say gaming industry, just not worth my time outside SV area.

So I just use llms to verify my coding results, they are fine for that, but I do the creativity. Its by far the best part of my software dev work, why the heck would I be automating that away? Its like automating sex away so you can have more time... reading HN or some other way to just waste time, dumb approach from all angles.

Of course this changes if one is working on personal projects, self-employed, small startup etc but most folks here are not in that category.

[dimitrios1]

I am another skilled senior, have been coding since I was 7, although you have a few more years of experience on me, and am commenting here just for the goldilocks moment, as I have read and reflected on both of your comments and find my reality is somewhere in the middle.

On personal projects, where I am in charge of all the hats (product development, UI, UX, backend, security, server admin, etc) -- absolutely crazy force multiplier. You get a nice suite of backend and e2e tests running, with full business scenario layered on top of that, and constantly running agents to do the coding, another agent on a higher level of reasoning to review that work, and sometimes occasionally poping into another competitors model to review their work just for added comfort -- it feels like wizardry. I am not vibing it, but I wouldn't say I am carefully scrolling through every line. I review whats fundamentally important, especially when it comes to data, overall structure, and large, cross cutting concerns, but I would be lying if I say some code doesn't land that I don't read. But I have the security of the test suites and validations , so I pour more effort into that.

It's a nice self reinforceing loop.

All of this might sound like I agree with you, and to some extent I do, but I am realizing as the apps I have built out like a cannon shot out of hell with tremendous speed and polish right out of the gate are starting to slow down. Feature adds are getting more complex. My memory is not what it used to be. Each run and pass through the code consumes more of my tokens and limits. I am starting to do less in the same amount of time. Codex did a vertical slice of a feature for me (well defined and well planned). It contained functionality that has historically plagued us developers -- the dreaded time. I used xHigh GPT 5.5. It had obvious bugs, but I wanted the robots to catch it. I popped it in claude (on the new sonnet 5! heyo!) -- Claude caught the bugs. Even said they "immediately stood out" I wondered how this happened. Frontier model from company A was evaluated by workhorse model from company B. All of this again took massive amounts of usage. And time.

And this is -- best case scenario, perfect world, everything is in perfect alignment.

Now for the work reality.

Multiple product and experience owners. Multiple dev teams. Different enterprise teams support services you rely on. You don't have full unfettered access to frontier models. You have to use copilot, or some other enterprise harness, and you run out of credits for the month, you are SOL. It's not as good as your claude, you think to yourself, but hey, its familiar enough, and you have 5k credits left for the month for Opus 4.8, better make the best of it. But now you burned half of them working on that Transactional Bug that was mixing synchronous and asynchronous semantics that the other guy's model should have picked up on. What happened? Maybe he didn't use Opus, maybe he used Haiku, maybe his prompt was bad. Who knows. Gotta fix it. Oh, you gotta reach across the isle and put in a request to get the Enterprise team to look at this caching inconsistency on user data that you need and is really the source of your race conditions. Tick tick tick. Model limits approaching. You start wondering if you just did all this by hand like "in the old days" would you have got it done correctly faster? Or at least, cheaper. You'll never know.

[sisiissjz77]

It’s simultaneously simple and deceptively difficult to coax a growing system into staying sane. It reminds me of forcing a fractal into growth yet somehow letting it remain similar at all scales, manually.

Scaling in this sense is not operational (“servers”), but conceptual (“features”).

I don’t want to be a downer but I find many devs are not great at this. Very clever folks, but they tend to not see these issues clearly. They’ll nod and recognize when you talk about separating content from form and the importance of various design principles like high cohesion and loose coupling but completely disregard them once in contact with reality.

Part of the problem, as you nicely showed, is that technology is only a single slice of this problematic pie. Organizations in general are systems as well and they tend to be either badly architected, badly maintained or often both. Some technological issues are downstream from organizational issues and IME those can be become rather dominant variables in the equation and no amount of AI - save full AGI taking control of the company - is going to save us from those factors.

[Revanche1367]

And you’ll never know because even if you could turn back time and do it from scratch, you’re likely to opt again not to do it all manually because the cognitive load is going to keep tempting you to reach for the agent again.

[stpedgwdgfhgdd]

+1

the distinction between personal projects and Enterprise development is a big one. A severe bug in my personal projects, i fix it on the fly. A bug in our products rolled out, nightmare.

[sensanaty]

We've just done an official evaluation at work, using extensive statistics on our gigantic monorepo in a company with ~2000 devs over the course of 2 years, everyone from hardware engineers to regular old frontend engineers. It's a highly profitable and mature public company, and has been for going on a decade at this point without missing a beat. We were given infinite access & budgets to basically any and all AI tooling we could imagine, and we have several "AI Native" teams (whatever the fuck that even means). We're doing agentic coding, we have harnesses of all kind, skills, we have many teams doing spec-driven development, designers using all the various things like Figma Make and access to tools like Devin/Factory Droid/Claude Code/Codex/etc.

This is all to say, we as a company are using AI a lot in all possible corners, but thankfully our leadership isn't schizophrenic and isn't mandating everyone hit token limits or whatever, it's more of a "Let's see what works and what doesn't" type of thing, and we measure a lot of statistics. Nobody here really cares whether LLMs are the next coming of Christ or not, as a company there are many people (even in SLT) that are indifferent to LLMs, and many who are reasonably hyped.

I wish I could link to the actual document we were all shown since it has a beautiful breakdown of the methodology and a fine-grained breakdown of the stats and the categories measured, but in the grand scheme of things, ALL the AI tooling we have implemented (at least on the engineering side of the equation) has contributed to a total of... drum roll please... 7 (seven) Percent overall productivity increase! The most productive teams saw a productivity increase of around 20%, while some teams actually saw drops in productivity into the negative percentage points. My team, none of us really give a shit about AI and we're somewhere in the 3-5% range on certain categories of tasks, which I'd say is a fairly good assessment.

Productivity here is measured in many ways, including but not limited to speed of MR review and merge times, feature/ticket/roadmap closure/delivery, rollback/revert incidence rate, how often people interact with the MR review bots and implement their suggestions/fixes, how many times people check back on AI transcriptions/meeting notes (hint: Nobody looks back on any of it, it's all just noise that gets generated and never actually referenced outside a few extremely rare cases) and many more things I'm forgetting. It is an imperfect number of course, because measuring productivity in engineering is a sisyphean task, but in my opinion it is accurate to the reality on the ground and outside of all the hype and marketing bullshit.

So, I remain thoroughly unconvinced of these personal anecdotes of people being "massively" more productive, especially once you factor in the fact that we now have a 2000EUR budget/month/dev for all the AI tooling, those productivity numbers start looking pathetic once you factor in the costs (which are only increasing as the AI companies need to start recouping the gazillions they've burned). Some teams have started begging to disable coderabbit and other similar tools in their MRs because they're producing nothing but walls of noise that makes reviewing any MR a nightmare of sludging through endless slop of useless bullshit, ours included.

[QwenGlazer9000]

It took me so long to finally have someone else not hyping up SDD. It's the new thing at work and it's driving me insane.

I'm surprised no one tried fudging numbers though. That's usually what happens IME.

[sensanaty]

> I'm surprised no one tried fudging numbers though.

Oh some of the "AI Native" teams tried in the past, before the large scale statistics came out. The funniest one was the team responsible for rolling out AI to the whole org citing numbers like 500% more productivity, compared to what? Who knows, since that team didn't exist until relatively recently.

It probably helps that we're in a highly regulated field (banking), so by default we don't chase trends, especially not ones that are potential regulatory nightmares :p Things go a bit slower, which has its pros and cons, one of the pros being that you're forced to consider the long term a bit more and you have room to plan things out and conduct reasonable experiments.

[vidarh]

> I don't think they're a net gain if you're a skilled senior

I've had Claude Code running a /loop for the last week driving down complex crashing bugs in a prototype compiler entirely unilaterally. I occasionally glance over.

A few of those crashing test cases were ones I've spent more than a week trying to track down myself. I have 30 years of experience of doing this.

It's worked 24/7.

So far it has fixed over 500 of them.

Will there be technical debt? Yes. But nothing that remotely compares to the cost I'd have incurred of fixing all of those myself.

It is hard to reconcile those gains without thinking that if people are saying these are not a net gain, they haven't really tried learning how to get the full benefit. If you sit and watch a model work and keep intervening all the time, then sure, they're not going to be a net gain.

[vidarh]

Btw, since this morning it's fixed another 200+ crashing test cases.

[kelnos]

Why even bother posting, especially as a reply to a completely unrelated comment? This is just not substantive or useful to the conversation.

(And I say this as someone who agrees with you that it's garbage that these companies are trying to legislate their way into an oligopoly.)

[jimbo808]

> Why even bother posting, especially as a reply to a completely unrelated comment?

[casey2]

If you give Anthropic money they will make your life worse in another aspect, it's relevant to all their models. The best principle is to not give money to people who want to harm you.

Anthropic has gone past fearmongering and well into terrorism. I think people on Hacker News should not recommend working with terrorist orgs.

[brookst]

“Terrorism” is so ludicrous a hyperbole that it completely discredits your position.

[theshrike79]

Should we work with companies that give the "Department of War" full access to their tools (OpenAI) or Chinese companies with completely opaque ownership and dev structures?

Or the largest ad company in the world (Google)?

[lemonteaau]

The irony is that an authoritarian country is leading the world in open models

[pjerem]

How is that ironic knowing that an authoritarian imperialist country is leading the world in closed models ?

[throwaway27448]

Making software illegal is not an easy task.

[anonzzzies]

Sure about Dario (and all billionaire) weirdness, but no gains if you are a skilled senior is well, very far out in our experience (our company is 30 years old with mostly the original employees and founders): what we deliver now at the speed and quality we deliver it would have been impossible 10 years ago with our team size of skilled seniors. We replaced all the commercial products our clients and ourselves used with our own, giving us millions more revenue and profit with the upselling and efficiency benefits. We work for regulated clients: our code is reviewed, pentested and audited regularly by us and 3rd parties so its not slop either. You are definitely leaving money on the table. We do mostly use chinese models on our own hardware (we colocate cages of racks) so this is not about Anthropic but about AI in general.

Skill athrophy is a real thing though; we try to prevent this by have hackethons (for lack of a better word) without AI where I pick something extremely non trivial and we implement it for fun and profit without AI (with would not matter much as they are currently bad at these things); last one was flex paxos for our in house db with obvious metrics for the endresult: data integrity (duh) under failure and performance better or at least the same as our raft production version.

[andyroid]

> We replaced all the commercial products our clients and ourselves used with our own

You’ll never guess what product your clients are looking to replace with their own next.

[anonzzzies]

Sure, that is why you need to be early. I fully believe my company won't make it another 30 years (or 10), so we prepare for that. Also, I will be dead by then, but that is unrelated.

For now everyone is still sufficiently crap at using AI to need help. We had enough clients trying to build something themselves and then come crying to us.

[topato]

Having a health problem that puts an end date on your effort must tint your business choices in a unique and interesting way. I find your ideas intriguing, and wish to subscribe to your newsletter.

[strangegecko]

We all have a health problem that puts an end date on our efforts.

[fragmede]

Not if Bryan Johnson can help it!

[hughw]

Sure but in the intervening 2 years there's money to be made.

[mastazi]

good luck actually enforcing that.

[andsoitis]

> They're actively trying to use lobbying power to make open weight models illegal.

What is your evidence?

[Robdel12]

Dario’s own mouth https://x.com/coinbureau/status/2071330294452666695/mediavie...

[anon373839]

He has also been telling bald-faced lies about open source/open weights models that are easily disproved. For example, he claimed that they lack the collaborative benefits of open source because "we can't see inside the model".

Open weights models are responsible for enabling reams of research on interpretability methods that do just that. And they have facilitated so much collaboration on architecture, inference optimizations, training and steering methods, and other topics that were completely out of reach with closed models like Anthropic's. It's really staggering to me.

[andsoitis]

Doesn’t this seem true / self-evident:

“His warns that once powerful models are released openly, companies lose the ability to monitor misuse, revoke access, or update safety guardrails.”

[anon373839]

The premise (that "powerful models" implicate "safety" concerns that must be controlled by "companies") does not seem true or self-evident to me. This tired fearmongering campaign centered on GPT-4 three years ago, a model now surpassed by open models that run on laptops.

Did fearmongers like Amodei say, "Oops, we were wrong! It wasn't that dangerous after all"? No. Of course they didn't.

[regularfry]

Either

1) the company has device-level control to the degree that they can not only restrict which API endpoints people can connect to but which accounts they use to do so (in which case this already isn't an issue); or

2) they don't, and all bets are off anyway, open weights or not.

[jquery]

Yeesh. “What shall we do sire, when the peasants learn to read?” vibes

[entropicdrifter]

You mean to tell me that anyone can own a nail-gun? We can't have people buying their own nail-guns, next thing you know they might build things that aren't up to code!

[pmarreck]

This does not load (or no longer loads) over here.

[halcyonblue]

https://x.com/coinbureau/status/2071330294452666695

[ta93754829]

that link doesn't exist anymore? what did it say?

[argee]

Strange, it opens just fine...as long as you aren't logged in (to X).

[pmarreck]

What is the point of an X link that only loads if you are NOT logged into X? That's more fascinating than anything this link has!

[xeromal]

Works for me

[jimbo808]

https://www.judiciary.senate.gov/imo/media/doc/2023-07-26_-_... https://xcancel.com/coinbureau/status/2071330294452666695 https://www.techpolicy.press/transcript-senate-hearing-on-pr...

> "Once the weights of a model are public, they cannot be retrieved. If a model possesses dangerous capabilities, it is permanently out in the wild... We need to consider regulatory frameworks that account for the unique risks of open-source distribution of highly capable frontier models."

[TurdF3rguson]

That's true I guess. If someone decides a model needs more guard rails, anthropic can adjust it, whereas with open weights it's too late.

It definitely sounds like the kind of thing that ends the world in B sci-fi thrillers.

[TurdF3rguson]

Curious if I'm being downvoted by people who think there will never be a model with dangerous capabilities? Or people who think when it happens we shouldn't add guard rails?