[loveparade]

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a different distribution of tasks.

A frontend dev doing tailwind integration for his day job is gonna see very different speedups than someone working in a niche scientific codebase. Taking the average makes about as much sense as taking the average of the speedup from calculators for a mathematician, a farmer, and an elementary school student.

[Cthulhu_]

And even if the refactor is 100x faster (which is unlikely) - would that work have been done at all without AI? What would the effect be on total speed / througput if it was or was not done? These things get more and more difficult to measure.

What I'm really wondering is how much extra tasks are being done that wouldn't have been done without AI, and whether those actually have a payoff. That is, 100x faster versus not doing the work at all.

I'm hoping some enterprises that collect metrics on e.g. time-to-market, customer satisfaction, revenue, costs, etc will release an authoritative report some day.

[rob74]

To add to that, the only way to reliably measure speedup would be to give the same developer the same task twice, first without AI, then with AI, and the developer should have no previous knowledge (or the same level of previous knowledge) about the task each time he starts - which is inherently impossible. I didn't read the study, but from the article it looks like they compared the actual speed to prior estimates, and we all know how reliable those are?

[Zababa]

>and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding

This isn't true in my experience, AI is great at gathering context through slack, repositories, emails, web pages. For building understanding too, provided you use it well.

[dakolli]

I have found llms to be utterly useless for frontend (tailwind included).

That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off.

I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "world", customer and product knowledge is required to design and implement great UX/UI?

I promise you are not going to be able to translate all this internalized understanding to an LLM and have it do your "tailwind integration" It actually sucks at all frontend outside of the 3 types of page layouts it understand.. Shitty landing pages, generic dashboards and shitty blog layouts.

Ya'll yearn for slop though so maybe everything will just become shit anyways.

[Zababa]

This isn't my experience at all, LLMs do graphs and more complex excel-like web pages very well. They also do dashboards very well. They even seem to do 3D stuff with three.js like video games pretty well too, although I haven't tried that myself. Maybe they can't do something "great", but they sure can do good enough in most cases, and good enough is already better than most websites.

>I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are.

Agree here, especially considering that usually "niche scientific codebases" have terrible code so you don't need a super smart model to get a good bost in software engineering.

[loveparade]

Fair point, I was more trying to make a statement about the amount of training data available, not the "difficulty" of the task. I just used Tailwind as an example because it is so ubiquitous with so much training data for LLMs to learn from, while any niche application doesn't have that.

[Zababa]

Training data has stopped being a good predictor of LLM abilities ever since they started doing heavy RL runs. I'm not sure how much corporate dashboards/I can't believe it's not excel stuff were in the training data, I guess not that many considering that stuff is almost always corporate and kept inside companies, and LLMs are still great at it, good enough to make people that used excel and used it well daily for 10+ years stop using it for lots of stuff.

[weitendorf]

I ran into this with my first attempt at building a static site generator "for agents" last year. https://statue.dev

I got very frustrated with LLMs and their inability to apply good taste or maintain consistent design languages, and put the project on ice. But I decided to double down on more tooling and learn as much about frontend as I could because I also realized that frontend itself - the problem domain, the engineering culture (or lack thereof), the historical baggage, the sheer size of the frontend api/language surface was part of the problem. And also there was/is a lack of good LLM and agent-oriented tooling that was a much deeper problem than I expected initially.

I originally thought I would just create skills/workflows and apis for generating sites from templates, but the problem is moreso that you need an entirely different kind of harness and development process for frontend, which doesn't really exist yet. Claude design is probably the most familiar gesture in that direction for most people but I think it's only scratching the surface. Our own "agentic playwright" is https://github.com/accretional/chromerpc/tree/main/chrome-pr... - IMO this kind of tool (both ours and Claude Design) is a major win for removing the largest, most frustrating frontend LLM painpoints (having a human doing QA and prodding the model to fix obviously-wrong outputs).

But the bigger problem is that the webdev tooling ecosystem is FUCKING AWFUL, and there are too many different ways to do something even using the actual base browser apis, let alone all the random ass low-quality tools and cargoculting that seeps into the models' way of working and thinking. That's not to say that tools like React are bad, necessarily, but that there is so much pre-LLM slop and churn and low quality/inconsistent work in the frontend ecosystem that you really need to be MUCH more knowledgable about the way browsers and the web actually work than the median frontend developer (especially the ones participating in the endless hype flavor of the months, generating all the noise that defines the engineering culture) to effectively use them. Or even better, if you know enough you can also NOT reach for them because you're able to just implement it via raw html/css/browser primitives instead of through 2000 node packages.

To be clear, I'm not saying frontend development is slop, but that it has a very high skill ceiling and requires a lot of very particular/thorny knowledge to be good at. I think the reason AI frontend looks so much like slop is that it hasn't been RLed against the actual web-standards in a way that lets it learn how to actually build good sites, it just has the median frontend engineer archetype from its pretraining and then some kind of RLVR to get it to produce workable, not-fucked-up code (the 3x3 grid, the slop hero, the unnecessary blinking green buttons, etc.). And also, for LLMs, maybe engaging with the webdev tool ecosystem beyond the core infrastructure layer and base apis/languages is more trouble than it's worth, because they often optimize for "I want a particular kind of UX and don't know how to implement it directly, but I do know how to find a package and call it, then prod it into working".

LLMs need something more like a browser-harness, a meta-design system, per-design-language component management tooling, and a non-slop build system. They also generally need much better support/more sophisticated UX for hierarchical iframes, CSP, etc. which is a space that is not very well-explored despite its potential, because most frontend devs find it too hard or complicated.

People are already starting to build these and I think we'll get there in the next year. The hardest piece of the puzzle is figuring out how to structure RL training envs to learn frontend directly against web standards, because web standards are very complex and high-surface; but this is also the most promising because it's how you get Mythos-like superhuman performance. We have a project to build some of the base domain modeling/search tooling needed for frontend RL, eg https://github.com/accretional/proto-css, but it's early days. You should definitely try agentic browser tooling if you haven't yet because it makes a huge difference in getting existing LLMs to be more effective at frontend, and automating most of the debugging. It's what allows us to eg fully automate creating gifs of models interacting with our site in the context of a user journey when we run tests: https://github.com/accretional/proto-css/tree/main/chrome-te...