[apimade]

This is what happens when you give people tools that let them achieve an outcome, without necessarily giving them the judgement or expertise to know whether the outcome is any good.

If you asked me to build a house, I could probably assemble something that would stand for a few months. Hopefully. It might even keep the rain out. But it might also fall on my head, because I do not know enough about building houses to be confident that it won’t.

And even if it didn’t fall on my head under normal conditions, I also would not know when I needed to design for earthquakes. Or floods. Or fire. Or wind. Or grandmother-cosplaying wolves with very strong lungs.

But if all I need is shelter for a day, would I necessarily care whether it lasts more than a week?

That is effectively what a website like this is. It is not really a product. People don’t depend on it. Tan’s visitors are probably using MacBooks and iPhones on fast networks, and most of them will never notice how bad it is under the surface.

That does not mean it is good. It means it is good enough for the context.

Most people also tolerated the hilarious gigabyte JSON parsing bug in Grand Theft Auto for years, until a hacker patched it and cut GTA Online loading times by around 70%: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...

It was good enough, even if people noticed how bad it was.

Business applications, and typical software really doesn’t have to be super tuned or perform fast. It just needs to work.

At least until your product category has been commoditized, and _then_ you’re competing on experience.

[operation_moose]

AI feels to me like having access to someone who got a D in literally every single course offered at a university. If you don't know anything about the subject they are smarter than you. If you do know the subject its unsettling how bad they are. Basically the Gell-Mann effect:

> The phenomenon of a person trusting newspapers for topics which that person is not knowledgeable about, despite recognizing the newspaper as being extremely inaccurate on certain topics which that person is knowledgeable about.

They've improved from someone who failed every single university course a couple years ago. Maybe they'll get to a C or even a B in the future; maybe not.

[greenavocado]

It's more like giving an army of crack heads with a heap of motivation access to textbooks and the internet to do your bidding

[logicchains]

>AI feels to me like having access to someone who got a D in literally every single course offered at a university. If you don't know anything about the subject they are smarter than you. If you do know the subject, its unsettling how bad they are. Basically the Gell-Mann effect:

I don't know how you can say that with a straight face when AI's capable of matching the best human students on the hardest exams we have, like the IMO, the Putnam and the bar exam. I can only assume you've only ever used the free tier of any AI service.

[klik99]

Exams are usually discussed and part of the corpus though right? Performance on novel problems are really the only good metric and those decay really quickly across models once they’re known, like the pelican on a bicycle test

[jrflowers]

I like that you start by saying that you don’t know how someone could draw that conclusion about AI and end by saying they probably got there by using AI

[msdz]

While that's true and pretty funny, it does make for a meaningful distinction IMO: Just like you wouldn't look at the "all Ds" student, but not the "straight A" student in order to get an impression of students' capabilities, you wouldn't ignore a newer, potentially SOTA-pushing model to then come out and say something along the lines of "all models suck"… right?

[MrToadMan]

The competence threshold can vary according to the model's complexity and cost to run, but the broader point remains that having a human operator of greater competence that can evaluate the output and understand potential harms that might result seems to be a valid one. Depending on domain, it can be irresponsible to deploy systems you don't fully understand that could harm others. Responsibly overseeing 37k lines of code a day sounds exhausting.

[wayland_jeremy]

We're trying to up the GPA on the code quality exam by giving them access to structural metrics calculated on graph representations of the programs they write. Hoping this is the study guide template they need to start getting an A in minimizing slop, even if they're actually a D student. They're good at optimizing against scores...

https://github.com/Krv-Labs/topos

[Voultapher]

Good on you for trying, and I'm not saying it can't improve things. Optimizing for structural metrics is still only as good as the metrics. I can say with confidence that no metric exists for good design, because good design is art. Art requires creativity and the evaluation lies in the eye of the beholder. It also doesn't help with the bigger picture, AIs often do the wrong thing because they have been tasked to do the wrong thing.

[dingaling]

There are no studies to suggest that "Gell-Mann" is a real phenomenon. It was invented by an author, Michael Crichton, on a whim.

Any rational reader will adjust their faith in a publication based on the identified errors it makes. Too many and they'll reject it as a source for anything beyond "a thing may have happened".

[shapefrog]

I thought you generally knew what you were talking about till i read this. Now I am wondering if you know anything at all.

[pjc50]

I think this is where aggregate effects have to be considered. One person building an idiosyncratic house out of found materials: neat little project. One million people doing so: shanty town that can be seen from orbit and is a disaster waiting to happen.

The Web already had a problem with externalizing costs onto users. Both the simple cost of poorly executing websites (power, mobile data, time), and more subtle ones (social media). AI is a huge accelerant for that.

[ethbr1]

Imho, this is the biggest concern for civilization-scale cost of AI being deployed all fields.

On the one hand: Amazing power! Look at how much more output everyone is generating!

On the other hand: Society maintaining guardrails that were engineered around human failure modes are completely inappropriate for bounding something that makes 10% random errors everywhere.

To re-use an example upthread... what if you had an LLM that could build an entire house? Foundation? Plumbing? Electrical? Roofing? Septic?

And then we let everyone guide that LLM to build their own house.

We'd drastically increase the housing supply, but every house would have 10% of its systems built incorrectly.

We could fix that by requiring grounding in building codes, automated reviews, etc. But none of that hard work updating guardrails is something AI accelerationists are interested in working on.

So you end up with a society where every septic system has a 10% chance of being misdesigned...

[27183]

> We could fix that by requiring grounding in building codes, automated reviews, etc.

It's an open question whether, with all those guardrails, AI would provide any speedup.

[ethbr1]

Absolutely. I have an inkling that there's an AI version of Amdahl's law to the effect of: the speedup obtained by AI is proportional to the amount of AI output that isn't verified.

Folks push back and say "But there are tons of things where deterministic verification is easy, but design is hard."

But I'm suspicious that in real world cases verification is a lot harder and more time-consuming (because you can't verify 95% of something and call it a day).

[27183]

I have a hunch that most of the things humans do are AI-complete. Very little of it can be easily automated directly without AGI. We can achieve automation by re-orienting our workflows around machines (factory assembly lines, for example), but that isn't always possible or useful. That's what makes ML applications difficult--finding how to re-orient a workflow s.t. an ML algorithm can be used productively in it.

[meetingthrower]

Correct. It's "good enough" for now and if it needs to be industrialized you can do that later. If it is brand new and has no traffic and is not exposing user data or opening up your keys to someone.... who cares how perfect the code is if it mostly works? Users don't have to use it...

[aljgz]

That is, until you have silent corruption of data because the admin tool is not aware of the new field you added to the website, and it's not written with forward compatibility in mind. (Have seen this lots of times)

Or a downstream service is overwhelmed and suddenly all the retired you added to different places DDOS your own service. (Also have seen this lots of times)

Some quality problems you can fix later. With others, once they happen, there's no "later"

[aljgz]

*retired -> retries

[rwmj]

In the thread the developer is only able to look at the front-end code. The real problem is more likely in the back-end code where there'll be hilariously bad security issues. Hopefully Tan isn't running this on the same server or network as anything important.

[officialchicken]

... and engineers don't have to consider this a good look coming from the head of YC. These are rookie mistakes - generated by both the human and AI model in a collaborative process. It's also the kind of mistake most of us aren't willing to make b/c we don't have the capital to sweep it under the rug if it goes pear-shaped.

[bluegatty]

It's not really a question about if GStack is 'good enough' - it's the claims about 37K LOC being legitimate.

Because it's producing 'functional slop' that has little bearing on how anything else would be made the 37K LOC is totally out of context and therefore a misrepresentation.

He has bragging rights over GStack - not 37K LOC, that's bad.

[meetingthrower]

That's a great way to put it: functional slop. That's what I mean tho - as long as it is minimally functional it does its job.

I totally get the shade thrown at ycombinator. But I guess my perception is that it has never been about the tech for them and more about getting quick hacks to get to market quickly. So it feels on brand?

[bluegatty]

You're right it's a fine line, but I think there was always a kind of underlying 'tech legitimacy'. PG is a big Lisp nerd. HN is ultra minimal and well designed.

It's one thing to say 'just make it work' - it's another to create hype on top of nonsense.

I don't want to diss vibecoding - because it has it's place, but GStack is being presented in the context of something new and amazing in terms of engineering productivity - not just 'a thing that works good enough'.

But there's an on-brand element to it.

It's a bridge to far though.

There is already backlash, whereas, I'm not sure there is backlash to AirBnB's initial attempts - literally nobody is going to care what their stack is.

[swat535]

> functional slop

I suppose the question is: will customers and business people care?

At the end of the day, if the system does what it needs to do to in exchange for money, then it's a success if people's eyes. Heck, even basic security and privacy features get ignored in most systems.

Average person treats their computers like any other appliance, they don't care if the app takes 10 minutes to load, or is slow, or ad ridden, they just want it to work.

Engineers obsess about code quality as an artistic expression but the reality is that executives and investors won't care. You can scream "maintenance burden!" all you want, but investors will happily throw money at the problem down the line to make it go away.

Our industry simply doesn't have any required rigor outside of a few edge cases in heavily regulated industries.

[DiskoHexyl]

>>if all I need is shelter for a day, would I necessarily care whether it lasts more than a week

An interesting choice of a metaphor here. If I needed a shelter for a day, I would have bought (or rented) a nice tent. Or booked a room in a proper hotel

[PunchyHamster]

> If you asked me to build a house, I could probably assemble something that would stand for a few months. Hopefully. It might even keep the rain out. But it might also fall on my head, because I do not know enough about building houses to be confident that it won’t.

Oh no. You'd go and research that, look at existing stuff, read about it, look at tutorials and while still making a ton of mistakes at least try to follow best practices.

People don't do that when they use AI to do task unfamiliar with them. They look at it working and just okay it the moment it "looks okay". They don't even ask the AI followup questions, just accept whatever

[close04]

> give people tools that let them achieve an outcome, without necessarily giving them the judgement or expertise to know whether the outcome is any good

This could be fine if all you're looking is to get the quick and dirty result at any cost, or private use, etc. When anything is better than nothing.

The problem starts when this extremely low bar becomes the baseline for anything. When you're willing to attach your name to a stream of absolute slop and you're even proud of it.

[klik99]

I recently did a feasibility test for porting complex c++ project to a stripped down DSP processor with no c or c++ stdlib without making changes to the actual project - sonnet was able to do it in half a day and stumbled for another half a day on a minor error until I pointed out its mistake, but the code was absolutely bad. Just the same, I would’ve taken two weeks to just know it was feasible and I have a reference to know how it avoided certain pitfalls. It’s taken about a month for two of us to do it right, but honestly I don’t think we would’ve attempted it without proof it was possible. I still wouldn’t put any AI code in production for actual end users (beyond the “shed built for a day” use you mention), but that doesn’t mean as a tool it’s not magical. It’s a false binary between the “it’s always amazing” viewpoint of Garry Tan and many others and the “it’s all slop” of the original author, it’s a tool that has specific uses and high costs right now. Its irresponsible to blindly assume it’ll continue to get better, but that doesn’t mean there aren’t valid use cases today

[6stringmerc]

“…without necessarily giving them the judgement or expertise to know whether the outcome is any good.”

Ahem…this man is A CHIEF EXECUTIVE OFFICER how DARE you cast aspersions on his judgment abilities! /s

The biggest takeaway for me is “failing upward” is much easier in the 21st Century.