[ekidd]

Having actually run some of the software produced by nearly "dark software factories," a lot of that software is completely shit.

Yegge's Beads is a genuinely good design, for example, but it's flakier and more broken the Unix vendor Motif implementations in 1993, and it eats itself more often than Windows 98 would blue screen.

I can actually run a bunch of orchestrated agents, and get code which isn't complete shit. But it's an extremely skill-intensive process, because I'm acting as product manager, lead engineer, and the backstop for the holes in the cognition of a bunch of different Claudes.

So far, the people promising completely dark software factories are either high on their own supply, or grifting to sell books (or occasionally crypto). Or so I judge from using the programs they ship.

[xg15]

I found it kind of fitting that didn't even describe what a human would still do at level 5 nor why it would be desirable. It's just the "natural" progression of a 5 step ladder and that seems to be reason enough.

[thenfcm]

Well isnt the point humans wouldn't need to do basically anything?

It would be 'desirable' because the value is in the product of the labour not the labour itself. (Of course the resulting dystopian hellscape might be considered undesirable)

[ekidd]

As I keep pointing out, if the model ever stops needing you to complete ambitious goals, then what does the model actually need you for?

People somehow imagine an agent that can crush the competition with minimal human oversight. And then they somehow think that they'll be in charge, and not Sam Altman, a government, or possibly the model itself.

If the model's that good, nobody's going to sell it to you.

[handoflixue]

A Dark Factory is a lot more work than the model, and often perpendicular to the goal of general model improvement. A Dark Factory specializes in building one particular thing, whereas the AI labs care about generalization and what you can do absent such advanced scaffolding.

It is so named because we have literal Dark Factories in the real world, run by robotics instead of AI, producing cellphones without any need for humans.

None the less, said literal Dark Factory that actually exists, in the real world, is still owned by the corporation that built it. The robots did not take over, the government did not seize it.

[darkwater]

> None the less, said literal Dark Factory that actually exists, in the real world, is still owned by the corporation that built it. The robots did not take over, the government did not seize it.

It's probably worth pointing out that hardware and software are two completely different things. At least until the day we have robots that can create and put to work other robots with 0 or minimal human intervention.