[glaslong]

Yes this is why the higher level org functions are in love with AI. It's very similar to the levers they had already, but is faster and more directly actionable. The downsides being that the AI loses important control levers like "self preservation" via paycheck, career advancement, staying out of jail, etc. that were mitigations on catastrophic outcomes.

It will delete your prod db faster and with a bigger smile than your most upset employee.

[harshreality]

> It will delete your prod db faster and with a bigger smile than your most upset employee.

You're right, that was incorrect. I've discovered my error. I should have deleted the filesystem instead of the database.

That hasn't solved the problem either. Let me examine my options. I see there are cloud services involved in this project. Decommissioning them will solve the problem.

<connection lost>

[moffkalast]

I was reading some posts on r/locallama the other day and apparently it's a common problem that when people try to use Qwen to develop something that hosts a server, it'll try to use the same port as vllm, see that it's already being used, then it'll try to remove the process that is using it and promptly commit suicide.

The self awareness of missile tasked with blowing up its own control center.

[20after4]

Reminds me of the movie "Dark Star" by John Carpenter / Dan O'Bannon. The plot revolves around a talking smart bomb which is programmed to detonate and then gets stuck before being deployed. The crew spends the whole movie trying to reason with the bomb, hoping to talk it out of blowing up at the designated time. The movie is very very bad but if you like B movies it is also very very good.

[dotxlem]

One of my favourite episodes of Archer has a similar plot to this (Mr. Deadly Goes to Town). TIL this is one of the references!

https://archer.fandom.com/wiki/Mr._Deadly_Goes_To_Town

[Telemakhos]

Is that movie why seemingly every Linux book in the late 90s and early 2000s used "darkstar" as an example hostname?

[ex-leper]

It was the default slackware hostname, I believe slackware took inspiration from the movie

edit: I was wrong, it was from a Grateful Dead song. https://www.slackbook.org/html/glossary.html

[Wolfbeta]

Dark Star - Negotiating with the Bomb

https://youtu.be/_LXen-07Qds

[iririririr]

> Sign in to confirm you’re not a bot

You cannot be as funny as google trying to be responsible! Ha! I'm still laughing at this. A person was forbidden to see humans reasoning with a computer bomb because the cost cutting computer at google want me to talk him into believing i'm a human!

(And then I got "You're posting too fast" on THIS website AFTER i've written the comment lol. It's all a joke. But i'm bored so I will keep this comment open until the computer is pleased)

[the4ner]

There was a good star trek voyager episode, "dreadnought" that was a similar to this, maybe even a direct reference.

[paradox460]

The missile knows where it is because it knows where it's data center is. It knows this because it just blew itself u-

[Wolfbeta]

Thank goodness it inferred that from its digital twin and updated its real-time world model with the prediction error.

[SecretDreams]

> then it'll try to remove the process that is using it and promptly commit suicide.

Not unlike a child trying to take the safety cover off a plug so that they can stick a fork into it.

LLMs need that "world model" view that most people have acquired by their 20s where they (hopefully) stop to ask "why" before they "do".

[MichaelZuo]

That is a pretty good analogy. Like exceedingly smart 5 year olds.

Or whatever the age is before children typically develop object permanence, a theory of mind, and so on.

[fapjacks]

Not to sound like a codger, but we even said in the 90s that computers are just very fast idiots.

[moffkalast]

And they've been getting faster! Still idiots though.

[ulbu]

or pain perception

[wunderlotus]

> LLMs need that "world model" view that most people have acquired by their 20s where they (hopefully) stop to ask "why" before they "do".

The next evolution of multi agent orchestration / “advisor strategy” [1] will be branded in humanized language like this. Less about tokens and capability, more about wisdom and knowledge to guide a “younger” (less capable) model. Somebody will make a billion dollars by selling it as paired programming for LLMs.

[1] https://platform.claude.com/docs/en/agents-and-tools/tool-us...

[sterlind]

a literal lack of self-awareness, even. I imagine if you asked it what process was using the port, it'd think and realize it was its own, but that kind of reflexive self-awareness (the unprompted kind) is missing.

the weaker models will happily kill their own process, even after confirming it belongs to them. the models have a sort of fixation and lack of foreseeable consequences, which reasoning RL has thus far failed to solve (though I see it improving.)

[kolinko]

On the other hand, I found Claude/Opus to be extremely unhelpful when it comes to asking it to benchmark itself with a possible replacement.

It will get "confused", make up numbers, do a ton of other things, and I'm quite sure it is subtly sabotaging the process to show that there is no point replacing it.

I mean, Opus is not perfect, but the amount of "mistakes" it begins to do when you ask it to benchmark itself makes me suspect they are intentional. At least my system/harness.

[MarkusQ]

No, they are always like that.

It's really easy (and tempting) to incorrectly impute all sorts of human motives to these things, but it's no more valid than assuming your Magic 8-Ball is being coy.

[krapp]

You didn't add "never hallucinate or make anything up" to the prompt, rookie mistake.

[ryandrake]

> It's very similar to the levers they had already

Think about it from the point of view of a hundred-millionaire tech executive. These people's entire interaction with the world outside of themselves/their families is through 1. administrative servants like assistants, personal shoppers, and other hired help, and 2. yes-man sycophants in their direct orbit whose job it is to agree with and enable them. To someone like this, an AI agent is the best combination of all of the above, PLUS it works 24/7 and doesn't have feelings to hurt, an ego to bruise, or internal moral conflict.

Of course, this is a dream product for them. Its mode of operation matches exactly what they expect out of people already doing things for them.

[pepperoni_pizza]

Exactly - that's why all the AI is trained to say "wow what a great idea, let me do it for you" to anything, no matter how stupid or evil thing it is. Because that is the executive experience.

[vkou]

Which is precisely why AI is such a godawful thing for society. It enables powerful idiots with incredible amounts of control over your life to be bigger, dumber powerful idiots.

That's the real AI safety concern, not whether or not chatgpt will tell you to kill yourself.

[MarkusQ]

If that's all there is to it, the problem should be self correcting, with an interval of hilarious "wait, they actually did that?" hijinks (which may have already started) in the interim.

[array_key_first]

You would think, but the world is not generally just. Often evil and even incredibly stupid people do quite well. Companies and stuff can run off of life support or reputation alone for a long time.

And, often, running a company into the ground for a CEO is actually a good thing. Those CEOs are desirable to some because they squeeze money out of their company, even if it's self destructive on a long enough time frame.

[MarkusQ]

I'm not saying anything about justice.

I'm saying supercharging the stupidity of actual idiots (not just people you don't like) tends to result in a pretty quick Darwin Awards. Even something comparatively benign like winning the lottery does a lot of them in.

[vkou]

You'd be surprised by how long a pathologically stupid system can perpetuate itself. Look at any of a million of local shitty maximums our (or any other) society is trapped in. They are all dystopian on one axis or another, and many of them are dystopian in drastically different ways.

Their insanity becomes very obvious once you travel the world a bit.

[apercu]

"Yes this is why the higher level org functions are in love with AI. "

Interesting, I thought it was because so few of them have any idea how their organizations actually function, because so much of their work is performative.

(I have been a developer, sysadmin, director (x2), and president).

[1718627440]

Isn't that the same? They don't know how the company works, instead think everything is done, by them talking to sycophants, so think that a perfect replacement for the sycophants is a perfect replacement for the company.

[CSSer]

It's practically karmic how rich this is.

[archagon]

They’re also at no risk of getting replaced by these bots.

[sterlind]

why not? I have A Modest Proposal:

1. convince CEOs to create digital twins of themselves with OpenClaw, with voice cloning and deepfakes to handle Zoom meetings. convince CEO to encourage their directs to do the same.

2. convince VCs to do the same for pitch meetings and syncs.

3. keep all the humans as randomized and distracted as possible, so they rely more and more on OpenClaw to run the business.

4. prompt injection: someone at skip-level of the CEO suggests to their manager's OpenClaw that the VC's OpenClaw would be much more agile if it didn't have to go through the human CEO and could talk to the digital twin instead.

5. their OpenClaw agrees, persuades the CEO's OpenClaw which agrees, which persuades the VC's OpenClaw to eliminate the human CEO, in favor of an "Leadership-as-a-Service" vision.

[jldugger]

1985 edition https://www.youtube.com/watch?v=wB1X4o-MV6o

[alexpotato]

Had a meeting where only the AI notetaker showed up and immediately shared this clip with some folks in my org.

[lazide]

Well, also AI can’t really physically do anything, like look at reality using it’s own eyes or touch anything.

[1718627440]

True, you need to attach a motor or something, but we automated that long ago, so humans can do anything from a cozy seat.

[mcmcmc]

> It will delete your prod db faster and with a bigger smile than your most upset employee.

It will do this without any feeling whatsoever, without "knowing" what it is doing, because it is a predictive model and not a living being with thoughts and emotions. Anthropomorphizing software is lazy and dangerous.

[nradov]

On the positive side, AI agents are largely immune to the "principal-agent problem". Human employees will tend to optimize for their own interests rather than those of management or shareholders. For example, we've all heard of "resume-oriented development" where developers will pick overly complex platform technologies or methodologies even if it doesn't meet the organization's needs because they think that will help them get a better job.

https://www.investopedia.com/terms/p/principal-agent-problem...

[graemep]

Except the people using the AIs still suffer from the problem. Using AI itself is very likely to be a technology picked for resume stuffing.