[jagged-chisel]

"Alignment"

In what world would I ever expect a commercial (or governmental) entity to have precise alignment with me personally, or even with my own business? I argue those relationships are necessarily adversarial, and trusting anyone else to align their "AI" tool to my goals, needs, and/or desires is a recipe for having my livelihood completely reassigned into someone else's wallet.

[seizethecheese]

Why would relationships with a commercial entity be "necessarily adversarial"? A commercial relationship depends on the product providing more utility than the cost (for the consumer) and providing more revenue than cost (for the commercial entity). This means that while some components of the relationship may be adversarial in some areas, it cannot really be entirely adversarial.

[jagged-chisel]

> Why would relationships with a commercial entity be "necessarily adversarial"?

Because they want to separate me from as much of my money as they can, and I want to keep as much of my money as I can.

[apsurd]

I think we're living in times where the one place that this doesn't hold is now somehow all legal: addiction.

[seizethecheese]

Yeah, addiction and monopoly.

[sigbottle]

Interesting you single out commercial and government entities but not people. What defines the difference? Bureaucracy? Concentration of resources? Legal theory?

I guess I'm trying to wonder why this line of thinking (in theory) doesn't turn to paranoia about everybody. I don't know much ethics or political theory or anything.

[jagged-chisel]

> … paranoia about everybody

It does. People drive these entities. People hide behind the liability shields and authority of these entities. Also notice that I generalized with the phrase “…and trusting anyone…”

[dgb23]

The issue is power.

I'm not an expert in political theory or ethics either, but in my worldview, power relationships matter in these discussions. I believe power and responsibility should go hand in hand, and I hold entities to a standard that is proportional to their power to influence others lives.

If an entity's power is decentralized, for example when it is democratically organized to some degree, then that disperses both power and responsibility.

[robot-wrangler]

You can tell that broad alignment between people is natural just by looking at the effort that corporations and governments make to undermine it. Alignment between people is perhaps not a state of nature, but it really is a pretty normal consequence of a fairly small amount of education and of middle-class existence that is left to itself (i.e. without brain-washing and deliberately working to create out-groups). If you're eating enough and have a few brain cells to rub together, then you definitely want that for your neighbors too because it promotes stability.

[JumpCrisscross]

> broad alignment between people is natural

Uh, what? People have been killing each other over values misalignments since there have been people. We invented civilization in part to protect our farms and granaries from people who disagreed with us on whose grain was in said granaries.

[bluefirebrand]

We would never have even reached "farms and granaries" if alignment between people didn't happen pretty naturally

[JumpCrisscross]

Fair enough. We are a social species. But those alignments occur in small groups. You don’t need effort by “corporations and governments” for nations of millions of people to schism. If anything, those large institutions drive broad-based alignment.

[uoaei]

Methinks you've been sitting in your armchair too long.

Broad-based alignment doesn't come from nothing, but it is surprisingly easy to achieve when a population recognizes a shared stake. A synthesis between selfishness and altruism emerges when you consider who you can call a "neighbor".

[robot-wrangler]

Couldn't read the next sentence before wading in, huh?

[JumpCrisscross]

> Couldn't read the next sentence before wading in, huh?

Whatever the difference between naturalness and a state of nature, it has nothing to do with education or middle-class existence.

[dns_snek]

Critical bit:

> i.e. without brain-washing and deliberately working to create out-groups

[JumpCrisscross]

And if my grandmother had wheels she’d be a bicycle. The process of creating an in group naturally creates out groups. The “brainwashing” OP describes is just as natural as social alignment through an innate drive for conformity.

[cwmoore]

Conformity I think follows the innate drive to coerce the nonconformant into compliance

[pineaux]

There is a pretty interesting study of a large group of chimps. I dont remember where exactly but they have been civil warring the last 15 years or so. Point is, it seems that there is some kind of innate group formation process.

[zozbot234]

> You can tell that broad alignment between people is natural

It really isn't. The whole point of the market system is to collectively align people's actions towards a shared target of "Pareto-optimized total welfare". And even then the alignment is approximate and heavily constrained due to a combination of transaction costs (which also account for e.g. externalities) and information asymmetries. But transaction costs and information asymmetries apply to any system of alignment, including non-market ones. The market (augmented with some pre-determined legal assignment of property rights, potentially including quite complex bundles of rules and regulations) is still your best bet.

[jasonwatkinspdx]

Please read David Graeber.

What you describe is factually not how human society formed.

[zozbot234]

AIUI David Graeber famously pointed out that people in small groups can form the equivalent of a "market" simply by exchanging favours ("I'll scratch your back if you scratch mine") in an informal gift economy, without any money-like token or external unit of account. That's quite in line with what I said.

[jasonwatkinspdx]

You understanding is mistaken. Graeber's "everyday communism" is not a market, and his whole larger point is that contorting everything to the lens of markets is simply ahistorical and unempirical.

I'd strongly suggest reading his books. They profoundly changed my understanding of how human institutions and society form.

[dns_snek]

Broad alignment =/= Wealth maximization.

[_DeadFred_]

The market aligned us with children working in sweat shops after we outlawed it by convincing us it was OK if it was foreign kids and we got to share in pocketing the savings not just the evil factory owner.

[dns_snek]

Yes I'm well aware. Of course that's not how things are advertised to people, and they absolutely hate it when this is pointed out to them. This tells me that deep down they don't actually agree with how the system operates.

[rockskon]

Incentives and resources to promote said incentives.

[MaulingMonkey]

> Interesting you single out commercial and government entities but not people. What defines the difference? Bureaucracy? Concentration of resources? Legal theory?

Not OP, but for me, kind family and friends, and various feel-good pieces of fiction and other writing, at least let me envision the possibility of a perfectly kind/dedicated/innocent/naieve individual who is truly on my side 100%. But even that is mostly imagination and fiction... although convincing others of that isn't necessairly an argument worth making.

Commercial entities have a fundamental purpouse of profit. While profit doesn't have to be a zero-sum game - ideally, everyone benefits in a somewhat balanced way - there's some fundamental tension, in that each party's profit is necessairly limited by the other party's.

Government entities have a fundamental purpouse of executing the will of the state, which is rather explicitly not the same thing as the will of you as an individual.

Both commercial and government entities also tend to involve multiple people, which gets statistics working against you - you really gathered that many people who would put your needs above their own, with exactly zero "imposters" - which in this context just means people with a bit of rational self interest?

> I guess I'm trying to wonder why this line of thinking (in theory) doesn't turn to paranoia about everybody. I don't know much ethics or political theory or anything.

Just because you're paranoid, doesn't mean they aren't out to get you. Trust, but verify.

You might not be able to put absolute blind trust in anybody. I certainly can't. However, one can hedge one's bets, and diversify trust. Build social circles of people with good character, good judgement, and calm temperments - and statistics will start working for you. It's unlikely they'll all conspire to betray you simultaniously, especially if you've ensured betrayal costs much and gains little. While petty and jealous people can indeed be irrational enough to betray under such circumstances, it'll be harder for them to create the kind of conspiracy necessary for mass betrayal that might cause significant enough damage to warrant proper paranoia. You might still have to watch out for gaslighters stealing credit (document your work!) and framing people (document your character!) and other such dishonest and manipulative behavior... but if everyone's looking out for the same thing, well, that's just everyone looking out for everyone else! That's a community looking out for each other, and holding everyone honest and accountable. Most find comfort in that, rather than the stress paranoia implies.

Put yourself in a room full of manipulators and schemers, on the other hand, and "parnoia about everyone" might be the only reasonable or rational response!

[bitwize]

> But even that is mostly imagination and fiction... although convincing others of that isn't necessairly an argument worth making.

There was a Japanese visual novel in the 2000s about a girl who was your personal maid, and was so devoted would always take your side in any conflict, accept and support you just the way you are, even if you were a horrid person to your friends. It turns out she was a ghost, or a kind of yokai, or something. Anyhoo, back on 2ch she attracted a fandom, and there was a second group of people on 2ch who labelled her a "useless person manufacturer" because if you actually had a person who always accepted you just the way you are and never pushed back, that can be actually a trap that prevents you from developing.

It's a theme that's relevant today when people have AI servitors that always glaze them. It puts even certain utopian AI fiction, like Richard Stallman's story "Made for You", into a whole new light.

[MaulingMonkey]

My family accepts me just the way I am a bit too much. I can't bring myself to blame them, when past "reformist" pressures have been misguided/misapplied and backfired, but I recognize the trap. It'd also be hypocritical to blame them, when I also accept me just the way I am a bit too much! I'd like to think I'm decent enough to people, but I'm certainly more useless than I'd like to be. (Un?)fortunately, I'm not in a position to suffer, and I'm at least aware of the problem!

One of the ideas I've toyed with, even before all the AI hype, is a dumb, semi-adversarial servitor. Something to nag or taunt me about chores not done, to interrupt me when I'm doomscrolling, to use as a vessel for precommitment, to challenge me in various ways. I've been too lazy to build it thus far. Many tools overlap the problem space, so I shouldn't be using that as an excuse - perhaps I should give StayFocusd another shot.

Conflict and other stressors - in moderation, within the limits of one's ability to handle - are important for growth and health. A tree shielded from wind is weakened as it fails to develop stress wood and structural strength. A good debate can sharpen my thoughts and mind, walking to lunch keeps my cardiovascular system healthy, rising to life's various challenges gives me the security of knowing I can rise to the occasion and gives me more skills.

[a_t48]

Which VN is this?

[bitwize]

It was called Suigetsu

[zozbot234]

> each party's profit is necessairly limited by the other party's

Profit is obtained by maximizing traded benefits and minimizing costs. None of this requires taking anything away from any other party.

[bigbadfeline]

> Profit is obtained by maximizing traded benefits and minimizing costs.

Gain is obtained by the easiest means available. Your narrow definition of profit is seldom the easiest, cheating is far "superior" especially when it's legal for some.

> None of this requires taking anything away from any other party.

"required" and "preferred" (e.g. because it's far easier) are different like night and day.

[MaulingMonkey]

Trade is just a combination of give and take. I give you X, and in exchange, take Y. Without the "take", it's not a trade, it's just a gift.

[__MatrixMan__]

You could expect such a thing in a world where consent was currency, rather than scarcity.

[tyrust]

> precise alignment with me personally, or even with my own business

Seems like a strawman, I don't think anyone means this when talking about alignment.

More general goals, like avoiding paperclip maximization, are broadly applicable to humanity.

[zozbot234]

If you've built an agent that can act even vaguely close to a paperclip maximizer, you've already solved 99.999% or more of the alignment problem. The hard part of alignment so far is getting the AI to do something useful in pursuit of the right goal, and not just waste energy. We still have no idea how to do this with any effectiveness: even modern "RL from verified feedback" systems are effectively toys, the equivalent of playing video games, not really of doing something useful in the real world.

[reasonableklout]

Huh? Modern RLVR systems are toys that can’t do anything useful in the real world?

We must be living in completely different worlds. Claude and other agents have completely upended work for me and every single other software engineer I know.