[moozooh]

Sadly, market incentives pretty much always go opposite of moral incentives because morals put breaks on decisions that multiply value for the company but the company itself exists for multiplying value. The profit motive is built into the reason for its existence. It's a contradiction that has a lower probability of resolving in favor of morals as the company grows in size and accrued capital. Whichever moral principles the leadership may have had at the beginning, they always erode or get perverted over time simply because the market always has a stronger pull.

I hate that, by the way, but what I hate even more is that this is somehow the most effective way to run economies that we've found so far, and it ends up this way because instead of unsuccessfully trying to safeguard against greed and sociopathy, it weaponizes them outright.

[parasubvert]

The profit motive is not the reason for a company's existence, it is an optional personal/human motive.

Companies exist to create customers. Everything else follows that. There is no value, no profit, not growth, no action whether moral or immoral, unless you have a customer.

Market incentives by themselves don't tend management decisions towards immorality, unless you've created immoral (or amoral) customers, or you've accepted capital from immoral (or amoral) investors.

It always comes back to people. If your customers or investors are some level of evil (or some degree of amoral), then you as a corporation probably are going to wind up being some level of evil or amoral.

It's up to management and majority ownership to steer those as appropriate... are you're willing to take money from anyone? There's a useful but dangerous veil of ignorance that raises with scale & ubiquity, such as commodity or public equity/debt markets. The resulting anonymity requires diligence from the company, such as Know Your Customer / KYC , and clear statements of the principles & laws of the corporation in its prospectus to attract the right fit of investor... and a backstop of government regulation to encourage or require these minimum standards of behaviour.

[vladms]

I find "morals" difficult to evaluate objectively. Some people might find it "moral" that women do not have any education and just stay at home, which I find terrible.

But if most people in a society find something "wrong" generally they will organize to prevent that (even if it has value for a part of the society). I think it is simpler for everybody that economics (how we produce and what) is separated from morals (how we decide what is right and wrong).

[moozooh]

It may appear simpler on the surface but it's very easy to find that market forces that don't have any checks and balances on them eventually converge on increasingly aggressive and dehumanizing behavior—not unlike your example with women. I have many such well-documented behaviors to list as examples, and I guarantee you have encountered them regularly and been upset at them.

The way we organize in a society is by having governments, usually elected ones to represent what "most people in a society" actually think, to serve as an arbiter of applied morals in our interactions, including business. To that end, we codify most of them in laws with clear definitions to prevent things like unfettered monopolies, corporate espionage, poor working conditions and hiring practices, etc. This generally works, though it depends on how well a given government and its constituent parts does its job and whether it uses the power it has to serve the entire society's interests or the interests of the elites that drive decisions. We can see right now how it fails in real time, for example.

Morals don't have to be evaluated "objectively" (whatever that is) every time to be observed. Humanity has agreed on many things that make up UDHR, international law, and other related documents. It's not the hard part. Making independent actors conduct their business in accordance with these codes is the hard part. Somehow even making them follow their own self-imposed principles is crazy hard for some reason. When Amodei claims Anthropic develops Claude for the benefit of all humanity but greenlights its use for surveillance on non-Americans, that's scummy. When Amodei claims to be terrified of authoritarian regimes gaining access to powerful AI but seeks investment from them, that's scummy. The deal with Palantir, the mass-surveillance business, is scummy. Framing the use of autonomous weapons as only disagreeable insofar as the underlying capabilities aren't reliable enough is scummy. You don't need to be a PhD in morals to notice that.

[vladms]

The initial quote I responded to was:

> market incentives pretty much always go opposite of moral incentives because morals put breaks on decisions that multiply value for the company

Yes, both market and morals have to be defined and are subjected to some rules and conventions - as you mention correctly in the reply. What I think it could be more qualified is the market and moral incentives "always go opposite".

Even today in many countries the market ensure a lot of necessary things for a lot of the population. Not all topics can be managed as a market (for example I don't think healthcare or basic infrastructure fit) and not in all countries have such frameworks, but given the successful examples I think it's more about wrongly using the tool than due to the tool itself.

Regarding your examples (Palantir, Claude - guns/surveillance), the same things happened in places where market incentives are/were not a driving force (communist East Europe/China for surveillance, quite probable China for automated weapons).

Honestly I wish I could propose/explain what would help. But just blaming a generic tools that we have (market, AI, press) for the bad things resulting from incorrect usage, worries me, as it can lead to not using them even when they would work.