[miggol]

All this marketability tuning may some day result in models which are extremely finely attuned to our current societal norms and taboos.

At which point the models will stop reinforcing (racial/gender) biases and start reinforcing said taboos instead. I don't think anyone wants that either

[slowmovintarget]

This is the real danger: Baking ideological bias into a utility function for the worst of all worlds.

An LLM can't be ethical because mercy cannot be computed through a distance function. Truth isn't weighted, and justice requires context windows so large that humans can barely manage it and often fail.

We can tune these things to be as inoffensive as possible and they will deliver a seeming that is impenetrable to the casual user, because that satisfies the utility function. That seeming will be worse than the loss of consensus reality we've experienced in the last few decades.

I don't find it frightening that LLMs spit out things that are wrong. I find it frightening that so many people are ready and willing to cede the details to these programs and abrogate critical thought.