[Xss3]

Will it though?

Fundamentally this is an alignment problem.

There isnt a single AI out there that wont lie to your face, reinterpret your prompt, or just decide to ignore your prompt.

When they try to write a doc based off code, there is nothing you can do to prevent them from making up a load of nonsense and pretending it is thoroughly validated.

Do we have any reason to believe alignment will be solved any time soon?

[aswegs8]

Why should this be an issue? We are producing more and more correct training data and at some point the quality will be sufficient. To me its not clear what speaks against this.

[Xss3]

Look up AI safety and THE aligment problem.

This isnt a matter of training data quality.

[skissane]

We don’t expect 100% reliability from humans-humans will slack off, steal, defraud, harass each other, sell your source code to a foreign intelligence service, turn your business behind your back into a front for international drug cartels-some of that is very low probability, but never zero probability-so is it really a problem if we can’t reduce the probability to literally zero for AIs either?

[Xss3]

Humans have incentives to not do those things. Family. Jail. Money. Food. Bonuses. Etc.

If we could align an AI with incentives in the same way we can a person then youd have a point.

So far alignment research is hitting dead ends no matter what fake incentives we try to feed an AI.

[aswegs8]

Can you remind me of the link between alignment and writing accurate documentation? Honestly don't understand how they are linked.

[Xss3]

You want the ai aligned with writing accurate documentation, not aligned with a goal thats near but wrong, e.g. writing accurate sounding documentation.