[jstummbillig]

> Then what is it they are trying to guard against, if its not simply protecting their moat ahead of their IPO?

Let's just assume it was "only" that?

It's unreasonable to assume they are aiming to upset people who are just giving them money in the way they want. It makes no business sense, for any company. So that has to be a byproduct.

Model training is one of the more expensive undertakings in the world right now and distilling models from competitors against the TOS is apparently something that is going on for very little money. Why would they not "just" try to take measures against that?

[thewebguyd]

It's about how they took measures against it. Sabotaging the requests is super shady and breaks all other areas of trust in the company their models.

All they had to do was have a simple, transparent output "Sorry, that request is against our terms of service. This session has been terminated"

[zozbot234]

The hidden safeguard was not against distilling, it was against "frontier" ML research with no indication whatsoever of what "frontier" might mean, but possibly even including research into model safety or alignment. That amounts to deliberately boobytrapping research across an entire legit academic field, which is ridiculously unaligned behavior.

[solenoid0937]

This is the same as saying "well some unaligned countries will use refined nuclear material for energy, too!" lmao.

The vast majority of frontier research is about how to build better models, not about alignment.

[zozbot234]

And as a matter of fact, there's a lot of meaningful research into how to have different sorts of nuclear material that might be usable for power production but not hidden malicious development. That's the closest analog to "safety" and "alignment" in your scenario.