[israrkhan]

do you have a suggestion or a way to measure if model capabilities are getting destroyed? how do one measure it objectively?

[mensetmanusman]

These are now questions at the cutting edge of academic research. It might be computationally unknowable until checked.

[RALaBarge]

Ask it a series of the same questions after you train that you posed before training started. Is the quality lower?

[israrkhan]

That series of questions will measure only a particular area. I am concerned about destorying model capabilities in some other area that that I do not pay attention to, and have no way of knowing.

[simonh]

Isn’t that a general problem with LLMs? The only way to know how good it is at something is to test it.