[rafaelmn]

How do you evaluate quality ? Also I suspect the performance between models would varry between datasets. Heck it would vary on same model/source if you included that your mother was being held hostage and will be killed unless you summarize the source correctly :).

I think you are still stuck with try if it works for you and hope it generalizes beyond your evaluation.

[mrlongroots]

I think summarization quality can only be a subjective criterion measured using user studies and things like that.

The task itself is not very well-defined. You want a lossy representation that preserves the key points -- this may require context that the model does not have. For technical/legal text, seemingly innocuous words can be very load-bearing, and their removal can completely change the semantics of the text, but achieving this reliably requires complete context and reasoning.

[anon373839]

There are actually some clever approaches to eval abstractive summarization.

Examples: https://eugeneyan.com/writing/evals/#summarization-consisten...

[imoreno]

>evaluate quality

[information content of summary] / [information content of original] for summaries of a given length cap?