[Vetch]

No, I meant finetuned. I also meant finetuned when I said trained. Experience with applying finetuned sentiment classifiers on real world data found gain vs cost of running to not be worth it. They remain nearly as brittle as cheaper classifiers and have a habit of gloming too much unto certain adjectives. They are also prone to overfitting on finetuned data's domain. Transformers trained not specifically on sentiment but on general domains like question answering or entailment are just leagues better for sentiment tasks.

[nl]

Well when you put the sentiment head on a pretrained language model you kind of have to train that head a bit on the sentiment task right?

But if the rest of your model is frozen the head will never see actual words, just contextual vectors from the LM.

It feels like we are in strong agreement but using slightly different terms or something