Hacker News new | ask | show | jobs
by Vetch 1327 days ago
> NN based sentiment analysis is certainly a lot better than non-NN based techniques.

I wouldn't say this. Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers. They remained brittle and easy to trick until recent flexible systems systems based on question answering, zero-shot entailment or lotsa instruction finetuning (improving in that order). I strongly advice against using something fine-tuned solely on sentiment datasets. It'd be a total waste.
1 comments
nl 1327 days ago
> Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers

Well yeah. But why would you do that?

Do what eveyrone does: Train on large scale a language corpus (or use a pre-trained model) then finetune for sentiment analysis.

> I strongly advice against using something fine-tuned solely on sentiment datasets

Did you mean trained on sentiment datasets? I agree with that.

Otherwise, well [1] is a decent overview of the field. I think Document Vectors using Cosine Similarity[2] at 17 is the highest rated that isn't a NN trained on large corpus and fine-tune on sentiment task. Even that uses document vectors that are trained on a large language corpus.

[1] https://paperswithcode.com/sota/sentiment-analysis-on-imdb

[2] https://paperswithcode.com/paper/the-document-vectors-using-...

link
Vetch 1327 days ago
No, I meant finetuned. I also meant finetuned when I said trained. Experience with applying finetuned sentiment classifiers on real world data found gain vs cost of running to not be worth it. They remain nearly as brittle as cheaper classifiers and have a habit of gloming too much unto certain adjectives. They are also prone to overfitting on finetuned data's domain. Transformers trained not specifically on sentiment but on general domains like question answering or entailment are just leagues better for sentiment tasks.
link
nl 1326 days ago
Well when you put the sentiment head on a pretrained language model you kind of have to train that head a bit on the sentiment task right?

But if the rest of your model is frozen the head will never see actual words, just contextual vectors from the LM.

It feels like we are in strong agreement but using slightly different terms or something

link