[orangepurple]

This will probably work as long as the material being annotated is similar to the material the LLM was trained on. When it encounters novel data (value) it will likely perform poorly.

[nihit-desai]

Partially agree, but it's a continuous value rather than a boolean. We've seen LLM performance largely follow this story: https://twitter.com/karpathy/status/1655994367033884672/phot...

From benchmarking, we've been positively surprised by how effective few-shot learning and PEFT are, at closing the domain gap.

"When it encounters novel data (value) it will likely perform poorly" -- is that not true of human annotators too? :)

[orangepurple]

> "When it encounters novel data (value) it will likely perform poorly" -- is that not true of human annotators too? :)

Some humans have intelligence and reasoning abilities. No LLMs do :)

[yawnxyz]

I always regret making that assumption