[wodenokoto]

The nice thing about active learning with a classic ML model is, everytime you annotate a data point, the model learns.

How do you update your prompt to take the new data point into account? Or do you just add it as an example inside the prompt and let it grow?

[gwern]

Yes. You can also do retrieval on your set of classified examples so the prompt only contains the most similar examples, and as your set grows, the prompt becomes more useful. (Even if this is still not as good as finetuning would be.) Note that you can do multiple prompts if you are willing to pay for even more accuracy, to ensemble different prompts.