[wokwokwok]

Did you actually try it and find it useful or are you just speculating?

I was not particularly impressed by the tutorial notebook. I’m not sure I believe that automatic prompt generation is nearly as easy as it sounds.

What task did you try it on?

[okhat]

What did you find underwhelming if I may ask?

It shows you how it takes some ~25 Pythonic lines of code to make GPT-3.5 retrieval accuracy go from the 26-36% range to 60%.

Not a bad deal when you apply it to your own problem?

[wokwokwok]

The examples appear to knowledge retrieval and factoids only.

The concept appears to be large scale chain of thought and automatic prompt generate and fine tuning… but there don’t appear to be actual examples of this.

[okhat]

Ah okay makes sense, yeah we'll release more examples.

This is just an intro to the key concepts/modules.

[wokwokwok]

The problem is, there is a big song and dance about string template prompts.

…but, carefully crafted string templates would be a) simpler and b) arguably better with existing solutions for this task, because it’s a trivial task and you can hand massage your string template prompts for that.

So, the narrative really doesn’t make sense, unless you’re doing something hard, but the example just shows doing something easy in a very complicated way.

I get it, maybe you can scale this up better… but you’re really not showing it off well.

[okhat]

@wokwokwok Okay now we disagree. This task is not easy, it's just easy to follow in one notebook. (If it were easy, the RAG score wouldn't be 26%.)

As for "carefully crafted string templates", I'm not sure what your argument here is. Are you saying you could have spent a few hours of trial and error writing 3 long prompts in a pipeline, until you matched what the machine does in 60 seconds?

Yes, you probably could have :-)