[the_tli]

Thanks for sharing. Why is the training dataset that contains instructions and output wrapped by another enclosing prompt (https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622...)

Why does this even work when during inference this wrapping prompt is absent? Wouldnt the model then work best against a inference prompt that follows the wrapping prompt structure, however the desired outcome is to have a model that just works without the wrapping prompt?

Edit: see reply from OP, the wrapping prompt is used for inference as well, so misunderstanding on my part

[minosu]

The wrapping prompt is also used during inference. (https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622...) Prompting like this is useful for instruct-finetunes, and similar prompts are used by other projects like stanford-alpaca.

[the_tli]

Thanks for the clarification, makes sense now!