Btw, it's kinda crazy how bad the GPT4-J results in the blog are compared to the Dolly one, which seem pretty good. Do we know why it works so well to use this 50k dataset?
Dolly is instruction fine tuned whereas GPT4-J is not. Which means that it doesn't even understand that it is being instructed to do something, it is just doing an autocomplete.