[brucethemoose2]

Honestly, I dunno. I think most people are using lit-llama or EasyLM (on TPUs) for finetuning?

QLORA is the gold standard for more affordable training.

As for datasets, just look at the open datasets the best-in-class models are using, like Vicuna or https://huggingface.co/NousResearch/Nous-Hermes-13b

Some model datasets like Manticore, Chronos or the infamous Pygmalion are more "secretive," but you can find the dataset gathering scripts on Github or in community chats.