[throwme_123]

> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used.

$300k of hardware! Guess it answers my previous comment from Hetzner server https://news.ycombinator.com/item?id=35662925

[lsaferite]

They called out the costs incurred:

  $30 - dataset generation (OpenAI GPT 3.5-turbo)
  $24 - llama 7b fine-tuning (8x A100 80GB instance costs)
  $84 - llama 13b fine-tuning (8x A100 80GB instance costs)

[brucethemoose2]

You can customize a model with a "cheap" 3090 (or maybe a 7900 XTX), see here: https://github.com/Lightning-AI/lit-llama#finetune-the-model

What OP did is more intense, but Lora/Adapter still can give excellent results.

Pure AI cards for mere mortals aren't really a thing yet.

[theolivenbaum]

Anyone tried fine tunning on CPU already? I expect it to be much slower, but is it even practical?

[brucethemoose2]

Also I bet the pytorch training code is written with CUDA semantics. Maybe a JAX version would work without messing with the code.

[brucethemoose2]

Oh no not a chance, finetuning is really compute intense.

[sp332]

A100's cost about $4/hour/GPU to rent. So the total cost depends on the amount of time, but 8x for 24 hours would cost $768.

[akiselev]

8x A100 80GB are $12/hour or so on Lambda Cloud. If you get lucky enough to snag capacity.