[aryamaan]

What’re some models and hardware combos we can run now? I am avoiding to go to OpenAI with my office’s stuff and can use some gpu(s)

[MINIMAN10000]

You would just need a computer which can fit 2 3090s in order to run those to run something like TheBloke/airoboros-65B-gpt4-1.3-GPTQ

https://www.reddit.com/r/LocalLLaMA/wiki/models/ gives you a list of VRAM requirements to load the model into GPU VRAM. the more VRAM the computer has, the larger the model you can load in, thus making 3090s the current consumer grade king due to price to max VRAM.

This being said however most models are LLAMA based which all fall under that specific research license.

So following the rules, you would be limited to a subset of models which are foundational models which allow for commercial use

[brucethemoose2]

I can easily run LLaMA 13B on my 6GB VRAM/16GB RAM laptop using llama.cpp (specifically Kobold.cpp as the frontend).

I can barely run 33B, but anything more than 800 context and I oom. But it would run very comfortably on a bigger GPU or a 24GB+ laptop.

Theoretically some phones can comfortably handle 13B on mlc-llm though in practice its not really implemented yet.

[int_19h]

llama-30B (which is actually 33B) and derivatives generally run fine with 4-bit quantization on a single RTX 3090 or 4090, although depending on group size used for quantization you may need to slightly dial down the context size.