[bioemerl]

And hey guys, there are two big open source communities running that focus heavily on running this stuff offline.

KoboldAi

oobabooga

Look them up, join their discords, rent a few GPU servers and contribute to the stuff they are building. We've got a living solution you can contribute to right now if you're super worried about this.

This stuff is actually a very valid way to move towards finding a use for LLMs at your workplace, they offer pretty easy tools for doing things like fine tuning, so if you have a commercially license model you could throw a problem at it and see if it works.

[indigochill]

Where I'm struggling at the moment is that I know about those but my local hardware is a bit limited and I haven't figured out how the dots connect between running those local interfaces against (affordable) rented GPU servers. The info I can find assumes you're running everything locally.

For example, I know HuggingFace provides inference endpoints, but I haven't found information for how to connect Oobabooga to those endpoints. The information's probably out there. I just haven't found it yet.

[bioemerl]

There is something called a run pod but I know I've seen a couple of these groups give quick easy links to use. You might want to look there.

> I know HuggingFace provides inference endpoints, but I haven't found information for how to connect Oobabooga to those endpoints

I've never heard of these so I'm guessing there isn't a way.

[amelius]

Where I'm struggling is how to keep up to date on the latest LLMs and their performance.

[akkartik]

I see https://github.com/oobabooga but where's the Discord posted?

https://github.com/KoboldAI/KoboldAI-Client does link its Discord.