Hacker News new | ask | show | jobs
by ok123456 1091 days ago
It's not a local model. It queries some endpoint on someone else's computer.
5 comments

Not local yet. Considering the LLM/generative AI velocity we’ve seen, it’s only a matter of time. It’s helpful to see what others build, providing signal it can be built.

If you’re not comfortable using it in your workflow, consider it a peek at what’s to come. Very exciting times. And it's open source.

Yep, some google cloud server:

    SERVER_URL = "http://34.135.112.197:8000"
Yes indeed. The models are too computationally expensive to run locally (7.5Billion parameters). Though you could in-principle swap in any local model.
Do y'all have plans to release the model for those who have 16gb graphics cards? (I'm assuming the model is fp16?)
What are you talking about? 7b parameter models run insanely fast if you can offload to gpu, and are entirely reasonable speed if CPU only.
Does it prompt for an API key?
Nope. No API key needed since we mostly serve our own Gorilla models.