|
|
|
|
|
by kylerush
696 days ago
|
|
1. The conda recommendation is in the JS App Walkthrough documentation page: > A Python package manager: we recommend conda or pip 2. Makes sense! Something like sqlalchemy/alembic would be cool for PostgreSQL support. 3. Ah, this is interesting. Will read up on the different ASGI implementations. I had just assumed that having LLM workloads, async or not, on your main web server would be a problem (memory and/or i/o), but maybe not. To do date I’ve been moving LLM i/o workloads to background jobs on different machines with Celery, but it’s a bit more work and also makes streaming impossible. I recently did a Qwik + Celery stack for heavy LLM use, but have wanted a pure Python solution. Thank you! |
|
You shouldn't generally run your AI model directly on your web server, but instead run it on a dedicated server. Or just use an inference service like Together, Fireworks, Lepton, etc (or use OpenAI/Anthropic etc). Then use async on the web server to talk to it.
Thanks for pointing our the JS app walkthru mention - I'll update that to remove conda; we don't have have FastHTML up as a conda lib yet! I also updated it to clarify we're not actually recommending any particular package manager.