Hacker News new | ask | show | jobs
by sterlind 1166 days ago
I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell.

also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.
2 comments
superkuh 1166 days ago
Using nix and then complaining about having to set up your compilation environment libs/etc is kind of like sticking a rod in your bike's wheel spokes and complaining about crashing. Don't give up on the idea of system libraries (ie, use nix) and this doesn't happen.

Also, hi? I don't recall you by that nick but the internet is a small place sometimes.

link
sterlind 1166 days ago
oh, I'm very aware that I've brought this upon myself, but I'm sticking out for the greater good (and stubbornness.)

specifically, I'm trying to benchmark a bunch of different GPU configurations on different workloads on vast.ai, which uses Docker containers. I abhor Dockerfiles and my experience building containers with nix has been pleasant, so that's what I'm doing and why. fortunately I think I'm getting past the learning curve.

did our channel survive the demise of freenode? I was andares, I think I used to be annoying but I've gotten better.

link
superkuh 1166 days ago
Ah. Hi! Yes. We still exist in the same place but on libera now.
link
anotherhue 1165 days ago
Care to share some of your progress? I have similar (stronger?) feelings regarding Dockerfile's big-ball-of-state nonsense.

(The irony of holding this opinion while dealing with pre-trained AI models is not lost)

link
sterlind 1165 days ago
I finished my work on poetry2nix and submitted a PR which works perfectly (at least with preferWheels=true.) now I have a wonderful live environment with torch, triton, transformers, etc. Docker builds are fast and lightweight since I use buildLayeredImage. it is, truly, the promised land my forefathers prophesized.
link
siraben 1166 days ago
Have you been successful in getting the LLM playground up with Nix?
link
sterlind 1166 days ago
yes, almost! I used poetry2nix and grafted a bunch of overrides to fix the torch-2.0 build, and I just got cuda working with it. I'm testing triton now. I'll submit my PR to poetry2nix so watch that space if you want it.
link