Hacker News new | ask | show | jobs
by embedding-shape 11 days ago
> edit: on bigger tests, got it to loop pretty easily unfortunately, probably local settings.

Been playing around for a few hours with the poolside/Laguna-S-2.1-NVFP4 + poolside/Laguna-S-2.1-DFlash-NVFP4 + vLLM, been seeing the same behaviour. Usually new model releases are plagued with issues at release though, best to wait 1-2 weeks then retry, or better yet, investigate yourself :) Personally I haven't found any obvious issues.
1 comments
embedding-shape 10 days ago
Update: Seems quite literally they have bugs on the hardware I'm trying to run this with:

From Poolside CEO Eiso Kant on Twitter:

> Learning we have some bugs on the RTX6000. We’re on it. Team has worked non stop last days and it’s getting late for a lot of the inference folks, so might be until tomorrow till we have a solution. - https://x.com/eisokant/status/2079693050796785720

Update2: I'm now running poolside/Laguna-S-2.1-NVFP4 with vLLM 0.23.1rc1.dev1378+gd6dbdb9b0 (FlashInfer 0.6.14) and seeing slightly better results in regards to the looping. I can't see any specific changes that would affect this though, strangely enough.

link
embedding-shape 10 days ago
Update3, summarized from RTX6kPRO Discord:

> The BF16 checkpoint doesn't exhibit any of the complaints [...] with our current quants, the models tend to choose the wrong logits sometimes [...] why we'll need a requant [...] We're not aware of any bugs in any runtimes themselves [...] We have two remaining things that we're trying to tackle: some people are reporting thinking being too hard to trigger (^^) , and others are saying it thinks too much. We've seen much more of the latter internally

link
Lwerewolf 10 days ago
Deleted earlier, didn't see you post, pasting here:

/* Just started testing with the gguf (with gpu offload, m5 max 128gb), q4_k_m, running seemingly well. Speed is initially slightly faster than antirez/ds4 - decode tok/s in the 30s, prefill ~400-ish. Expected, given the slightly smaller size. Looks to be working fine, but too early to tell. Definitely likes to "think". */

Anyways, guessing that discord might be focusing on the nvfp4 stuff. I've noticed spelling mistakes in the thinking traces, tool calls have been fine so far.

link
Lwerewolf 10 days ago
Well, just ran said gguf on the GeneralsX codebase with a pretty open-ended "Explain this codebase to me, and the general game loop." prompt, and...

  Let me also look at the GameLogic::update() to see the rest of the update flow, especially the object update loop.

  Actually, I think I have enough information now. Let me also check the GameLogic::UPDATE to understand the full update flow.
...repeating forever.

Since they mentioned that they're working on new quants, guess I'll wait. From earlier tests on work stuff, it's definitely capable.

link
Lwerewolf 10 days ago
Right:

https://huggingface.co/poolside/Laguna-S-2.1-FP8/discussions...

Wait time.

link
embedding-shape 10 days ago
More updates from the Discord (also mostly about the NVFP4):

> we've got an NVFP4 checkpoint that has a KD of 0.135 relative to the BF16 checkpoint (for W4A4, on some GSM8K prompts). [...] We're still seeing some looping on W4A4 (although much less than before), whereas for W4A16 we've not been able to make it loop. Given that W4A4 is still a little broken, we won't make this an official release (we have one more idea planned to fix that). [...] The RC1 for NVFP4 is reachable as poolside/Laguna-S-2.1-NVFP4 @ RC1

Seems to be available already indeed: https://huggingface.co/poolside/Laguna-S-2.1-NVFP4/tree/RC1

link