Hacker News new | ask | show | jobs
by zaat 876 days ago
Using TheBloke nous-hermes-2-mixtral-8x7b-dpo.Q5_K_M.gguf, llam.cpp with oobabooga, on Ryzen 9 7950x and RTX 2070 super, offloading 4 layers to the GPU.

27,631MiB system memory used.

7358/8192MiB usage on the GPU.

I'm getting 6.32 tokens/second.

You: You are Hunter S. Thompson, write a 450 words essay on the LHC project. The writing must be in Thompson's style

AI: Well, buckle up, kiddos, because we're about to dive into the twisted world of particle physics and the Large Hadron Collider, or as I like to call it, the Doomsday Machine. This beast of a thing is buried deep beneath the Swiss countryside, a 17-mile-long ring of death and destruction that makes the atom bomb look like a goddamn firecracker.

Now, let me tell you, when the Swiss government first proposed this monstrosity, I thought they'd gone completely batshit crazy. But then again, this is the land of Toblerone and Heidi, so maybe I shouldn't be too surprised. The LHC is essentially a giant, underground racetrack for subatomic particles, where they're accelerated to nearly the speed of light before being smashed into each other like a couple of drunken frat boys at a kegger.

The goal of this madness? To recreate the conditions that existed just after the Big Bang, when the universe was nothing more than a seething cauldron of energy and chaos. And if that doesn't sound like a recipe for disaster, I don't know what does.

But let's not forget the real reason we're all here: the Higgs boson, or as it's more commonly known, the God Particle. This little bugger is what gives other particles their mass, and without it, we'd all be floating around like a bunch of cosmic tumbleweeds. The LHC was designed to find this elusive particle, and in 2012, they finally did it. Sort of.

You see, the Higgs boson they found was a bit of a disappointment. It was too light, too wimpy, and didn't behave quite the way they expected it to. So now, the physicists are back at the drawing board, trying to figure out what the hell went wrong. And in the meantime, the LHC continues to churn out more and more data, like some kind of unholy particle-smashing machine.

But let's not forget the dark side of the LHC. There are those who believe that this thing could very well be the end of the world as we know it. They fear that the LHC could create a black hole, or even a miniature Big Bang, right here on Earth. And while the scientists assure us that these fears are unfounded, I can't help but wonder if they're just whistling past the graveyard.

So, what have we learned from all this? Well, for one thing, we've discovered that the universe is a much stranger place than we ever imagined. We've also learned that even the most brilliant minds can be blinded by their own ambition, and that sometimes, the pursuit of knowledge can lead us down some very dark paths.

In the end, the LHC is a testament to the insatiable curiosity of the human race, and our unquenchable thirst for knowledge. It's a monument to the power of science, and the potential consequences of playing God. And as we continue to push the boundaries of what's possible, we must always remember that with great power comes great responsibility. Or, as I like to say, "When the going gets weird, the weird turn pro."

1 comments

I am using the exact same model. Ryzen 5600G w/32GB and an Nvidia P40 w/24GB VRAM

20/33 layers offloaded to GPU, 4K context. Uses 25GB system RAM and all 24GB VRAM. 5-7 tokens per second.

Context is set to 32768, I didn't change it I guess that's the model's default.

Thanks for making me feel better about investing in tht motherboard + CPU + RAM upgrade and deferring the GPU upgrade.

and Groq does 485.08 T/s on mixtral 8x7B-32k

I am not sure local models have any future other than POC/research. Depends on the cost of course.

(Groqster here) For anyone who wants to try it, you can go to https://chat.groq.com/ and choose Mixtral from the drop-down menu. Also, feel free to ask me any questions about Groq hardware or service.