[carterschonwald]

amusingly ive been working on ultra sparse llm inference/ training/ model design because nature loaths a dense graph/matrix and cause i think it shoukd be possible. i actually stood up a 20-25 percent faster than sota causal fast attention kernel yesterday, will be standing up cuda/metal/armv8 kernels too and thats gonna be fun.

i genuinely think these models should be like 0.1 percent sparse for same capabilities we associate with them today, but theres no sane way to do that with extent tools. i built the right core tech for that in 2014 when there wasnt a market, but now there is and the experimentation velocity is wild.

amusingly llms really have a hard time using my simple apis because its not in distribution array programs. but i literally stood up cpu custom memory format and micro kernel for dense causal attention in less than 24-36 hours and outperforms the equivalent fused ggml/llama cpp fast oath by like 20-25 percent

[torginus]

I am not much of a math person, but if we look at the how the brain is wired, we see that the dendrites (the inputs) of a neuron are hundreds of micrometers in length, and the axons (the outputs) are millimeters, and very rarely can stretch to tens of centimeters. So they can sample only a tiny amount of internal state, and affect a much larger, but usually still small output.

In math terms, this means a layer of a network can be represented with a block matrix in the whole 'layer' matrix, which I think means its sparse as you said.

As I said, my math knowledge is rusty, but I remember that a lot of matrix optimization techniques center around decomposing large matrices into these smaller blocks, which are then evaluated, and the output is combined in a final pass. Which leads to a huge reduction on parameter numbers and the time it takes to evaluate the result

[carterschonwald]

exactly. you certainly know more about the brain than i :)

[jacktang]

Look forwarding your future releases

[carterschonwald]

i definitely will be doing some drop of some faster attention kernels in the next few weeks.

like i can do all sorts of memory layout of tensors/matrices etc tricks that if you dont have the abstractions for it would just never happen. so i can optimize the kernel flops

[blagie]

Curiosity:

For most of the past five years, I've known ways to do better than Anthropic, OpenAI, and friends in many ways, at least on paper. I know I was right about many of them since many would show up 6-24 months later tools from the major providers, or otherwise become standard practice.

A central problem is the Mythical Man-Month. True, I could do those, beating then-state-of-the-art, but only given 2-5 years. I suspect many other people knew about them too and could do so as well. As I noted above, throwing people and dollars caused many of those to be built in less time than I could have regardless.

Other methods, I'm less confident about (>50%, <80%), but would lead to similar improvements orders-of-magnitude as you're predicting, but mine would need $$$$$ in compute and engineering infrastructure to build out. E.g. they need to not just theoretically work, but to try, I would need to convince someone to invest in them working.

So the TL;DR is that my knowledge was not at all helpful towards e.g. competing with OpenAI, Anthropic, or even building a small business.

However, where it was useful was in predicting where the industry was going. This is true in investing (but not easily, at least with my skill set), but in developing startups and systems, there were capabilities which I (correctly) assumed would be there, whereas there were many arguments that "AI will never be able to ____."

If I know how to do something, it will almost certainly happen, regardless of whether I'm the one who does it.

To be clear, my expertise is almost certainly nowhere as deep as yours. I'm not providing a direct analogy, or claiming others know what you do or can do the same. My point was really that if you believe you can have these models be 0.1 percent sparse for same capabilities we associate with them today:

a) You're probably right. They were built quickly for capabilities. A slower process can almost certainly lead to much smaller models too. That's a radical statement: Historically people claiming a 1000x improvement somewhere were crackpots, but that's very possible in an industry as fast-changing as this one.

b) Someone at Anthropic or OpenAI might be working on building out extent tools right now. Even if so, there are indirect ways to capitalize on that knowledge.

c) Critically, that predicts a future where Fable is $1/month instead of $100/month, and that's something which CAN be acted upon in planning.

It also suggests -- much less strongly -- the existence of much more sophisticated models at $100/month. There are open discussion in planning about whether models plateau, continue improving, singularity, or otherwise. That changes the biases there.

[homarp]

>Fable is $1/month instead of $100/month

will Anthropic (or OpenAI) lowers their price, or increase margin (to justify valuation)

[carterschonwald]

thx for the kind response!

at the very least i have tools that let me easily hit better perf for fancy dense memory layouts, and the same tooling lets me experiment with frankly wildly wacky sparse and structured memory formats. the performance claims at least on the dense side are solid so far!

the sparsity angle is because i want magic in the world. like anyone with a really chunky computer like any of those mac mini pros or serious workstation / server tier compute should be able to train from scratch their one 31b equivalent model in a week or so tops is the goal post i have in mind