[vkaku]

Very true. In fact, there's a lot more under the surface that will come to light once more efficient tiny models come out.

P.S. Building something that proves that you don't need those many params even.

https://github.com/guilt/tinytot

[teodosin]

The premise of your project seems compelling. Is it novel or building on existing work? Anything one could read or watch to get introduced to that area of research?

[vkaku]

You would already be able to see RSI and loops being a thing.

So there's going to be more and more research coming out telling you what size of models are really needed to solve a specific task. When you combine that with RSI, the next frontier would be to get this model as close to the real world with instant data and instant reasoning, no fine tuning at all. And see how this one could iterate itself.

I'll also be trying out more tiny models and seeing if they can call each other, and be as effective; You will also be seeing me trying this, in addition to what I'd already mentioned in the other reply.

[vkaku]

I did some research about scaling laws and what's unsolved there.

This project was built on structural sparsity, do away with any requirements to train models on largely outdated corpus of data, and only train it on reasoning logic. Because if models can use tools well and retrieve everything else, it greatly would reduce the need for large language models.

Further, when generating traces, I ran a benchmark that showed that on grounded data, n-grams and simple retrieval outperformed transformers on efficiency (See https://github.com/guilt/Transformers) and wanted to try scaling this up to much larger traces.

It worked quite well. Next improvement would be to run this with a BERT model and add more do-not-know traces and the Claude style neurotic self-doubt/self-verify traces and test it out a bit more.

If the entropy required to route people to the relevant answers is very low, full dense attention is a waste of compute. That's the theory being pressure tested here. The more I do this exercise, the more I am getting convinced that smaller models are good enough.

[dineshdb]

I took a look at the codebase and found that this is just pattern matching. There is no any novelty here. It looks good for knowledge-base retrieval system, but other than that, this is just nothing more than a pattern-matcher.

Would love to be proven wrong with future updates.

[vkaku]

Sure. Like throwing money at problems is novel.

Why don't you go and actually build something novel? People need to understand better how current models work, and if retrieval indeed solves 99% of the problems people care about, then those other models aren't doing anything super novel either.

I'm not arguing here: The code is out there, data is out there. Do better if you know something actually better. Thanks!