[vkaku]

I did some research about scaling laws and what's unsolved there.

This project was built on structural sparsity, do away with any requirements to train models on largely outdated corpus of data, and only train it on reasoning logic. Because if models can use tools well and retrieve everything else, it greatly would reduce the need for large language models.

Further, when generating traces, I ran a benchmark that showed that on grounded data, n-grams and simple retrieval outperformed transformers on efficiency (See https://github.com/guilt/Transformers) and wanted to try scaling this up to much larger traces.

It worked quite well. Next improvement would be to run this with a BERT model and add more do-not-know traces and the Claude style neurotic self-doubt/self-verify traces and test it out a bit more.

If the entropy required to route people to the relevant answers is very low, full dense attention is a waste of compute. That's the theory being pressure tested here. The more I do this exercise, the more I am getting convinced that smaller models are good enough.