[ben_w]

Wall-clock or subjective time?

I think it would take a human about 2.6 million (waking) years to actually read Common Crawl[0]; though obviously faster if they simply absorb token streams as direct sensory input.

The strength of computers is that transistors are (literally) faster than synapses to the degree to which marathon runners are faster than continental drift; the weakness is they need to, too — current generation AI is only able to be this good due to this advantage allowing it to read far more than any human.

How much this difference matters depends on the use-case: if AI were as good at learning as we are, Tesla's FSD would be level 5 autonomy years ago already, even with just optical input.

[0] April 2024: 386 TiB; assuming 9.83 bits per word and 250 w.p.m: https://www.wolframalpha.com/input?i=386+TiB+%2F+9.83+bits+p...

[TeMPOraL]

Subjective time doesn't really matter unless something is experiencing it. It could be 2.6 million years, but if the wall-clock time is half a year, then great - we've managed to brute-force some degree of intelligence in half a year! And we're at the beginning of this journey; there surely are many things to optimize that will decrease both wall-clock and subjective training time.

As the saying goes - "make it work, make it right, make it fast".