[mullingitover]

Interesting to think about where the cost will go in a few years.

I remember in college intro to CS class back in 1998, where I heard the story of building the first computer that could perform at 1 TFLOPS[1]. It cost $46 million and took up 1600 square feet. Now a $600 Mac Mini will do double that.

[1] https://en.wikipedia.org/wiki/ASCI_Red

[DSingularity]

And on a much more programmable platform - which is just as important in terms of accessibility !

[miohtama]

It is not going to go down much anymore, because the end of Moore's law has been reached as physical limitations become a factor. You cannot scale chips close to 1 atom wide transistors.

[runnerup]

Moore's law isn't dead. Only Dennard's law. See slide 13 here[0]. Moore's law stated that the number of transistors per area will double every n months. That's still happening. Besides, neither Moore's law nor Dennard scaling are even the most critical scaling law to be concerned about...

...that's probably Koomey's law[1], which looks well on track to hold for the rest of our careers. But eventually as computing approaches the Landauer limit[2] it must asymptotically level off as well. Probably starting around year 2050. Then we'll need to actually start "doing more with less" and minimizing the number of computations done for specific tasks. That will begin a very very productive time for custom silicon that is very task-specialized and low-level algorithmic optimization.

[0] Shows that Moore's law (green line) is expected to start leveling off soon, but it has not yet slowed down. It also shows Koomey's law (orange line) holding indefinitely. Fun fact, if Koomey's law holds, we'll have exaflop power in <20W in about 20 years. That's equivalent to a whole OpenAI/DeepMind-worth of power in every smartphone.

0: (Slide 13) https://www.sec.gov/Archives/edgar/data/937966/0001193125212...

1: "The constant rate of doubling of the number of computations per joule of energy dissipated" https://en.wikipedia.org/wiki/Koomey%27s_law

2: "The thermodynamic limit for the minimum amount of energy theoretically necessary to perform an irreversible single-bit operation." https://en.wikipedia.org/wiki/Landauer%27s_principle

[runnerup]

Also even MHz increases have had a bit of a comeback lately, with the fastest mid-2000's Pentium 4's reaching 3.8-4.2GHz and the latest Ryzen 7000's reaching 6GHz.

[Our_Benefactors]

I’ll take this 10 year bet. You really think nvidia is just gonna stop releasing new revisions? “Moores law is dead” is way over-memed, it’s more of an axiom about how computers continually improve than really being about transistor count at this point.

[andromeduck]

Moore's law and more importantly dennard scaling both died in the mid 2000s. Nvidia is in fact successful because of the end of dennard scaling and the shudts do more mission specialized silicon like TPUs, and codec accelerators, inference engines are also a consequence of that.

Nvidia's performance gains in recent years has been about scaling chip size and making more efficient use of each transistor both in terms of power and count than anything else. A large part of that is minimizing how far data physically moves for any given workloads via stuff like HBM, memory compression, and smarter/larger caches.

In fact, Nvidia doesn't even really try to be on the bleeding edge nodes anymore because per transistor costs has been trending up or level on bleeding edge nodes for at least 5 years now.