[jfaucett]

> On the other hand, cost will eventually limit the parallelism side of the trend and physics will limit the chip efficiency side.

Anyone working on chip architecture care to give their opinion on the next 10-20 years in chip design? It would really interest me to know if chip designers think Moore's law will continue, since that is probably going to be a big factor in the timeline for AGI.

[deepnotderp]

Not gonna predict the future 1-2 decades out, since that's a fool's errand, but here's a grab bag of relevant points:

1. Moore's Law is undoubtedly slowing, but in the foreseeable future, it will likely continue. On the other hand, Dennnard Scaling which is already basically dead, will be the crunch you will likely feel more. Exponential transistors aren't too useful if they still consume so much power. To mitigate leakage we moved to FinFETs... Which actually made dynamic power worse.

2. You might be interested to know that data movement (predominantly memory access) costs orders of magnitude more than computation, especially relevant to AI compute which requires large amounts of access. These global wires already suck and don't seem to be getting any better in the foreseeable future.

3. Foundries have already been using (and thus expending) "scaling boosters" to reach their density goals. Most of these are one-time use effects that won't provide significant continuous scaling capability.

[p1esk]

Analog computing has a lot of yet unrealized potential for machine learning algorithms.

However, currently it does not make sense to build a specialized analog chip to run specific type of ML algorithms, because algorithms are still being actively developed. I don't see GPUs being replaced by ASICs any time soon. And before you point to something like Google's TPU, the line between such ASICs and latest GPUs such as V100 is blurred.

[sanxiyn]

I define GPU as something that can efficiently implement DirectX. Hence TPU is not GPU. And I predict ML algorithms will run on non-GPU, soon-ish.

[deepnotderp]

Please explain where analog computation has a benefit over digital that outweighs its numerous disadvantages.

[p1esk]

Wait, aren’t you working on analog chips?

[deepnotderp]

No.

You may have confused me with the Isocline/Mythic guys or a red herring comment. Our approach to deep learning chips is very public and amongst the craziest...A̶n̶d̶ ̶e̶v̶e̶n̶ ̶I̶ ̶w̶o̶u̶l̶d̶n̶'̶t̶ ̶t̶o̶u̶c̶h̶ ̶a̶n̶a̶l̶o̶g̶ ̶c̶o̶m̶p̶u̶t̶a̶t̶i̶o̶n̶

To clarify: I'm always open to opposing evidence, but based on the data at the moment, I believe that analog computing buys you very little.

[p1esk]

I'm sure you know both cons and pros of analog computing. As long as you can significantly improve digital tech every year, keep doing that. But as soon as that stops, or becomes too expensive, analog is the way forward.

[deepnotderp]

Again, what advantage does analog have?

People seem to assume that analog intrinsically consumes less power, which due to bias and leakage currents isn't true in the general case.