Hacker News new | ask | show | jobs
by gwern 169 days ago
Could you explain why you think that? I'm looking at the lottery ticket section and it seems like he doesn't disown it; the reason he gives, via Abhinav, for not pursuing it at his commercial job is just that that kind of sparsity is not hardware friendly (except with Cerebras). "It doesn't provide a speedup for normal commercial workloads on normal commercial GPUs and that's why I'm not following it up at my commercial job and don't want to talk about it" seems pretty far from "disowning the lottery ticket hypothesis [as wrong or false]".
2 comments
oofbey 169 days ago
I think that was pretty clear even when this paper came out - even if you could find these sub networks they wouldn’t be faster on real hardware. Never thought much of this paper, but it sure did get a lot of people excited.
link
sailingparrot 169 days ago
It was exciting because of what it means regarding how a model learns, regardless on whether or not its commercially applicable.
link
gwern 169 days ago
(Cerebras is real hardware.)
link
oofbey 169 days ago
It is real in that it exists. It is not real in the sense that almost nobody has access to them. Unless you work at one of the handful of organizations with their hardware, it’s not a practical reality.
link
aaronblohowiak 168 days ago
how long will that be the case?
link
oofbey 168 days ago
They have a strange business model. Their chips are massive. So they necessarily only sell them to large customers. Also because of the way they’re built (entire wafer is a single chip) no two chips will be the same. Normally imperfections in the manufacturing result in some parts of the wafer being rejected and other binned as fast or slow chips. If you use the whole wafer you get what you get. So it’s necessarily a strange platform to work with - every device is slightly different.
link
IshKebab 168 days ago
At least for the foreseeable future (next 50 years say).
link
laughingcurve 168 days ago
i saw how it nerdsniped an extremely capable faculty member
link
swyx 168 days ago
he pretty much always says it offline haha but i maay have mixed it up with the subsequent convo we had at neurips https://www.latent.space/p/neurips-2023-startups
link