Hacker News new | ask | show | jobs
by davidwihl 2310 days ago
Well stated. The author also misses two other critical points: (1) accuracy is a poor measure of quality for non-numeric, classification type problems, (2) increasing model complexity has an asymptote in order to prevent overfitting. You can’t arbitrarily increase the number of weights and expect that the NN will continue to improve.
2 comments
superpermutat0r 2310 days ago
The thing about NN is that increasing the weights does improve the performance. The standard way to get good performance and see if your architecture works is just get the network huge (wide). After you see it works you get it small.

The "common wisdom" of "too many parameters will make you overfit" is most definitely not that important for the way modern NN training works.

link
svantana 2310 days ago
Overfitting shouldn't be an issue for approximation of a known function, where you can generate an arbitrary amount of "training data". Of course you may not have the resources to do so, but that's a whole different tradeoff.
link
bitforger 2310 days ago
Even then, it might not really be an issue.

https://youtu.be/5-Kqb80h9rk

link