[civilized]

You don't need neural networks to do polynomial regression. Polynomial regression, perhaps surprisingly, can be implemented using only (multivariable) linear regression. You just include powers of your predictor x as terms in the regression formula:

  y = a + bx + cx^2 + dx^3 + ...

The resulting model is linear, even though there are powers of x in your formula. Because x and y are known from the data. They're not what you're solving for, you're solving for the unknown coefficients (a, b, c, d...). This gives you a linear system of equations in those unknown coefficients, which can be solved using standard linear least squares methods.

So fitting polynomials is easy. The problem is that it's not that useful. Deep learning has to solve much harder problems to get to a useful model.