Hacker News new | ask | show | jobs
by drdeca 1514 days ago
Do you mean that a network trained to imitate a calculator won’t do so accurately or that there is no combination of weights which would produce the behaviors of a calculator?

Because, with RELU activation, I’m fairly confident that the latter, at least, is possible.

(Where inputs are given using digits (where each digit could be represented with one floating point input), and the output is also represented with digits)

Like, you can implement a lookup table with neural net architecture. That’s not an issue.

And composing a lookup table with itself a number of times lets one do addition, etc.

... ok, I suppose for multiplication you would have to like, use more working space than what would effectively be a convolution, and one might complain that this extra structure of the network is “what is really doing the work”, but, I don’t think it is more complicated than the existing NN architectures?
1 comments
SemanticStrengh 1514 days ago
I am talking about training a neural network to achieve calculations. And yes look-up tables might be fit for addition but not for multiplication. The accuracy would be <90% which is a joke for any serious use.
link
drdeca 1514 days ago
Well, the main issue I see is where to put the n^2 memory (where n is the number of digits) when doing multiplication. (Or, doesn’t need n^2 space, could do it in less, but might need to put more structure into the architecture?)

If the weights are designed, and the network architecture allows something to hold the information needed, then there is really no obstacle to having it get multiplication entirely (not just 90%).

Now, would that be learnable? I’m not so sure, at least with the architecture one would use if designing the weights.

But,

I see no reason a transformer model couldn’t be trained on multiplication-with-work-shown and produce text fitting all of those patterns, and successfully perform multiplication for many digits that way.

And, by “showing all work” I don’t necessarily mean “in a way a person would typically show their work”, but in a easier-for-machine way.

link