[dreamlessfate]

My journey with Machine Learning so far:

:D Oh, nonlinear equations! This is something I know a lot about.

:) I think I see...so they use nonlinear equations in the activation function. This helps to create divergence, or sensitive dependence on initial conditions.

:| Wait it's a sigmoid function?? Wtf that's boring.

:( They're just trying to min/max a data set, and figure out probability as it relates to that min/max. But that sucks, because most of the interesting phenomenon in nature exists in BETWEEN zero and one! All the fun, cool stuff happens in the middle! You can't reduce it down to a probability, there's no way that's going to do a good job describing anything!

[PaulHoule]

If you don't like sigmoids you might like ReLU

https://iq.opengenus.org/relu-activation/

For many learning tasks ReLU performs better than sigmoid.

My favorite use of sigmoid functions is

https://scikit-learn.org/stable/modules/calibration.html

where they are very good at turning arbitrary scores (say from a full-text search engine) into probabilities. For IBM Watson they tried a lot of things and found logistic functions dominated. Turning scores into probabilities was how Watson could decide if and when hitting the button would help win the game.

The big trouble with probabilities is that, potentially, every event is contingent on every other event and the joint probability distribution of all possible inputs and outputs is a huge dimensional space. In principle you could learn any function by sampling the joint probability distribution exhaustively but practically you can't get that much data. The miracle of machine learning is that the methods we use can guess at the joint probability distribution of inputs and outputs with only a limited sample.

If there was one great unsolved problem of the "old AI" namely expert systems it was doing logical reasoning over probability functions. It's not good enough to estimate that A has a 80% probability of being true, in general you need to estimate what the probability of A is if B is true and C is false. If the problem cooperates you can use half-baked methods to reason about uncertainty (like the MYCIN medical diagnosis program) but general and correct methods are elusive.

[dreamlessfate]

Yeah I've come across ReLU, and Leaky ReLU, tanh, Maxout, ELU.

I guess I really, really need someone to explain to me like I'm a total idiot why I'd want to use a 2D function on data that might be (and probably is) multidimensional and interdependent.

>"The big trouble with probabilities is that, potentially, every event is contingent on every other event and the joint probability distribution of all possible inputs and outputs is a huge dimensional space."

Maybe! But this seems like a way more interesting challenge, with the potential to handle way more dimensions, and be way closer to "reality" and "truth", whatever those definitions really are or really mean.

>" It's not good enough to estimate that A has a 80% probability of being true, in general you need to estimate what the probability of A is if B is true and C is false."

Doesn't that still seem a little lame though? And massively error prone?

Filtering problems through a stochastic process of probabilities kinda feels like trying to rig a pachinko machine.

[PaulHoule]

Neural networks compose complex functions out of simple functions. A three layer network can approximate any function w/ multiple inputs and outputs. Now, a network has to be able to learn that approximation with a reasonable amount of time and computation and all of the innovations in network architectures are about making networks that actually do learn.

As for reasoning with uncertainty people do a lot of that. There was that time I went to my doc because I had rashes on on my arms and neck and my doc (e.g. like MYCIN) thought some of them looked allergic and some looked bacterial. Instead of overthinking it, he wrote me scripts for two different creams and I tried both.

My take is that rapid progress in neural networks is not about people having a lot of understanding or a general theory or even that the average work is of very good quality, but an awful lot of people are working on it and throwing a lot of resources at it. People are discovering things empirically that work.

One can consider many counterfactuals which would have sped up development a lot. If the Japanese had developed a 5th generation computer project around accelerating neural networks instead of accelerating logical inference, if people had tried ReLU in the 1980s, etc.