[grandiego]

Beginner here. A takeaway I got from the Andrew Ng's Coursera course (specifically for neural networks) is that adding more neurons and layers than the "minimum needed" is usually okay (that is, no risk of overfitting when considering reasonable regularization terms.) Sadly, there is no rule for that minimum, so you must do trial and error; on the other side, carelessly extending the network will be inefficient and eventually slow. For the activation functions, the output layer's is mostly determined by the problem being tackled, and for the inner layers you usually start with ReLU and then try some of the common variants using some heuristics (again related to the current problem.) Of course you should consider other successful models for similar problems as your starting point.