[doobwa]

I agree these methods still require a fair amount of expert knowledge and intuition in order to make the various choices you mention. On the other hand, Bayesian optimization can prove useful for exploring such a space. A recent paper (http://arxiv.org/pdf/1206.2944.pdf) used Bayesian optimization with GPs to find hyperparameter settings for a deep convolutional network. The resulting hyperparameters gave state of the art performance, beating those chosen by an expert (Krizhevsky, the researcher who recently won ImageNet).

[danger]

Agreed. Here's a previous HN discussion on that topic: http://news.ycombinator.com/item?id=4281630