[vinn124]

> Usually you throw everything and see what sticks.

most practitioners start with the simplest possible learner, then gradually, and thoughtfully, increase model complexity while paying attention to bias/variance. this is far from a "kitchen sink" approach.

[joshvm]

Certainly that's what most sensible practitioners do. I somewhat doubt that most people follow this to the letter every time.

It's a nice theory, and it works intuitively with models where you can ramp up complexity easily (like neural nets). It's less obvious if you have a "simple" problem that might be solved with a number of techniques. In that situation I don't see why you would be criticised for trying say any of SVM, random forest, logistic regression, naive bayes and comparing. Pretty much the only way you can categorically say that your method is better than others is by trying those other methods.

The simple approach actually came up in the iceberg challenge. The winning team won because they bothered to plot the data. It turned out that the satellite incidence angle was sufficient to segment a large number of icebergs with basically 100% reliability. So they simply thresholded that angle range and trained a bunch of models to solve the more complicated case when there might be a ship.

[stevepike]

This was one of the things Andrew Ng really hammers on in the coursera course. This alongside separating out the training set from the cross validation set for tuning parameters went a long way to dispel some of the "magic" in how you iterate towards a sensible model.