[markovbling]

I threw all of the above plus CNNs at MNIST problem and boosted decision trees outperformed CNNs.

Granted if I tuned both perfectly, CNNs probably would have outperformed but with defaults and a small amount of parameter search, boosting worked best.

[akhilcacharya]

That's pretty interesting, why are boosted decision trees so effective? I've heard the same meme applied to kaggle competitions (everything is just a way to shove data into xgboost, etc)

[romaniv]

The grandparent post contains a hint of something I've already heard in lectures and from ML practicioners: boosting and random forests are more resilient to improper tuning (to put it another way, they are more universal and work well "out of the box").

Which, BTW, makes them more appealing to me personally. In many real-life cases extra few percent of accuracy matters very little, but ability to just apply something to a problem without much fuss matters a lot.