The article recommends RF for tabular data because it is easier. In general I agree, but newer tools are making NN for tabular data as easy as can be... see, for example, fastai https://docs.fast.ai/tabular.html
The biggest reason to use RFs is that with sufficient trees it's basically impossible to overfit your data. You also don't need to spend days optimizing your hyperparameters. Hence, if you need a quick model where time is tantamount, and you want to err in the side of caution, I feel like an RF is the best choice.
right, it could be simple multicollinearity, or more complex relationships. Because RF is such a good first-try model, I often want to use it on feature sets I haven't carefully pruned, which can be dangerous if you're measuring the same underlying thing in multiple ways.
Since it seems you know a bit of data science, may I ask you a quick question?
In my line of research I am frequently trying to use high dimensional data, but with few examples (<100 per class). Thus methods like SVM are used. I've been thinking about how I might leverage my sample to artificially simulate new training examples via pairwise warping of images within each class, with the assumption that informative features will be preserved with warping.The training examples within class are already quite variable, so I don't think a little increase in redundancy will hurt me much..but I am not sure.
Without knowing more concretely, do you have thoughts on such a strategy?
Data are 3D brain images and classes are disorder groups.
I would have assumed the opposite, with enough trees you are guaranteed to overfit your data? Boosting increases the VC dimension of the aggregate model, which makes it more prone to overfitting.
But we are merging hundreds of trees each of which has been handicapped by removal of multiple features and a fraction of the data. Sounds to me like overfitting is not easy (no single data point or feature contributes to every tree so it can't be represented all the time).
False claims as they maybe, these are claims I've seen in at least two of the most commonly studied statistical learning text books, so given that it makes sense and that it's in the text books, it seems reasonably not false to me. Someone else posted that if too many features or data points are very similar then it will overfit, and that totally makes sense. Whatever you say doesnt. Clarification would be useful.
Nope, RF works very different to boosting. RF trees give unbiased fits, but they're high variance. Bootstrapping is used to reduce the variance of the parallel tree fits.
Boosting is sequential, and relies on early stopping to control the magnitude of bias.
Individual trees are high variance. The random forest itself is an ensemble of many trees - a "forest" if we've being cute. Each tree in the forest is randomized training on a bootstrap sample. This is sometimes called "bagging", a portmanteau of "bootstrap" and "aggregating." Each tree may be further randomized by selecting a different subset of dimensions to consider each time we split a node. The end result is that each tree uses very different rules to make its prediction. When all of these predictions are combined (by voting for classification, or by averaging for regression) error due to overfitting tends to cancel out, while signal due to the same true pattern being discovered independently by many trees is amplified. Thus we can continue to add trees indefinitely without worrying about over-fitting. Other hyperparameters of random forest, such as max tree depth or the minimum number of samples in a node necessary for splitting, can result in overfitting or underfitting so need to be tuned. However, because forest will eventually fit the data set even if we use so-called "stump" learners (max tree depth=1) we can choose very conservative parameters like max-depth=3 which makes trees less likely to overfit. And if they underfit, well, that's not a problem, the ensemble will take care of that. The number of trees in a forest can be cranked up as high as we want without worrying about overfitting; the only downside is that training takes longer, the model takes more space on disk and in memory, and predictions take longer to run.
Yes that’s exactly overfitting the bootstrapped samples, thus the high variance.
The « variance » won’t just magically vanish as you average things out[1], you need to change the scale and check out the asymptotic law of your estimator (CLT, Kolmogorov-Smirnov… etc.) and confront it to your data.
[1] the variance of the estimator itself vanishes thanks to LLN (in case of convergence), but that’s not actually the quantity of interest
Edit: don't get me wrong, I'm not saying that RFs are good or bad, just reacting to the bias/variance thing.
Based on the downvotes, it seems people think reducing variance is the same as reducing overfitting.
Think of the bias/variance tradeoff as a spotlight, and we are shining the spotlight on a bunch of cats, who reflect back the spotlight when their eyes are open. Eyes are open or closed randomly. Cat eyes are either green or brown. We want to know the distribution of cat eyes in parts of the population, which in general is an even 50/50 split. We determine the distribution in a certain location by taking the average of the eyes we see.
If variance is large, then the spotlight is very large, and we don't learn anything because we just average the entire population.
If the spotlight is small, then we can learn something, but only if there are enough samples in the region we shine the light.
So, what if we start with a large spotlight, and then when we see a region with a large number of open eyes of one color, we narrow the light down to just that region? Won't that allow us to avoid overfitting, while maximizing our ability to learn?
It unfortunately does not, because with a large enough population that is evenly distributed, there will always be pockets that exhibit what appear to be a pattern, but is just an accident of which cats happened to open their eyes.
This scenario of starting with the spotlight large and then zooming into a patterned region is the same as reducing variance with the training data. With a large enough dataset it is always possible to find these accidental patterns and then zoom into them by reducing variance.
Sometimes, but even simple trees are high variance given that they're estimated using greedy algorithms rather than some more global optimization. Overfitting in RF does not occur as a function of the number of trees.
fast.ai has some pretty good defaults. The main HP to choose in their tabular learner are the neurons in the linear layers and the number of embeddings for each category.