[lokimedes]

When I worked at CERN around 2010, Boosted Decision Trees were the most popular classifier, exactly due to the (potential for) explainability along with its power of expression. We had a cultural aversion for neural networks back then, especially if the model was used in physics analysis directly. Times have changed…

[ekjhgkejhgk]

I used to be in physics but theory, not experiment. I have experience at work with decision trees in a different field.

I've always thought that the idea that decision trees are "explainable" is very overstated. The moment that you go past a couple of levels in depth, it becomes an un-interpretable jungle. I've actually done the exercise of inspecting how a 15-depth decision trees makes decision, and I found it impossible to interpret anything.

In a neural network you can also follow the successive matrix multiplications and relu etc through the layers, but you end up not knowing how the decision is made.

Thoughts?

[lokimedes]

I completely agree, as you may infer from my comment. The second multivariate models are relevant we effectively trade explainability for discrimination power. If your decision tree/model needs to be large enough to warrant SGD or similar optimization techniques, it is pretty much a fantasy to ever analyze it formally.

My second job after physics was AI for defense, and boy is the dream of explainable AI alive there.

Honesty anyone who “needs” AI to be understandable by dissection, suffers from control issues :)

[Otterly99]

Do you have any good resource on explainable AI (XAI)? Most books I read on the subject tend to be a bit shallow and not very informative.

[srean]

> Times have changed…

This makes me a little concerned -- the use of parameters rich opaque models in Physics.

Ptolemaic system achieved a far better fit of planetary motion (over the Copernican system) because his was a universal approximator. Epicyclic system is a form of Fourier analysis and hence can fit any smooth periodic motion. But the epicycles were not the right thing to use to work out the causal mechanics, in spite of being a better fit empirically.

In Physics we would want to do more than accurate curve fitting.

[lokimedes]

If you sum up experimental physics into one heuristic it is “avoid fooling yourself with assumptions” - I left physics over a decade ago, but I feel confident that physicists still work hard to understand what they observe and don’t let LLMs have all the fun. If there’s one field of science where the scientists are legitimately allowed to go all the way back to basics, it’s elementary particle physics.

[YeGoblynQueenne]

This guy says physicists now leave most of the work to LLMs:

https://www.youtube.com/watch?v=PctlBxRh0p4

(Original title was "Physicists are surrendering to AI" but it was changed since to something less clickbaity, which I appreciate. I posted it here on the original title: https://news.ycombinator.com/item?id=46859804)

[srean]

In general I would agree. I think it holds true at the highest levels.

What worries me is the noticeable uptick of presentations of the sort -- look ma better fit ... deep neural nets. These are mostly by more junior folks, but not necessarily. I have been in the audience in many.

These and the uptick in research proposals funded by providers of infra for such DNNs. I have been in the audience of many.

A charitable read could be that they just want the money and would do the principled thing.

[lokimedes]

Again, said as someone out of the fray, let’s hope it self-corrects. Physics is a very community driven field, and the young must try new things, and be allowed to, it is part of progress. It is when the seniors surrender the standard of quality they carry, we have trouble. And here, indeed, particle physics can be uniquely vulnerable - given the complexity and economics of the research, it is hard to falsify claims made with new methods if the established researchers cave too easily.

[srean]

Amen.

To support your point of view, I haven't encountered this in particle physics. It's from other branches. I am not a Physicist myself, happened to be in a position to observe funding requests, request for collaborations.

[wodenokoto]

Are boosted decision trees the same as a boosted random forest?

[boccaff]

short answer: No.

longer answer: Random forests use the average of multiple trees that are trained in a way to reduce the correlation between trees (bagging with modified trees). Boosting trains sequentially, with each classifier working on the resulting residuals so far.

I am assuming that you meant boosted decision trees, sometimes gradient boosted decisions trees, as usually one have boosted decision trees. I think xgboost added boosted RF, and you can boost any supervised model, but it is not usual.

[hansvm]

The training process differs, but the resulting model only differs in data rather than code -- you evaluate a bunch of trees and add them up.

For better or for worse (usually for better), boosted decision trees work harder to optimize the tree structure for a given problem. Random forests rely on enough trees being good enough.

Ignoring tree split selection, one technique people sometimes do makes the two techniques more related -- in gradient boosting, once the splits are chosen it's a sparse linear algebra problem to optimize the weights/leaves (iterative if your error is not MSE). That step would unify some part of the training between the two model types.