[zengid]

I'm quite familiar with deep learning because it's so highly hyped up, but am less familiar with Probabilistic programming and Bayesian methods. So, I have a general question: is anyone using Probabilistic Programming in industry? Have people ditched it for DNNs? Are people taking hybrid approaches to try and mix the two?

[ssivark]

Can’t answer for the whole industry. The two methods have very complementary strengths and weaknesses—so which one you apply will depend on the constraints of the domain (eg: large amounts of training data vs intelligent priors). If you’re lucky, and the situation enables it, you could build out both and ensemble them and hope to get the best of both worlds.

I still think the software stack for probabilistic programming has a ways to go before it becomes as easy to use as a NN using PyTorch, but it should get there in the near future. I’m personally very very excited about the probabilistic programming approach — conceptually it’s a very smooth segue from structured numerical algorithms, and allows you to really exploit problem structure if you have good domain understanding.

For me, it helps organize a lot of well-known algorithms as special cases of a general framework—which is worthwhile in itself. If I can code in the generic framework, and have the compiler generate the appropriate (optimized) special case algorithm (as one hopes), that’s icing on the cake.

[zengid]

Thanks for explaining. That seems to be the neat thing that I'm picking up on about what Microsoft Research people keep calling "Model Based Machine Learning" [0], in that you construct a model based on your assumptions about the problem, and by implementing it, the compiler can fit it to an appropriate algorithm.

[0] https://www.youtube.com/watch?v=zKUFSKRjTIo and also https://github.com/dotnet/infer

[ssivark]

Yup! Their WIP book is a fantastic introductory read: http://www.mbmlbook.com/ (Needs only high school background)

Helps gain very nice and concrete intuition, before getting lost in math or code.

[tr352]

> For me, it helps organize a lot of well-known algorithms as special cases of a general framework—which is worthwhile in itself.

Which well known algorithms do you have in mind here?

[ssivark]

For a flavor, see Tom Minka’s recent talk/slides “From automatic differentiation to message passing”.

Difficult to give a quick answer. I’m also not aware of any good resources where this is spelled out. If you’re seriously interested, feel free to hit me up for a deeper discussion.

[tr352]

Thanks I’ll check that out. My own understanding is that pretty much any probabilistic graphical model can be constructed as a probabilistic program, combined with pretty much any mode of inference. How such programs compare to specialized algorithms in terms of efficiency is not clear to me. I’m asking because my understanding is based mostly on theory and I’d like to learn more about probabilistic programming in practice.

[rsrsrs86]

I use both, and for very different problems.