[unclesaamm]

My impressions after familiarizing myself with Stan, BUGS, JAGS, and a few Python libraries for Bayesian inference:

Probabilistic programming languages are designed to specify and evaluate statistical models. They usually sit somewhere between declarative and imperative languages. They're basically DSLs in the sense that they come with built-in primitives for various statistical distributions and linkage functions, and the model is sometimes but not always lazy evaluated, but they also often have procedural components like variable assignment and loops, and in some software like Stan, the model has to be specified in a certain order since it is evaluated procedurally.

The languages are also de facto coupled with the engines that run them -- I don't know of any probabilistic languages that have been formalized without a corresponding sampling engine, though there have been cases where a language is "forked", and separate engines built for the same language (like WinBUGS and OpenBUGS).

[marmaduke]

I think Stan internally decouples the model from the algorithm. In other words, a model is just an implementation of an interface, providing log probability values (and gradient) for a given proposal. To wit it supports optimization in addition to sampling.