The State of Probabilistic Programming

I&#x27;ve tried to use probabilistic programming for building a model of real data, and it seems that there is a long way to go before it&#x27;s practical and fast.<p>On one hand, there are the Monte Carlo-based methods that will support modeling almost any distributions, but are slow to use for large amounts of data.<p>On the other hand, there are interesting cases like Infer.NET that use a completely different technique (approximate, deterministic inference) but are brittle for many real-world use cases.<p>Then, there is the general issue that one has to be familiar with probabilistic models and the inner workings of the inference algorithms to have any hope of debugging the inevitable errors and convergence issues that arise. That seems to realistically require a machine learning or statistics PhD and the population of those is very small.

What&#x27;s interesting about most complaints of these systems is people talk about their poor performance or scalability? That is usually more a consequence of using MCMC or other inference algorithm than the language itself.<p>MCMC is a very slow inference algorithm. Its primary advantage was that for well-known models it could be coded up much more simply than a fancier inference technique. When you consider variational methods and newer streaming methods based on things like Assumed Density Filtering you can get really great scalable performance. The point of probabilistic programming is write inference algorithms once for a large class of models and be done. So the advantage of using a fancier method is amplified.<p>This means paradoxically probabilistic programming should eventually be faster than existing methods rather than slower, since you can reuse these fancier inference methods for new models. This is a very active field so this progress is only starting to be appear in the existing systems.

DrBayes (<a href="https:&#x2F;&#x2F;github.com&#x2F;ntoronto&#x2F;drbayes" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ntoronto&#x2F;drbayes</a>) is another interesting probabilistic programming language. It implements measure-theoretic probability in Racket, so it&#x27;s not limited to what you can express with probability density functions and standard distributions -- you can condition a random variable on any measurable subset of the probability space, for example, so you could condition Y on 0 &lt; X &lt; 3, which wouldn&#x27;t be possible to evaluate with density functions. Or you can construct random variables which have no density function.<p>I&#x27;m not sure this has great practical use, but it&#x27;s a very interesting system.

This strikes me as a perfect application for quantum computers---but I&#x27;m just an amateur, so I&#x27;d love to hear an expert opinion. My understanding is this 1982 talk by Feynman [1] more or less launched the study of quantum computers, and it&#x27;s all about how they can carry a probabilistic value through their computations rather than a definite one. And one of the lessons from that paper (if I&#x27;m reading &amp; remembering right---maybe it&#x27;s another paper) is that simulating quantum behavior with a non-quantum computer turns a lot of polynomial-time problems into exponential-time problems, so that having a real quantum computer would be very helpful for solving the performance &amp; scalability issues described in the OP. Thoughts?<p>[1] <a href="http:&#x2F;&#x2F;www.cs.berkeley.edu&#x2F;~christos&#x2F;classics&#x2F;Feynman.pdf" rel="nofollow">http:&#x2F;&#x2F;www.cs.berkeley.edu&#x2F;~christos&#x2F;classics&#x2F;Feynman.pdf</a>

It seems Markov Logic nets can scale to really big problems:<p>* <a href="https:&#x2F;&#x2F;code.google.com&#x2F;p&#x2F;rockit&#x2F;" rel="nofollow">https:&#x2F;&#x2F;code.google.com&#x2F;p&#x2F;rockit&#x2F;</a><p>* <a href="http:&#x2F;&#x2F;i.stanford.edu&#x2F;hazy&#x2F;hazy&#x2F;tuffy&#x2F;" rel="nofollow">http:&#x2F;&#x2F;i.stanford.edu&#x2F;hazy&#x2F;hazy&#x2F;tuffy&#x2F;</a><p>But they seem to be getting less and less coverage in probabilistic programming. Perhaps they are considered too inexpressive?

Probabilistic programming in Common Lisp:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;nikodemus&#x2F;screamer" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;nikodemus&#x2F;screamer</a>

<i>&quot;Having been rescued from the clutches of object-orientation by early exposure to Mathematica&quot;</i><p>I kind of laughed at this line, because this seems like a situation where the author came to the correct conclusion for all the wrong reasons.<p>Maybe things have changed, but the Mathematica programming language never struck me as useful for anything over a couple thousand lines.<p>edit: this is completely beside the purpose and message of this article, but it&#x27;s worth noting that modularity is a pretty significant design criteria that none of these languages have a particularly strong story about.

I wonder if kids 30y from now look upon this as the new Prolog.