Probabilistic programming systems (PPS) define languages that discretize modeling and inference such that any generative model can be easily composed and run with a common inference engine. The main advantage over traditional ML systems in deterministic code (i.e. Python) being concise, modular modeling where the developer doesn't have to write custom inference algorithms for each model/problem. For more info see, for example, [1] and [2].<p>I'm curious though, what applications of PPS are realized in practice? Notably Uber [3] and Google [4] are developing/supporting their own (deep learning focused) PPS, but is it known if/how they're used within these companies? Are the frameworks (Pyro [5] and Edward [6], respectively) used by other companies?<p>[1] Frank Wood (Microsoft) tutorial: https://www.youtube.com/watch?v=Te7A5JEm5UI<p>[2] MIT ProbComp lab's page of resources: http://probcomp.csail.mit.edu/resources/<p>[3] https://eng.uber.com/pyro/<p>[4] https://medium.com/tensorflow/introducing-tensorflow-probability-dca4c304e245<p>[5] http://pyro.ai/<p>[6] http://edwardlib.org/
Improbable [0] is building (and has open-sourced) Keanu, "a general purpose probabilistic programming library built in Java", where Bayesian networks are represented as DAGs. It's feature list includes Probabilistic Programming Operators and Distributions, Auto-differentiation, Inference, Maximum a posteriori, Metropolis Hastings, Hamiltonian Monte Carlo, Sequential Monte Carlo (Particle Filtering), and Support for Kotlin. [1]<p>[0] <a href="https://improbable.io/" rel="nofollow">https://improbable.io/</a><p>[1] <a href="https://github.com/improbable-research/keanu" rel="nofollow">https://github.com/improbable-research/keanu</a>
We use PPLs at Triplebyte for matching software engineers to jobs where we predict they're a strong fit. We recently published <a href="https://triplebyte.com/blog/bayesian-inference-for-hiring-engineers" rel="nofollow">https://triplebyte.com/blog/bayesian-inference-for-hiring-en...</a> which starts to explain our framework, though PPLs would have to be a "part 2" blog post if anyone's interested.
Stan is probably the most mature, stable PPL, and it's an extremely popular tool, although it isn't really deep-learning-adjacent so gets much less hype.<p>It's generally used more for modeling and prediction than for creating "products", if that makes sense. More popular among people with statistics or social science backgrounds than among programmers and computer scientists.<p>If you dig around Stan-related websites you can see various companies and institutions that use it. One I found quite quickly was Metrum Research Group, which does consulting work for the pharmaceutical industry.<p><a href="https://metrumrg.com/" rel="nofollow">https://metrumrg.com/</a>
pymc3 provides this for Python in a way that is very concise and modular (certainly much more concise than tensorflow-probability) -- and it is an open question if TensorFlow might be used to replace Theano as the backend execution engine for the next versions.<p>In particular, pymc3's use of ADVI to automatically transform discrete or boundary random variables into unconstrained continuous random variables and carry out an initialization process with auto-tuned variational Bayes automatically to infer good settings and seed values for NUTS, and then to automatically use an optimized NUTS implementation for the MCMC sampling, is incredibly impressive.<p>For most problems, you use a simple pymc3 context manager and from there on it acts kind of like a mutually recursive let block in some functional languages: you define random and deterministic variables that inter-depend on each other and are defined by their distribution functions, with your observational data indicating which values are used for determining the likelihood portion of the model.<p>After the context manager exits, you can just start drawing samples from the posterior distribution right away.<p>I've used it with great success for several large-scale hierarchical regression problems.
At Semantic Machines [0] we rely heavily on probabilistic programming to build state-of-the-art dialogue systems. In particular, we use a library called PNP (probabilistic neural programming) on top of Dynet to allow us to express structured prediction problems in a simple and elegant form. If there are questions I am happy to elaborate to the extent I can. (Also, we are hiring! My email is jwolfe@.)<p>[0] <a href="http://www.semanticmachines.com/" rel="nofollow">http://www.semanticmachines.com/</a>
Just a quick correction – Frank Wood is not at Microsoft, but at UBC:<p><a href="http://www.cs.ubc.ca/~fwood/index.html" rel="nofollow">http://www.cs.ubc.ca/~fwood/index.html</a><p>Microsoft Research does have multiple excellent researchers working on probabilistic programming. Infer.NET in particular is a highly advanced piece of technology for models in which you would use message passing algorithms to perform inference:<p><a href="http://infernet.azurewebsites.net" rel="nofollow">http://infernet.azurewebsites.net</a>
Given Avi Bryant[1] recently released Ranier from there, I'd guess Stripe is.<p>[1] <a href="https://twitter.com/avibryant" rel="nofollow">https://twitter.com/avibryant</a>
[2] <a href="https://github.com/stripe/rainier" rel="nofollow">https://github.com/stripe/rainier</a>
Anglican <a href="https://probprog.github.io/anglican/" rel="nofollow">https://probprog.github.io/anglican/</a> also based on a prototype by Frank Wood, using Clojure syntax.
Facebook is working on probabilistic programming. Rather than develop it as a library, they're trying to provide language support directly. It was recently discussed at a conference; you could ask Erik Meijer for the details (<a href="https://twitter.com/headinthebox/status/993972303863070720" rel="nofollow">https://twitter.com/headinthebox/status/993972303863070720</a>).
I just made public my Master Thesis project that I completed at the University of Oxford.<p>It is called CPProb and it is a C++ general purpose probabilistic programming library that uses a version of Variational Inference to learn proposals for Importance Sampling.<p>It aims to be usable directly in preexisting C++ codebases.
For the fulfillment of the Master Thesis, I also wrote a tutorial on Particle filters via SMC-like methods, and I described the design choices that one finds when implementing one of these systems.<p>The C++ library with the corresponding Pytorch-based neural network and the tutorial can be found in<p><a href="https://github.com/Lezcano/CPProb" rel="nofollow">https://github.com/Lezcano/CPProb</a><p>and are available under a MIT license.
I've used rJags for R programming language which is quite old and based on Jags for windows but is quite straight forward though Edward, Stan and pymc3 seem to be state of art.<p>My project is quite simple but You can check it via homepage[0] or directly[1]<p>[0]<a href="http://www.vladovukovic.com" rel="nofollow">http://www.vladovukovic.com</a>
[1]<a href="https://bit.ly/2Krtkfi" rel="nofollow">https://bit.ly/2Krtkfi</a>
I designed a system which used probabilistic programming (Stan) to combine various deep learning based feature extractors to predict social behaviours.
I'm working on probabilistic programming right now, and a lot of the papers I'm reading are from Microsoft Research. They have a cool Infer.NET project, and an ecosystem based on it is beginning to form. For example take a look at Tabular, which is an excel addon to do "Bayesian inference for the masses" based on Infer.NET: <a href="https://www.microsoft.com/en-us/research/project/tabular/" rel="nofollow">https://www.microsoft.com/en-us/research/project/tabular/</a><p>Overall though, by freeing developers from writing custom inference algorithms, all the work gets pushed to the language designer/implementer. It is not at all clear to me that one (or even a few) generic inference algorithms will be able to satisfy the needs for different problem domains. So there could be not one general purpose PPL but multiple ones with different problem domains.
We use kind of a tiny in-house PPL for document analysis at <a href="https://scanye.io/" rel="nofollow">https://scanye.io/</a>. We haven't open-sourced it though.