I spent two years[0] designing, building and maintaining a system which used contextual multi-armed bandits at large scale. A couple pieces of advice relating to this post and this subject:<p>1. Thompson sampling is great. It's intuitive and computationally tractable. The literature is full of other strategies, specifically semi-uniform strategies, but I strongly recommend using Thompson sampling if it works for your problem.<p>2. This is broadly true about ML, but for contextual bandits, most of the engineering work will probably be the feature engineering, not algorithm implementation. Plan accordingly. Choosing the right inputs in the first place makes a big difference. The hashing trick (a la sklearn's dictvectorizer) can make a huge difference.<p>3. It can be difficult to obtain organizational alignment on the intention of using reinforcement learning. Tell stakeholders early and often that you're using bandit algos to produce some kind of outcome — say, clicks or conversions — and not to do science which will uncover deep truths.<p>[0] along with an excellent data scientist and a team of excellent engineers, of course :)
Though mentioned it the article, I'll add it here for posterity: <a href="https://web.stanford.edu/~bvr/pubs/TS_Tutorial.pdf" rel="nofollow">https://web.stanford.edu/~bvr/pubs/TS_Tutorial.pdf</a><p>Great tutorial on Thompson sampling.
I would strongly recommend the post he cited. It is the same style but features interactive visualizations: <a href="https://dataorigami.net/blogs/napkin-folding/79031811-multi-armed-bandits" rel="nofollow">https://dataorigami.net/blogs/napkin-folding/79031811-multi-...</a><p>I implemented something like this for my company and found the latter article quite helpful in explaining the concept to people who understood the basics of probability but not programming.
I want to recommend the book <a href="http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/" rel="nofollow">http://camdavidsonpilon.github.io/Probabilistic-Programming-...</a>
a nice little book about probabilistic programming in Python.
There was a free Bandits algorithm book discussed on HN about a month ago.<p><a href="https://news.ycombinator.com/item?id=17642564" rel="nofollow">https://news.ycombinator.com/item?id=17642564</a>