Ben Letham
Core Data Science, Facebook.

Ben Letham I am a data scientist in the Core Data Science group at Facebook. I build statistics tools, develop new ways of analyzing data, and run large online experiments. Prior to this I finished my PhD in operations research at MIT. I can be contacted at
News from the last year:

- February 2017: The Prophet time series forecasting package that I have been working on at Facebook has been open sourced. It has both R and Python versions and can be installed directly from CRAN and PyPI. (blog post, site).

- February 2017: I spoke about adaptive field experiments in the CS department seminar series at the Naval Postgraduate School.

- November 2016: I spoke about our recent work on adaptive field experiments at the MIDAS Annual Symposium, at the Michigan Institute for Data Science.

- October 2016: We presented some recent work on Bayesian optimization and field experiments at the CODE@MIT conference.

- September 2016: I extracted commonly repeated phrases from Hillary Clinton and Donald Trump speeches to create Bingo cards in preparation for the 2016 presidential debates (here).

- June 2016: I made a short stop-motion animation describing our KDD paper. (Youtube).

- June 2016: Our paper on measuring and reducing prediction uncertainty in the context of fitting nonlinear dynamical systems to systems biology data was accepted at Chaos. (Chaos).

- May 2016: Our paper on estimating demand and substitution from retail transaction data in the presence of stockouts was accepted to the research track of KDD. See you in San Francisco! (pdf).

- May 2016: I wrote a post analyzing the balance between talking and fighting in Star Wars lightsaber duels (here).

- March 2016: I provided computational expertise for a paper on the dynamics of HIV inhibition by interferon which was published in PLoS ONE (pdf).

Nonlinear dynamics and computational biology

Prediction uncertainty and optimal experimental design

During the work below on modeling the immune response to HIV infection, we ran into two questions for which we couldn't find a satisfactory answer. The first was to determine the extent to which the collected data constrained the model predictions (prediction uncertainty). The second was to figure out what additional experiments would be most helpful for reducing that uncertainty (optimal experimental design). While eating a hamburger one weekend, I had an idea for how both of these questions might be answered by a particular optimization problem. After hashing out the details, running some experiments, and doing some theoretical work, we wrote it up in this paper.

Benjamin Letham, Portia A. Letham, Cynthia Rudin, Edward P. Browne (2016) Prediction uncertainty and optimal experimental design for learning dynamical systems. Chaos 26: 063110. (Chaos) (arXiv)

The immune response to HIV infection

I provided computational and statistical expertise for this study of the dynamics of HIV inhibition by interferon.

Edward P. Browne, Benjamin Letham, Cynthia Rudin (2016) A computational model of inhibition of HIV-1 by interferon-alpha. PLoS ONE 11(3): e0152316. (pdf) (PLOS)

Learning from sales transaction data

Estimating arrivals and demand in the presence of stockouts

Suppose we wish to use sales transaction data to estimate demand for a collection of substitutable products. When an item goes out of stock, these data no longer reflect the original demand, since some customers leave with no purchase while others substitute alternative products for the one that was out of stock. We developed a Bayesian hierarchical model for simultaneously estimating the customer arrival rate and primary demand, which we use on data from a local bakery.

Benjamin Letham, Lydia M. Letham, and Cynthia Rudin (2016) Bayesian inference of arrival rate and substitution behavior from sales transaction data with stockouts. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16. (pdf) (Video)

Bundle pricing from retail transaction data

Bundle offers are often a low-risk way for retailers to increase their profits, if they are priced correctly. We show how historical sales data for the individual items in the bundle can be leveraged to predict how profits will be affected by introducing a bundle at a particular price. We use a copula model that considers the correlations in item demands, and requires far less stringent assumptions than past work in bundling. This work was done during my summer internship at IBM Research.

Benjamin Letham, Wei Sun, and Anshul Sheopuri (2014) Latent variable copula inference for bundle pricing from retail transaction data. In: Proceedings of the 31st International Conference on Machine Learning, ICML '14, pages 217-225. (pdf) (errata)

Predictions from rules

Learning a decision list to predict stroke risk

A decision list is a particular way of organizing association rules to form a classifier which can be both powerful and interpretable. We developed a Bayesian approach for fitting decision lists and used it to create a simple system for predicting stroke risk in atrial fibrillation patients (a la CHADS2).

Benjamin Letham, Cynthia Rudin, Tyler H. McCormick, and David Madigan (2015) Interpretable classifiers using rules and Bayesian analysis: building a better stroke prediction model. Annals of Applied Statistics 9(3): 1350-1371. (pdf) (Python code)

Sequential event prediction

Here we discuss using techniques from supervised ranking to make predictions on data that are sequentially revealed, using a model that is a linear combination of rules. It makes predictions using only the set of past events, not their order, and is demonstrated with email recipient recommendation, medical symptom prediction, and an e-commerce recommender system.

Benjamin Letham, Cynthia Rudin, and David Madigan (2013) Sequential event prediction. Machine Learning 93: 357-380. (pdf)

Association rules, collaborative filtering, and recommender systems

The ECML PKDD Discovery Challenge 2013 was to develop a recommender system for recommending given names to parents looking for the perfect baby name at For the competition, I developed a recommender system based on association rules, combined with ideas from user-based collaborative filtering. Despite its simplicity and a lack of feature engineering, I won 3rd place for the offline challenge. A major advantage of my approach was that it was fast enough that recommendations could be made entirely online, and I won 2nd place in an online challenge, where my recommender system was actually implemented into the nameling website.

Benjamin Letham (2013) Similarity-weighted association rules for a name recommender system. In: Proceedings of the ECML PKDD 2013 Discovery Challenge Workshop. (pdf)

Learning theory and association rules

We developed a learning theory framework for using association rules for classification and sequential event prediction. We use both VC bounds and results from algorithmic stability. The work was first published at COLT, and then an extended version at JMLR.

Cynthia Rudin, Benjamin Letham, Ansaf Salleb-Aouissi, Eugene Kogan and David Madigan (2011) Sequential event prediction with association rules. In: Proceedings of the 24th Annual Conference on Learning Theory, COLT '11, pages 615-634. (pdf)
Cynthia Rudin, Benjamin Letham, David Madigan (2013) Learning theory analysis for association rules and sequential event prediction. Journal of Machine Learning Research 14: 3385-3436. (pdf)

Information retrieval

Growing a list using the Internet

There are many situations in which important information is scattered across the Internet in many incomplete lists. Given a few seed examples of what we're interested in, can we find and compile all of these incomplete lists of related items to grow a complete list in an unsupervised way? We developed an algorithm for doing this and which works well enough to be useful on a wide range of list growing problems. This work was presented at ECML PKDD 2013.

Benjamin Letham, Cynthia Rudin, and Katherine Heller (2013) Growing a list. Data Mining and Knowledge Discovery 27: 372-395. (pdf) (supplement)

This project received some attention in the media, including a segment about the project on 89.7 WGBH Boston Public Radio. A recording of the live interview is here.


Cross-sensory interactions

While working at The Martinos Center for Biomedical Imaging at Mass. General Hospital I was involved in a study of cross-sensory interactions (auditory stimuli activate visual cortex and vice versa) using MEG and fMRI.

Tommi Raij, Jyrki Ahveninen, Fa-Hsuan Lin, Thomas Witzel, Iiro P. Jaaskelainen, Benjamin Letham, Emily Israeli, Cherif Sahyoun, Christos Vasios, Steven Stufflebeam, Matti Hamalainen (2010) Onset timing of cross-sensory activations and multisensory interactions in auditory and visual sensory cortices. European Journal of Neuroscience, 31(10): 1772-1782. (pdf)

The difficulties of measuring the timing of weak cross-sensory interactions led to this work:

Benjamin Letham and Tommi Raij (2011) Statistically robust measurement of evoked response onset latencies. Journal of Neuroscience Methods, 194(2): 374-379. (pdf)

Data science posts

Debate bingo: 2016 presidential edition

Presidential candidates tend to repeat themselves, and our two for 2016 are no exception. I extracted some of their most commonly used phrases and used them to create a bingo card generator, for use during the debates.

The best part of a lightsaber duel is the talking

Why are the original Star Wars lightsaber duels so much better than those from the Prequels? It's not the simple fight choreography, it's the talking. In this post I quantify the balance between talking and fighting in Star Wars lightsaber duels and show that there is a clear difference between the Original Trilogy and the Prequels. Unfortunately, The Force Awakens is a typical Prequels duel.

Was 2015 Boston's worst winter yet?

When I finally had some time after spending the entire month of February shoveling snow, I did an analysis of snowfall data to determine exactly how unusual of a winter it was. Although 2015 beat the all-time total snowfall record by only a couple inches, a deeper look at the data revealed that this was by far the worst winter in recorded history.

Technical expositions

On NP-completeness

The theory of NP-completeness has its roots in a foundational result by Cook, who showed that Boolean satisfiability (SAT) is NP-complete and thus unlikely to admit an efficient solution. In this short paper I prove an analogous result using binary integer programming in the place of SAT. The proof is notationally cleaner and more straightforward to those for whom the language of integer programming is more natural than that of SAT.

Benjamin Letham (2011) An integer programming proof of Cook's theorem of NP-completeness. (pdf)

On Bayesian Analysis

In this work I provide a somewhat rigorous yet simultaneously informal introduction to Bayesian analysis. It covers everything from the theory of posterior asymptotics to practical considerations of MCMC sampling. It assumes the reader is comfortable with basic probability and some mathematical rigor.

Benjamin Letham (2012) An overview of Bayesian analysis. (pdf)

About me

When I'm not working on research, I enjoy sailing, traveling, playing the violin, and programming. I developed (and continue to develop) the free, open source race timing software fsTimer, which has been used for timing races around the world.

My Erdös number is 4.