The 2019 ACM SIGKDD Workshop on Causal Discovery


August 5, 2019, Alaska

Keynote speech

Title: Trustworthy Online Controlled Experiments and the Risks of Uncontrolled Observational Studies

Presenter: Ronny Kohavi, Microsoft

Abstract: Randomized controlled experiments are the gold standard for causal discovery. Online (randomized) controlled experiments are used heavily by software companies; for example, at Microsoft we start 100+ such experiments every work day. At such scale, self-service experimentation is critical and it is important to alert experimenters of trust issues. We share several such tests and pitfalls. Establishing causality from uncontrolled observational studies is harder with surprising pitfalls. We look at highly referenced studies whose results were later shown incorrect by follow-on controlled experiments.

Biography: Ronny Kohavi is a Microsoft Technical Fellow and corporate VP of Microsoft's Analysis and Experimentation at Microsoft's Cloud and AI group. He was previously a Distinguished Engineer and Partner Architect at Bing. He joined Microsoft in 2005 and founded the Experimentation Platform team in 2006. Prior to Microsoft, he was the director of data mining and personalization at, and the Vice President of Business Intelligence at Blue Martini Software, which went public in 2000, and later acquired by Red Prairie. Prior to joining Blue Martini, Kohavi managed MineSet project, Silicon Graphics' award-winning product for data mining and visualization. He joined Silicon Graphics after getting a Ph.D. in Machine Learning from Stanford University, where he led the MLC++ project, the Machine Learning library in C++ used in MineSet and at Blue Martini Software. Kohavi received his BA from the Technion, Israel. He was the General Chair for KDD 2004, co-chair of KDD 99's industrial track with Jim Gray, and co-chair of the KDD Cup 2000 with Carla Brodley. He was an invited speaker at the National Academy of Engineering in 2000, a keynote speaker at PAKDD 2001, an invited speaker at KDD 2001's industrial track, a keynote speaker at EC 10 (2010), a keynote speaker at Recsys 2012, a keynote speaker at Emetrics 2014, a keynote speaker at KDD 2015, and and a keynote speaker at Conversion Hotel 2017. He was an invited speaker at all five MIT Code conferences (Conference On Digital Experimentation) in 2014, 2015, 2016, 2017, 2018. His papers have over 42,000 citations and three of his papers are in the top 1,000 most-cited papers in Computer Science. In 2016, he was named the 5th most influential scholar in AI and the 26th most influential scholar in Machine Learning. He is currently co-authoring a book (due Q4 2019) titled Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing with Diane Tang and Ya Xu.