Adversarial MLOps on Other People's Money: Lessons Learned from Ad Tech

Take-aways

MLOps is much more than alerting on broken feature pipelines: - Correct measurement is critical -- more important than machine learning. Most MLOps are at the wrong level of abstraction because the underlying eventing and measurement aren't sound. Standard, auditable, and accountable measurement comes from ads. - ML is like an evil genie: it tends to give you the least valuable answer that is exactly what you wished for. MLOps is about keeping vigilant that the ML objectives align with business objectives. For example, it's not enough to predict a sale, you need to predict will seeing this item _cause_ an incremental sale. Incrementality (or lack of it) comes from ads. - Design ML to be components in a larger system with stable interfaces. It's not tracible to monitor the entire stack as a black box. You need intermediate ground-truth signals. Ads are designed in this way.

In this episode

Andrew Yates

CEO, PROMOTED.AI INC

Andrew Yates formerly led ads ranking, auction, and marketplace engineering and research teams at Facebook and Pinterest. He specializes in designing billion-dollar content marketplaces that maximize long-term revenue while protecting both seller and user experiences. Andrew has published over a dozen patents in online advertising optimization.

Demetrios Brinkmann

Host

Demetrios is one of the main organizers of the MLOps community and currently resides in a small town outside Frankfurt, Germany. He is an avid traveller who taught English as a second language to see the world and learn about new cultures. Demetrios fell into the Machine Learning Operations world, and since, has interviewed the leading names around MLOps, Data Science, and ML. Since diving into the nitty-gritty of Machine Learning Operations he felt a strong calling to explore the ethical issues surrounding ML. When he is not conducting interviews you can find him making stone stacking with his daughter in the woods or playing the ukulele by the campfire.

Abi Aryan

Host

Abi is a machine learning engineer and a live streamer. Over the past six years, her focus has been building machine learning models for various industries including short-form video hosting, OTT, e-commerce, insurance tech, etc., for startups across the US, UK, Canada, and India. Prior to that, she was a Data Science Fellow with Insight Toronto and a Visiting Research Scholar at UCLA working in AutoML, MultiAgent Systems, and Emotion Recognition. In her free time, she helps produce the MLOps Community Podcast.