MLOps EngineeringLabs: What We Learned Building End-to-end ML Applications on Flyte / How Flyte Orchestrates Tasks and Workflows
This talk provides some background on the March 2022 MLOps EngineeringLabs in collaboration with the team behind Flyte, then goes into lightning talks given by the three winning teams. Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft, Spotify, BlackShark, and many others. It is used across business-critical applications ranging from ETA, Pricing, Mapping, Autonomous, and many more. At its core is a Kubernetes native workflow engine that executes 10M+ containers per month as part of thousands of workflows. The talk focuses on: - What does a Simple ML User Journey look like? - The sacrifices ML Practitioners have to do to achieve Robustness? - How does Flyte help bridge the gap?
Take-aways
In this episode
Niels Bantilan
Machine Learning Engineer, Union.ai
Niels is a machine learning engineer and core maintainer of Flyte, an open-source ML orchestration tool, and author and maintainer of Pandera, a statistical typing and data testing tool for dataframes. His mission is to develop open-source tools for making data science and machine learning practitioners more productive. He has a Masters in Public Health with a specialization in sociomedical science and public health informatics, and prior to that a background in developmental biology and immunology. His research interests include reinforcement learning, AutoML, creative machine learning, and fairness, accountability, and transparency in automated systems.
Haytham Abuelfutuh
CTO, Union.Ai
Niels is a machine learning engineer and core maintainer of Flyte, an open-source ML orchestration tool, and author and maintainer of Pandera, a statistical typing and data testing tool for dataframes. His mission is to develop open-source tools for making data science and machine learning practitioners more productive. He has a Masters in Public Health with a specialization in sociomedical science and public health informatics, and prior to that a background in developmental biology and immunology. His research interests include reinforcement learning, AutoML, creative machine learning, and fairness, accountability, and transparency in automated systems.
Amale El Hamri
ML Engineer, Artefact
Niels is a machine learning engineer and core maintainer of Flyte, an open-source ML orchestration tool, and author and maintainer of Pandera, a statistical typing and data testing tool for dataframes. His mission is to develop open-source tools for making data science and machine learning practitioners more productive. He has a Masters in Public Health with a specialization in sociomedical science and public health informatics, and prior to that a background in developmental biology and immunology. His research interests include reinforcement learning, AutoML, creative machine learning, and fairness, accountability, and transparency in automated systems.
Ali Abbas Jaffri
Engineering Consultant, ML Reply AG
Niels is a machine learning engineer and core maintainer of Flyte, an open-source ML orchestration tool, and author and maintainer of Pandera, a statistical typing and data testing tool for dataframes. His mission is to develop open-source tools for making data science and machine learning practitioners more productive. He has a Masters in Public Health with a specialization in sociomedical science and public health informatics, and prior to that a background in developmental biology and immunology. His research interests include reinforcement learning, AutoML, creative machine learning, and fairness, accountability, and transparency in automated systems.
Demetrios Brinkmann
Host
Demetrios is one of the main organizers of the MLOps community and currently resides in a small town outside Frankfurt, Germany. He is an avid traveller who taught English as a second language to see the world and learn about new cultures. Demetrios fell into the Machine Learning Operations world, and since, has interviewed the leading names around MLOps, Data Science, and ML. Since diving into the nitty-gritty of Machine Learning Operations he felt a strong calling to explore the ethical issues surrounding ML. When he is not conducting interviews you can find him making stone stacking with his daughter in the woods or playing the ukulele by the campfire.
Ben Epstein
Host
Ben was the machine learning lead for Splice Machine, leading the development of their MLOps platform and Feature Store. He is now a founding software engineer at Galileo (rungalileo.io) focused on building data discovery and data quality tooling for machine learning teams. Ben also works as an adjunct professor at Washington University in St. Louis teaching concepts in cloud computing and big data analytics.