Orchestrating Machine Learning Workflows with Prefect

Prefect is a general-purpose workflow orchestration management system that enables users to build, run, and monitor data pipelines at scale. In this talk, Kevin shows how to orchestrate a full ML hyperparameter grid search pipeline over a Dask cluster. Other tools included will be Pandera, Tune, and Evidently AI.

Take-aways

1. How to orchestrate workflows with Prefect 2. How to scale workflows on top of Dask (grid search) 3. Setting up notifications for Slack 4. Prefect as a glue for other machine learning tools

In this episode

Kevin Kho

Open Source Community Engineer, Prefect

Kevin Kho is an Open Source Community Engineer at Prefect, an open-source workflow orchestration management system. Previously, he was a data scientist for four years working in the energy and HR spaces. Outside of work, he is a contributor for Fugue, an abstraction layer for Pandas, Spark, and Dask. He also organizes the Orlando Machine Learning and Data Science Meetup.

Demetrios Brinkmann

Host

Demetrios is one of the main organizers of the MLOps community and currently resides in a small town outside Frankfurt, Germany. He is an avid traveller who taught English as a second language to see the world and learn about new cultures. Demetrios fell into the Machine Learning Operations world, and since, has interviewed the leading names around MLOps, Data Science, and ML. Since diving into the nitty-gritty of Machine Learning Operations he felt a strong calling to explore the ethical issues surrounding ML. When he is not conducting interviews you can find him making stone stacking with his daughter in the woods or playing the ukulele by the campfire.