Coffee Sessions #98

Racing the Playhead: Real-time Model Inference in a Video Streaming Environment

Runway ML is doing an incredibly cool workaround applying machine learning to video editing. Brannon is a software engineer there and he’s here to tell us all about machine learning in video and how Runway maintains their machine learning infrastructure.

Take-aways

In no particular order (and certainly more than we'll get to): - Video is hard! - ... Especially if you are bound to the timing constraints of an HTTP request. - Caching more usually helps (but it comes at the cost of complexity). - Statelessness is your friend. - Merge to master quickly. - Autoscaling is an incredibly valuable and complex topic. - Queuing has value beyond batch jobs. Naive HTTP load balancing doesn't always cut. - I'm a big fan of Kubernetes. It's a beast, but so is the territory. - Monitoring and alerting is important in both (pre-production and production environments). Good visibility leads to good performance.

In this episode

Brannon Dorsey

Brannon Dorsey

Software Engineering Manager, Runway

Brannon Dorsey is an early employee at Runway, where he leads the Backend team. His team keeps infrastructure and high-performance models running at scale and helps to enable a quick iteration cycle between the research and product teams. Before joining Runway, Brannon worked on the Security Team at Linode. Brannon is also a practicing artist who uses software to explore ideas of digital literacy, agency, and complex systems.

Twitter

LinkedIn

Demetrios Brinkmann

Demetrios Brinkmann

Host

Demetrios is one of the main organizers of the MLOps community and currently resides in a small town outside Frankfurt, Germany. He is an avid traveller who taught English as a second language to see the world and learn about new cultures. Demetrios fell into the Machine Learning Operations world, and since, has interviewed the leading names around MLOps, Data Science, and ML. Since diving into the nitty-gritty of Machine Learning Operations he felt a strong calling to explore the ethical issues surrounding ML. When he is not conducting interviews you can find him making stone stacking with his daughter in the woods or playing the ukulele by the campfire.

Vishnu Rachakonda

Vishnu Rachakonda

Host

Vishnu Rachakonda is the operations lead for the MLOps Community and co-hosts the MLOps Coffee Sessions podcast. He is a machine learning engineer at Tesseract Health, a 4Catalyzer company focused on retinal imaging. In this role, he builds machine learning models for clinical workflow augmentation and diagnostics in on-device and cloud use cases. Since studying bioengineering at Penn, Vishnu has been actively working in the fields of computational biomedicine and MLOps. In his spare time, Vishnu enjoys suspending all logic to watch Indian action movies, playing chess, and writing.