Coffee Sessions #97

Real-Time Exactly-Once Event Processing with Apache Flink, Kafka, and Pinot

A few years ago Uber set out to create an ads platform for the Uber Eats app that relied heavily on three pillars; Speed, Reliability, and Accuracy. Some of the technical challenges they were faced with included exactly-once semantics in real-time. To accomplish this goal, they created the architecture diagram above with lots of love from Flink, Kafka, Hive, and Pinot. You can dig into the whole paper here (https://go.mlops.community/k8gzZd) to see all the reasoning for their design decisions.

In this episode

Jacob Tsafatinos

Jacob Tsafatinos

Staff Software Engineer, Elemy

Jacob Tsafatinos is a Staff Software Engineer at Elemy. He led the efforts of the Ad Events Processing system at Uber and has previously worked on a range of problems including data ingestion for search and machine learning recommendation pipelines. In his spare time, he can be found playing lead guitar in his band Good Kid.

Twitter

LinkedIn

Demetrios Brinkmann

Demetrios Brinkmann

Host

Demetrios is one of the main organizers of the MLOps community and currently resides in a small town outside Frankfurt, Germany. He is an avid traveller who taught English as a second language to see the world and learn about new cultures. Demetrios fell into the Machine Learning Operations world, and since, has interviewed the leading names around MLOps, Data Science, and ML. Since diving into the nitty-gritty of Machine Learning Operations he felt a strong calling to explore the ethical issues surrounding ML. When he is not conducting interviews you can find him making stone stacking with his daughter in the woods or playing the ukulele by the campfire.

Mihail Eric

Mihail Eric

Host

Mihail optimizes for impact. he strives to work with talented people to do amazing things. He's an engineer, researcher, and educator that has helped start teams at innovative organizations such as Amazon Alexa and RideOS. Mihail runs Pametan, a consultancy helping companies across verticals deliver machine learning and data-driven solutions to their hardest problems with a special focus on NLP, recommendation systems, tabular data, and computer vision domains. They’ve helped teams deliver 33% lift on key business metrics in <6 weeks. Mihail also built Confetti AI, the premier educational platform for training the next generation of machine learning practitioners.