tchh22 | Batching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
Vortrag – Data | 30 min | Englisch
MI | 10:00 | Alte Liebe
Please note that we would like to hold this talk together: Felix (CEO of Baqend) and Wolle (Head of Data Engineering at the time when we developed the system described in the talk).
Fast analytics are critical for business success, because you cannot act quickly when your dashboards are stale. But how do you keep latency down when your growth is through the roof? This is a war story about scaling our analytics stack from 1 to 100+ million users in a single month using Apache Flink.
As a web performance startup, accelerating websites is our day-to-day business and tracking user performance is part of the job: We collect traces for every single page load that happens on our infrastructure to monitor critical business KPIs in realtime. But with our user base doubling every other month, performing heavy-weight analytics with sub-minute end-to-end latency turned from “difficult” to “seemingly impossible”. Together with the AWS Prototyping Team, we tackled the problem head-on and reengineered our analytics stack from scratch: We replaced our ETL-heavy Athena warehouse with a continuously running Kinesis Data Analytics application based on Flink and devised a blithesome dashboard on top of Elasticsearch and Kibana. In this talk, we will share our most exciting learnings of that journey and provide details on how we implemented a massively scalable real-time aggregation pipeline on top of Apache Flink.
Wolfram Wingerath / Felix Gessert
Uni Oldenburg / Baqend
Felix is CEO and co-founder of Baqend. During his PhD studies at the University of Hamburg, he developed the core technology behind Baqend’s Web performance service Speed Kit. Felix is passionate about making the Web faster by turning research results into real-world applications. He frequently talks at conferences about exciting tech trends in data management and Web performance. As the CEO, Felix is responsible for establishing Baqend’s solution “Speed Kit” as a new category in the software market.
Wolle is a professor for data science at the University of Oldenburg and Data Science Advisor at Baqend. During his PhD studies at the University of Hamburg, Wolle conceived the scalable design behind Baqend’s real-time query engine and thereby developed a strong background in real-time databases and related technologies such as NoSQL, stream processing, and Big Data analytics. Before becoming a professor in 2022, Wolle headed Baqend’s data engineering team and was responsible for developing and operating Baqend’s real-user monitoring pipeline which enables zero-latency analytics for Baqend customers.
