Complete gig history
All talks by Olena Kutsenko
4 talks·4 with video·3 years
Loading talks…
For updates and more, join our community 👉 https://www.linkedin.com/company/devoxx-united-kingdom Real-time data can be messy, unpredictable, and hard to manage. To unlock its full potential, you need a way to turn raw streams into clean, structured data. In this talk, we’ll show you how to use Apache Kafka, Apache Flink, and Apache Iceberg to organize real-time data streams efficiently and prepare them for advanced use cases, including AI applications. We’ll start by explaining how Kafka handles high-speed data streams and how Flink processes these streams in real time. You’ll learn how to use Flink to transform raw data into structured formats, ensuring it’s ready for storage and analysis. Then, we’ll dive into Iceberg, demonstrating how it stores and organizes structured data for easy querying, versioning, and integration with machine learning pipelines. Through clear examples, we’ll walk you through building a practical pipeline that turns chaotic data streams into organized schemas. By the end of the session, you’ll know how to manage real-time data effectively and set the stage for downstream AI and analytics. Whether you’re a beginner or an experienced developer, this talk will give you the tools to simplify and enhance your data pipelines!
🎙 Olena Kutsenko, Senior Developer Advocate @Aiven 🔗 https://twitter.com/OlenaKutsenko ☑ Website: https://devoxx.com.ua/ ☑ Facebook: https://www.facebook.com/DevoxxUkraine ☑ Instagram: https://www.instagram.com/devoxxua/ ☑ Twitter: https://twitter.com/DevoxxUA ☑ YouTube: https://www.youtube.com/@DevoxxUkraine Devoxx Ukraine 2023 partners: 🫶 Platinum Partner & Organizer - EPAM Ukraine https://careers.epam.ua 🫶 Silver Partner - SPD Technology https://spd.tech/ 🫶 Streaming partner - Mediastream https://mediastream.com.ua/
For more info on the next Devoxx UK event 👉 www.devoxx.co.uk Apache Kafka is a powerful tool to connect multiple systems together, allowing the data to flow across multiple services and be reused for multiple purposes. This can be useful in many scenarios both for mission-critical applications, as well as for fast data explorations. In this talk I’ll show one such data exploration. Mastodon, as a tool for microblogging, is rising in popularity in recent months. If you just recently joined Mastodon and are still exploring it, you might find that scrolling the timeline has its limits to understand all that is happening there. That being the case, applying some engineering skills will give a better overview on topics and discussions happening on the platform. Since Mastodon's timeline is nothing more than a collection of continuously arriving events, its feed is well-suited for Apache Kafka. Adding Kafka connectors on top of that opens multiple opportunities to use data for aggregations and visualizations. During this talk you'll learn how to bring data from Mastodon to Kafka using TypeScript and a couple of helpful libraries. Once the data is in the topic, we'll use Kafka Connect to bring the data into OpenSearch and use it for search, aggregations and visualizations. This talk is for both beginners in Apache Kafka and intermediate users. We'll use some more advanced concepts, but will keep it all simple, so that everyone can follow along and experiment with Mastodon data!
For more info on the next Devoxx UK event 👉 www.devoxx.co.uk Apache Kafka is amazing at handling real-time data feeds. However, in certain cases we need to come back to older records to analyse and process data at later times. This is challenging because storing records indefinitely and doing large scans of data in Apache Kafka is not optimal. ClickHouse, on the other hand, is a scalable and reliable storage designed to handle petabytes of data and, at the same time, it is a powerful open-source tool for fast online analytical processing. It was initially developed by Yandex and is now used by many companies for data analytics. What's more ClickHouse has a built-in table engine to publish and subscribe to Apache Kafka data feeds. Hence, we can use Apache Kafka and ClickHouse in collaboration to transition older records to a data lake in order to perform analytics at scale on top of fresh data that Kafka provides. In this talk you’ll learn how to use Apache Kafka together with ClickHouse and how to query the data stored in the data lake. This session is for those who want to perform analytics with fast response time over a huge volume of data without the need to downsample it.