Real-time Pattern Detection and Event Processing: Bridging Apache Flink with Serverless on AWS

In today’s data-driven world, organizations need to process and react to streaming data in real-time while maintaining cost-effectiveness and scalability. This talk demonstrates how to leverage Apache Flink’s powerful stream processing capabilities alongside AWS serverless technologies to build a robust, scalable, and cost-efficient event processing pipeline.

In this talk, I will delve into what CEP (Complex Event Processing) is and how a CEP layer such as Apache Flink works internally at a high level. The goal of this session is to provide an overview of how Serverless components can integrate with Apache Flink to build a real-time data processing pipeline.

We’ll explore a real-world architecture that uses MSF (Managed Service for Apache Flink) to process high-volume data streams from MSK (Managed Service for Kafka) and detect complex patterns in real-time. The detected patterns trigger serverless workflows using AWS Lambda, which orchestrates actions across DynamoDB for enrichment, SNS for notifications, and SQS for reliable event queuing.