Apache Flink vs Apache Kafka: Stream Storage Vs Stream Processing Explained

What is Apache Kafka

Apache Kafka is an open-source, distributed event-streaming platform designed to build high-throughput, real-time data pipelines and streaming applications.

Apache Kafka is a very popular tool, used by 80% of the Fortune 100 companies for real-time data flow.

Apache Kafka is highly trusted in the banking and insurance domains.

The telecom and energy sectors also rely on Kafka as a core part of their digital infrastructure.

Background of Kafka

Kafka was developed by LinkedIn in 2010, and the people behind its creation include Jay Kreps, Neha Narkhede, and Jun Rao. In 2011, Kafka was given to the Apache Open Source Community

What is Apache Kafka & Event Streaming?

Apache Kafka is an event streaming platform. Event streaming is the continuous flow of a sequence of real-time events.

To understand this, let’s look at two real-world examples:

Example 1: Uber Ride Booking (Sequence of Events)

When a user books a cab on Uber, a continuous stream of events is triggered:

• User books a ride $\rightarrow$ RideRequested Event occurs.
• Uber finds a nearby driver $\rightarrow$ DriverNotified Event occurs.
• Driver accepts the request $\rightarrow$ RideAccepted Event occurs.
• Live ride tracking starts $\rightarrow$ Continuous LocationUpdated Stream begins.

Kafka acts as the central backbone, reliably storing and routing every single one of these real-time events without losing data.

Example 2: E-Commerce User Clicks (High-Volume Data Streaming)

Apache Kafka is also built to handle massive volumes of real-time data ingestion. Consider an e-commerce platform like Amazon:

Every user click on a product page generates a data point. The analytics and recommendation engines need this data to suggest relevant products in real-time. Kafka effortlessly ingests and stores millions of user clicks per second from thousands of concurrent users.

What Happens Without an Event Streaming Platform?

Imagine building these systems without an Event Streaming Platform like Kafka:

1. Direct API to Database Writes: Every single user click or GPS movement would fire an API call directly writing to a traditional database. Under heavy traffic, this creates an immense load on both the servers and the database, inevitably leading to system crashes and high latency.
2. The Batch Job Workaround: To avoid crashing the database, engineers often store events in log files and run a batch job every 2 hours or once a day. While this protects the database, the data is no longer real-time, making instant recommendations or live tracking impossible.

Apache Kafka solves this exact problem. It sits between your applications and database systems as a high-throughput buffer, allowing you to store and stream millions of events in real time without straining your database.

Apache Kafka is highly optimized for writing data to disk. It achieves exceptional performance by batching messages together and leveraging low-level OS system calls.

Saving Infrastructure Cost

Kafka saves infrastructure cost for Organizations; we already discussed the system without an event streaming platform will require massive amount of raw database queries to store that data, and Kafka is a highly optimized system for storing data.

The following are the ways Kafka saves Infrastructure cost

• Acts as a Buffer: Prevents expensive databases from crashing under sudden traffic spikes.
• Eliminates Over-Provisioning: You don’t need to pay for massive, idle servers just to handle peak traffic hours.
• Efficient Hardware Usage: Uses sequential disk storage and OS Page Cache, delivering high throughput even on low-cost hardware.

What is Apache Flink

Apache Flink is an open-source, distributed stream-processing framework designed for stateful, real-time computations over unbounded (continuous) and bounded data streams with ultra-low latency

Apache Kafka is a great system for storing and transporting streaming data, but performing complex computations on this data is beyond Kafka’s scope.

Why a stream-processing system is needed

For a good understanding of why a stream-processing system is needed.

Example: Real-Time Fraud Detection in Banking

• The Raw Stream: Millions of credit card transaction events flowing in every second (e.g., Card ID, Amount, Location, Timestamp).
• The Streaming Processing Logic: A processing engine (like Apache Flink) maintains the state of each user’s recent transactions in memory and calculates the physical possibility:
  • Event A: Card swiped in New York at 10:00 AM.
  • Event B: Same Card swiped in London at 10:05 AM.
• The Action: The processing engine instantly calculates that traveling between these locations in 5 minutes is physically impossible, flags the anomaly in sub-seconds, and blocks the card before the transaction settles.

Difference between Apache Kafka and Flink