Apache Kafka: Event-Driven Architecture & Streaming

Event-Driven Architecture with Kafka

Event-driven approach is now used when developing software, and it is used in real-time applications as it improves the scalability and responsiveness of an application. It solves the problem of traditional approaches when dealing with extensive data and having to update the data with new information. Event-driven is not a programming language; it is a way of dealing with the events that occur by a user and how your application interacts with these events. 

Event-driven architecture is mainly used in real-time systems that respond to data changes and events as they occur in real-time; something that is very useful in IoT systems.

An event can be anything generated by the user, It can be a mouse click, or pressing a key on the keyboard.

Event-driven architecture is used when you want to manage different systems that respond to an event without having to write custom code for each consumer.

So when designing your application you should consider using this architecture if your application uses microservices and decoupled components.

Event-driven Architecture Explanation

Intuition for an event

Event-driven architecture consists of three main components: producer, consumer, and broker. The producer waits for the event to occur then it sends a message to the broker service which connects it to the consumer.

The consumer can read the message from the broker and takes action in the form of a callback – a function that responds to that event. Here the producer doesn’t expect a message from the consumer, unlike the RestAPI, there must be a response that may lead to a timeout after waiting for a response.

Events can be stored in something called event-logs, which can store the last state of an event.

What is Kafka?

Apache Kafka is a distributed system designed for streams. It’s designed to be horizontally scalable, fault-tolerant, and to also distribute data streams. It is an open-source publish/subscribe messaging system and often described as an event streaming architecture, and it’s used by thousands of companies.

Let’s suppose you have data that you need to transfer between the source system and destination system. By increasing the sources and destination systems, the integration places more and more pressure on your system. For every integration, you’ll need to specify a protocol and data format. 

But this problem can easily be solved with the help of Kafka event-driven architecture that decouples the data and your system. With Kafa in place, your source systems will publish the data to Apache Kafka, and target systems will get the data directly from Apache Kafka, which solves the problem of integration complexity.

Amazon also has an event-driven architecture called Kinesis, which is used for processing big data in real-time. Its architecture is similar to Kafka in many components such as producers, consumers, and brokers. The difference between Kafka vs Kinesis is that the Kafka concept is based on streams while Kinesis also focuses on analytics. Kafka wins in performance when compared to Kinesis. As far as the setup process is concerned, Kafka can take weeks while Kinesis can be set up in mere hours.

Why Use Kafka for an Event-Driven System?

The main reason for using Kafka for an event-driven system is the decoupling of microservices and creation of a Kafka pipeline to connect producers and consumers. Without having to check for new data, instead, you can simply listen to a particular event and take action.

Kafka offers a hybrid model that includes processing and messaging that is always scalable.

You can get started using Kafka by following this link for a quick start.

Patterns of Event-Driven Architecture

There are several ways to implement event-driven architecture, and which method you use depends on the use case and how many microservices are included. Here are some  common examples:

 Saga

The idea here is to put up sequence transactions that provide bigger and higher business purpose to manage data across microservices then we chain these transactions together. Each transaction updates the database and publishes an event to trigger the next transaction. If a transaction fails, saga will run compensation logic to go backwards and undo that effect or failure.

Command and Query Responsibility Segregation (CQRS)

With CQRS, you divide the application into two segments. One is intended for the update and delete which is called the writing model. You do the read on the other side of the model which is called the read model. Each side is optimized for quick and effective operation, whether reading or writing. It may join pieces of data across various entities.

Event Sourcing

An event source pattern is an approach that handles data after a sequence of events. These events are stored in an append-only store. These event stores act as a history of records of the changes in the states of the data and notify the consumer to handle them.

Additionally, the applications can access the history of events for the past 7 days to check the current state of an entity at any given time. 

Publish-Subscriber

The application will send message events to the consumers that are interested in updates. The sender will input a message to the input channel and pass it to the consumers. Here, the sender is the publisher and for each consumer, there is one output called subscribers.

A message broker is responsible for copying each message from the publisher and sending it to all the subscribers. It helps make the application run smoother and more reliable.

Conclusion

Event-driven microservices with Apache Kafka give you an effective way to implement coupled applications. As far as patterns such as sourcing and sagas are concerned, event-driven architecture can make the complexity more manageable.

It has many advantages like performance control, event flow tracking, reliable event sourcing. It also coordinates systems between teams within different regions.

Event-driven is mainly used in real-time applications and IoT. It can be used in fraud detection, anomaly detection, rule-based alerting, and web applications.

Learn more about the ins and outs of Kafka architecture

Try SQLake for free (early access)

SQLake is Upsolver’s newest offering. It lets you build and run reliable data pipelines on streaming and batch data via an all-SQL experience. Try it for free. No credit card required.