Whilst I was thinking about Kafka use cases, I had the idea to create a data-streaming application with data from GitHub and GitLab.

The idea is simple.

1. First, fetch and normalize the data from both Git APIs.
2. After you have normalized the data, produce messages to Kafka topics.
3. Finally, consume the messages with Kafka consumers. You can have several consumers polling from one topic, each of them performing a different operation.

On first glance, this “data pipeline” seems pretty trivial, and it is. To be honest, the reason I built this with Kafka is that I wanted to understand the ins and outs of Kafka, and I sure did.

While I was building the app, I realized the role Kafka plays in agile projects. Kafka gives you an enormous amount of flexibility. As the producers and consumers are separate microservices, they don’t rely at all on each other. Kafka gives you the ability to implement new requirements without the complexity of managing software architecture.

Let us look at an example. Right now, I only have one topic called git_commits. I also have only one consumer that polls from that topic. It is a simple consumer that sinks the data into a PostgreSQL database.

For this setup, I don’t need Kafka at all. I could have fetched the data from the API, normalized it, and then inserted the data into the database. Kafka is not needed for this task.

But as soon as I would like to change something, the reasons for Kafka in a microservices architecture become clearer. Because of the fact that all parts of the application are loosely coupled with Kafka, organization and management get easier. This is the definition of an event-driven architecture.

To put it in simple words, an event-driven architecture running on Kafka takes away the burden of finding out where the change needs to happen in the code. Kafka is the engine of an event-driven architecture, with microservices that are loosely coupled and do not rely on each other. This has several benefits.

• Loose coupling – Services do not need to know about each other, making the system easier to maintain and extend.
• Scalability – Individual services can be scaled independently based on workload.
• Fault tolerance – If one service is temporarily unavailable, events remain in Kafka until they can be processed.
• Extensibility – New consumers can subscribe to existing events without modifying the producer.
• Simpler evolution – New features are often implemented by adding new event consumers instead of changing existing services.

But what are the practical benefits of a loosely coupled architecture? Let’s extend our example with the PostgreSQL database. There is a new requirement to additionally upload the data from the Git APIs into an AWS S3 bucket.

What I would have to do if I were not using Kafka is search for the method or function that polls and normalizes the data. This is easy when the codebase is rather small, but if I try to do this in a project with hundreds of microservices, things get messy. I will not know where exactly to apply the change.

With Kafka, this is not a problem. All I have to do is take a look at what Kafka topics exist. I could use the CLI tool for this, or the project documentation if one exists. Next, all I have to do is to create a consumer that polls from the desired topic and uses the PutObject API from AWS to upload the data to S3. There is no searching around I need to do. The only complicated part is understanding the Kafka API.

Understanding Kafka, however, also gets easier as time goes on. I started hearing about Kafka for the first time early in my career. Since then, I have looked at the documentation for Apache Kafka from time to time. CommitFlow is my first PoC for Kafka, and this is where it clicked for me. The project is open-source and available on my GitHub if you want to take a closer look.

This was an introductory article about my new project CommitFlow. It highlights the benefits of an event-driven architecture built on top of Kafka. Future articles will dive into the project’s implementation in more detail.