Spring Kafka - How to use ReplyingKafkaTemplate send and reply synchronously
Microservice Spring with Apache Kafka |
Today, I want to share how to use Kafka. It's quite interesting once you get the hang of it, and I hope my explanation will make it a bit easier for you.
So, Kafka is primarily known as an asynchronous messaging queuing system. This means it allows different parts of your application to communicate with each other without waiting for immediate responses. However, there are times when you might need a more immediate reply from another service in your project. Luckily, Kafka can handle this too.
Here’s how it works: The client side sends a message to the server side via Kafka. This message includes a special identifier in the Kafka header request, known as KafkaHeaders.CORRELATION_ID, along with a topic where the server should respond. The server, after processing the request, sends a message back to the same topic with the same correlation ID. This way, the client knows that the message is a response to its initial request and can act accordingly.
Although Kafka can be integrated with various frameworks and languages, I’ll focus on how to send and receive Kafka messages using Spring Boot, which is a popular choice among developers. Spring Boot simplifies many aspects of working with Kafka and makes the process more straightforward.
The Project Overview
The client-project will send a string message to the server-project over kafka, then the server-project will reverse the string and return it back to the client.First let’s start with our pom.xml, for both services we named spring-kafka-client and spring-kafka-server. The dependencies required are as follows.
And also the spring configuration files are the same for both services. Here’s the application.yml that you must create.
1. Client Service
Spring projects the client side, which produces a message and waits for the result.
This is an entry application containing the main method, we annotated them with @EnableKafka.
Main application for spring-kafka-client
Kafka configuration
- We registered bean ReplyingKafkaTemplate into our application, so this kafka template will be available in our application.
- We also register ConcurrentMessageListenerContainer, in this configuration we put REPLY_TOPICS and CONSUMER_GROUPS that are in our Yaml configuration.
Service that produce message
This service will be reusable throughout your client application, it will send synchronous requests and expect results from the producer. Basically we send messages into the SEND_TOPICS topic, with specific payload and headers, then it will be detected by the consumer somewhere on our server project.
2. Server Service
Spring project that will do the process of a given input. It can be some specific business logic, database query, etc.
Main application for spring-kafka-server.
Consumer and producer
- It listens to only a specific topic Which are into topic “${myproject.send-topics}” and by consumer “${myproject.consumer-group}”, those are variables that we set early in properties.
- It then @SendTo the reply topic, Kafka will automatically set the Header and Reply Topic that is sent by the client.
- Then inside the method it’s just a simple stringBuilder reverse string method. You can put your real business logic here.
Test the Application
Because it’s not a rest API web based spring application, we can just do an integration test of our send and reply kafka project, using Junit tests.
Write and run the test.
The test should pass if the server responds with a reversed string, from the message that client is sending. It’s just a simple logic for a test.
You can create a rest API if you want, all you have to do is just autowiring the KafkaService and use the kafkaRequestReply method.
We only do the test in our client project, because we just want to know the replying result. We don't necessarily have to write the test in the server project. But you can write a test for your server(producer) project yourself.
The full project can be clone from this Github repository Spring Kafka Request and wait Reply.
You can find more on the Spring kafka documentation here basically about anything you want to know about Kafka in Spring, here's the spring documentation https://docs.spring.io/spring-kafka/reference/html/#replying-template.
You can find more on the Spring kafka documentation here basically about anything you want to know about Kafka in Spring, here's the spring documentation https://docs.spring.io/spring-kafka/reference/html/#replying-template.