Seeking a Third Opinion on Kafka Consumer Implementation and Architectural Disagreements

I've been at my current job for about 4-5 months, mainly working with Go, and I have no prior experience with Kafka. Before this, my background was in JavaScript, Node.js, React, etc. I recently got a simple task: write a Kafka-based agent that listens to a topic and stores all events from this topic in a database. I believe I've done it correctly and have sent it for code review. However, our architect pointed out several flaws in my code, especially disagreeing with my architecture. I divided my code into three layers: The first is a thin wrapper around the kafka-go consumer for logging, tracing, etc. The second listens to Kafka messages and sends them via a channel to the user. The third is the processor, which receives the channel, listens to events, and processes each one or in batch. It's designed so you can pass any handler; in my case, it's one that writes to a database. Our architect argues that my design is overly complex and could be simpler. He suggests there's no need for go routines and channels and that I could use kafka-go's built-in ReadBatch. I found that this method uses minimum and maximum batch sizes and doesn't work with atomic messages, so I would need to handle this myself. My research suggests it's unusual to use this low-level approach due to its pitfalls, with most users preferring ReadMessage, FetchMessage, and Commit. I believe my approach is simpler, especially for batch processing. I aggregate messages in a batch using FetchMessage and process the batch by storing it in DB. So, my questions for those experienced with kafka-go are: Is my approach valid, or should I reconsider it? Is the solution proposed by my architect more complex than it seems, or am I being lazy? Is there another method in kafka-go for reading messages in batches that I missed? If the architect's assumption is correct, I'm willing to make the effort to change it. But if my solution is better, how can I argue this, especially since he seems a bit emotional about it? Thanks in advance.

Apr 21, 2025 - 11:44

I've been at my current job for about 4-5 months, mainly working with Go, and I have no prior experience with Kafka. Before this, my background was in JavaScript, Node.js, React, etc. I recently got a simple task: write a Kafka-based agent that listens to a topic and stores all events from this topic in a database. I believe I've done it correctly and have sent it for code review. However, our architect pointed out several flaws in my code, especially disagreeing with my architecture.

I divided my code into three layers:

The first is a thin wrapper around the kafka-go consumer for logging, tracing, etc.
The second listens to Kafka messages and sends them via a channel to the user.
The third is the processor, which receives the channel, listens to events, and processes each one or in batch. It's designed so you can pass any handler; in my case, it's one that writes to a database.

Our architect argues that my design is overly complex and could be simpler. He suggests there's no need for go routines and channels and that I could use kafka-go's built-in ReadBatch. I found that this method uses minimum and maximum batch sizes and doesn't work with atomic messages, so I would need to handle this myself. My research suggests it's unusual to use this low-level approach due to its pitfalls, with most users preferring ReadMessage, FetchMessage, and Commit. I believe my approach is simpler, especially for batch processing. I aggregate messages in a batch using FetchMessage and process the batch by storing it in DB.

So, my questions for those experienced with kafka-go are:

Is my approach valid, or should I reconsider it?
Is the solution proposed by my architect more complex than it seems, or am I being lazy? Is there another method in kafka-go for reading messages in batches that I missed?
If the architect's assumption is correct, I'm willing to make the effort to change it. But if my solution is better, how can I argue this, especially since he seems a bit emotional about it?

Thanks in advance.