Kafka 4.0 Release: Simplified Architecture with Default KRaft

Originally published at ssojet Apache Kafka 4.0 marks a significant milestone by adopting KRaft mode as the default operation, eliminating the need for Apache ZooKeeper. This architectural change streamlines deployment and management, reducing operational overhead while enhancing scalability. The release notes detail these enhancements and express gratitude to the ZooKeeper community for its long-standing support. Image courtesy of Confluent documentation Lalit Moharana, an AWS Community Builder, emphasized the significance of this shift: "ZooKeeper is stepping aside as Apache Kafka adopts KRaft... This shift simplifies Kafka’s architecture by ditching the separate ZooKeeper system, boosting scalability, and paving the way for a self-sufficient future." Kafka 4.0 also introduces KIP-848, a next-generation consumer group protocol designed to improve rebalance performance, significantly reducing downtime and latency in large-scale environments. This new protocol is enabled by default on the server side, requiring consumers to opt in by setting the configuration group.protocol=consumer. In addition, Kafka 4.0 provides early access to Queues for Kafka (KIP-932), enabling point-to-point messaging patterns using regular Kafka topics. This enhancement broadens Kafka's application scope, catering to various messaging use cases. Kafka 4.0 Enhancements Kafka 4.0 introduces several notable improvements and features: Java Requirements: Kafka Clients and Kafka Streams now require Java 11, while Kafka Brokers, Connect, and Tools require Java 17. This encourages the use of modern Java features and aligns Kafka with current technology stacks. Removal of Deprecated APIs: The release eliminates APIs deprecated for at least 12 months, ensuring a cleaner codebase and promoting the adoption of newer features. Support for New Protocols: Several new enhancements include: KIP-890 reduces "zombie transactions" during producer failures. KIP-966 introduces Eligible Leader Replicas (ELR) in preview, preventing data loss by ensuring safe leader elections. KIP-996 reduces unnecessary KRaft leader elections, minimizing disruptions caused by network issues. Administration Improvements: New command-line tools such as kafka-groups.sh allow users to manage consumer and share groups effectively, providing accurate information even when existing APIs fail. Kafka Streams and Connect Enhancements Kafka Streams has also seen notable upgrades in version 4.0: KIP-1104 allows foreign key extraction from both keys and values in KTable joins, simplifying data processing. KIP-1112 introduces the ProcessorWrapper interface, enabling easier application of cross-cutting logic in Kafka Streams. In Kafka Connect, several improvements have been made: KIP-970 removes redundant task configuration endpoints, streamlining the API. KIP-1074 allows the replication of user internal topics, enhancing flexibility. Conclusion For enterprises looking to implement secure authentication, SSOJet’s API-first platform offers comprehensive solutions for single sign-on, multi-factor authentication, and user management. Ensure your organization is prepared for the future of data streaming with Kafka 4.0 while securing user credentials through SSOJet's features, including directory sync, SAML, OIDC, and magic link authentication. Explore our services or contact us at ssojet.com.

Apr 9, 2025 - 10:54

Kafka 4.0 Release: Simplified Architecture with Default KRaft

Originally published at ssojet

Apache Kafka 4.0 marks a significant milestone by adopting KRaft mode as the default operation, eliminating the need for Apache ZooKeeper. This architectural change streamlines deployment and management, reducing operational overhead while enhancing scalability. The release notes detail these enhancements and express gratitude to the ZooKeeper community for its long-standing support.

Image courtesy of Confluent documentation

Lalit Moharana, an AWS Community Builder, emphasized the significance of this shift:

"ZooKeeper is stepping aside as Apache Kafka adopts KRaft... This shift simplifies Kafka’s architecture by ditching the separate ZooKeeper system, boosting scalability, and paving the way for a self-sufficient future."

Kafka 4.0 also introduces KIP-848, a next-generation consumer group protocol designed to improve rebalance performance, significantly reducing downtime and latency in large-scale environments. This new protocol is enabled by default on the server side, requiring consumers to opt in by setting the configuration group.protocol=consumer.

In addition, Kafka 4.0 provides early access to Queues for Kafka (KIP-932), enabling point-to-point messaging patterns using regular Kafka topics. This enhancement broadens Kafka's application scope, catering to various messaging use cases.

Kafka 4.0 Enhancements

Kafka 4.0 introduces several notable improvements and features:

Java Requirements: Kafka Clients and Kafka Streams now require Java 11, while Kafka Brokers, Connect, and Tools require Java 17. This encourages the use of modern Java features and aligns Kafka with current technology stacks.
Removal of Deprecated APIs: The release eliminates APIs deprecated for at least 12 months, ensuring a cleaner codebase and promoting the adoption of newer features.
Support for New Protocols: Several new enhancements include:
KIP-890 reduces "zombie transactions" during producer failures.
KIP-966 introduces Eligible Leader Replicas (ELR) in preview, preventing data loss by ensuring safe leader elections.
KIP-996 reduces unnecessary KRaft leader elections, minimizing disruptions caused by network issues.
Administration Improvements: New command-line tools such as kafka-groups.sh allow users to manage consumer and share groups effectively, providing accurate information even when existing APIs fail.

Kafka Streams and Connect Enhancements

Kafka Streams has also seen notable upgrades in version 4.0:

KIP-1104 allows foreign key extraction from both keys and values in KTable joins, simplifying data processing.
KIP-1112 introduces the ProcessorWrapper interface, enabling easier application of cross-cutting logic in Kafka Streams.

In Kafka Connect, several improvements have been made:

KIP-970 removes redundant task configuration endpoints, streamlining the API.
KIP-1074 allows the replication of user internal topics, enhancing flexibility.

Conclusion

For enterprises looking to implement secure authentication, SSOJet’s API-first platform offers comprehensive solutions for single sign-on, multi-factor authentication, and user management. Ensure your organization is prepared for the future of data streaming with Kafka 4.0 while securing user credentials through SSOJet's features, including directory sync, SAML, OIDC, and magic link authentication. Explore our services or contact us at ssojet.com.