_posts/2021-04-19-what-s-new-in-apache5.html

--- layout: post status: PUBLISHED published: true title: What’s New in Apache Kafka 2.8.0 id: 8f90e3be-a2e9-4af1-96d5-a55d7ae256d8 date: '2021-04-19 16:37:22 -0400' categories: kafka tags: [] permalink: kafka/entry/what-s-new-in-apache5 --- I'm proud to announce the release of <a href="https://kafka.apache.org/downloads">Apache Kafka 2.8.0</a> on behalf of the Apache Kafka® community. The 2.8.0 release contains many new features and improvements. This blog post highlights some of the more prominent ones. Be sure to see the <a href="https://dist.apache.org/repos/dist/release/kafka/2.8.0/RELEASE_NOTES.html">release notes</a> for the full list of changes. You can also watch the <a href="https://youtu.be/vp-hV_li_bk">release video</a> for a summary of what's new. This release offers an early access version of KIP-500, which allows you to run Kafka brokers without Apache ZooKeeper, instead depending on an internal Raft implementation. This highly anticipated architectural improvement enables support for more partitions per cluster, simpler operation, and tighter security. <h1>Kafka broker, producer, and consumer</h1> <h2>KIP-500: Replace ZooKeeper with a self-managed quorum</h2> We are excited to announce that 2.8 introduces an early-access look at Kafka without ZooKeeper! The implementation is not yet feature complete and should not be used in production, but it is possible to start new clusters without ZooKeeper and go through basic produce and consume use cases. At a high level, <a href="https://cwiki.apache.org/confluence/x/KoxiBw">KIP-500</a> works by moving topic metadata and configurations out of ZooKeeper and into a new internal topic named <code>@metadata</code>. This topic is managed by an internal Raft quorum of "controllers" and is replicated to all brokers in the cluster. The leader of the Raft quorum serves the same role as the controller in clusters today. A node in the KIP-500 world can serve as a controller, a broker, or both, depending on the new <code>process.roles</code> configuration. <a href="https://github.com/apache/kafka/blob/2.8/config/kraft/README.md">See the README</a> for quickstart instructions and additional details. This release has been a massive effort by the community over the past year, and this will continue over the course of this year. We expect significant improvements when it comes to feature completeness and hardening by the mid-year and end of year releases. Here is a quick look at the most significant KIPs that have been merged: <ul> <li> <a href="https://cwiki.apache.org/confluence/x/KoxiBw">KIP-500</a>: lays out the vision for an event-driven model for managing metadata with a replicated log managed with the Raft protocol </li> <li><a href="https://cwiki.apache.org/confluence/x/Li7cC">KIP-595</a>: specifies the Raft protocol, which is used for the <code>@metadata</code> topic </li> <li> <a href="https://cwiki.apache.org/confluence/x/4RV4CQ">KIP-631</a>: specifies the event-driven controller model, including the new broker lifecycle and the metadata record schemas </li> <li> <a href="https://cwiki.apache.org/confluence/x/dyfcC">KIP-590</a>: specifies a new protocol to allow forwarding client requests from brokers to the controller </li> </ul> <h2>KIP-700: Add Describe Cluster API</h2> The Kafka <code>AdminClient</code> has historically used the broker's Metadata API to get information about the cluster. However, the Metadata API is primarily focused on supporting the consumer and producer client, which follow different patterns than the <code>AdminClient</code>. <a href="https://cwiki.apache.org/confluence/x/jQ4mCg">KIP-700</a> decouples the <code>AdminClient</code> from the Metadata API by adding a new API to directly query the brokers for information about the cluster. This change enables the addition of new admin features in the future without disruption to the producer and consumer. <h2>KIP-684: Support mutual TLS authentication on SASL_SSL listeners</h2> Historically, Kafka disabled TLS client authentication (also known as mutual TLS authentication) for SASL_SSL listeners even if <code>ssl.client.auth</code> was configured. This behaviour was introduced at a time when this configuration option could only be configured broker-wide. In the common case where SASL_SSL used SASL authentication without requiring key store distribution, enforcing TLS client authentication for SASL_SSL clients was not desirable. <a href="https://cwiki.apache.org/confluence/x/oCDZCQ">KIP-684</a> allows you to combine TLS authentication with SASL-based client identity on a per-listener basis. This is important for organizations where mutual TLS authentication is mandatory. KIP-684 builds on the listener-prefixed configuration options introduced by <a href="https://cwiki.apache.org/confluence/x/qwkIB">KIP-103</a>. <h2>KIP-676: Respect logging hierarchy</h2> Log4j uses a hierarchical model for configuring loggers within an application. Each logger's name is delimited by periods (.), which are treated as levels in the logger hierarchy. Individual loggers and intermediate hierarchy levels can both be configured (for example, to enable debug logging). If an individual logger is not explicitly configured, it inherits the configuration of its nearest ancestor, all the way up to the root logger, which is the common ancestor of all loggers in the system. Historically, the Kafka broker's APIs for viewing log levels did not respect this hierarchy, instead reporting only the root logger's configuration for any unconfigured individual logger. <a href="https://cwiki.apache.org/confluence/x/yQ7ZCQ">KIP-676</a> corrects this behavior by instead resolving the logger configurations the same way that the logging framework does. <h2>KIP-673: Emit JSONs with new auto-generated schema</h2> Kafka brokers offer debug-level request/response logs. Previously, they were semi-structured logs produced by the request/response classes' <code>toString</code> override. <a href="https://cwiki.apache.org/confluence/x/dkt4CQ">KIP-673</a> adjusts these logs to be JSON structured so that they can more easily be parsed and used by logging toolchains. <h2>KIP-612: Limit broker connection creation rate</h2> Creating a new connection adds overhead to the broker. <a href="https://cwiki.apache.org/confluence/x/VQPgB">KIP-306</a> mitigated the issue of connection storms due to unauthorized connections. However, connection storms may also come from authorized clients. To make it easier for you to ensure the stability of the brokers, <a href="https://cwiki.apache.org/confluence/x/ZhgRCQ">KIP-612</a> adds the ability to set a limit on the rate at which the broker accepts new connections, both overall and per IP address. <h2>KIP-516: Topic identifiers</h2> Previously, topics in Kafka were identified solely by their name. <a href="https://cwiki.apache.org/confluence/x/ZRGYBw">KIP-516</a> introduces topic IDs to uniquely identify topics. Topic IDs are unique throughout their lifetime, even beyond deletion of the corresponding topic. They are also a more efficient representation on the wire and in memory compared to topic names. Starting in 2.8.0, existing and new topics will be given topic IDs. Clients can now receive topic IDs through metadata responses. Work for this KIP is still ongoing, and future releases will include support for more efficient deletes, as well as adding topic IDs to other requests. <h1>Kafka Connect</h1> <h2>KIP-661: Expose task configurations in Connect REST API</h2> Kafka Connect exposes a REST API allowing callers to view the configuration of running connectors. This is very useful, as it allows you to determine the workload of connectors. In Connect, connectors process the configuration before actually beginning execution and in some cases specialize and map this configuration to each individual task that will perform the actual work of transferring data to or from Kafka. You have historically been able to view the nominal configuration, but not the actual resolved configurations used by the running tasks. <a href="https://cwiki.apache.org/confluence/x/GDV4CQ">KIP-661</a> adds a new API endpoint and method to allow callers to retrieve the actual runtime configuration of a connector's tasks. This can be used for debugging but also for understanding the impact of failures (for example, a task crashing). <h1>Kafka Streams</h1> <h2>KIP-696: Update Streams FSM to clarify <code>ERROR</code> state meaning</h2> Kafka Streams exposes a <a href="https://kafka.apache.org/28/javadoc/org/apache/kafka/streams/KafkaStreams.State.html">state machine</a> to help you to reason about the state of your applications in logs and metrics, as well as to <a href="https://kafka.apache.org/28/javadoc/org/apache/kafka/streams/KafkaStreams.StateListener.html">trigger user-defined behavior on state transitions</a>. This state machine contains an "<code>ERROR</code>" state that has historically meant that all threads have died. Until <a href="https://cwiki.apache.org/confluence/x/FDd4CQ">KIP-663</a>, there was no way to replace dead threads, so <code>ERROR</code> was a terminal state. However, with the addition of recent resilience improvements (KIP-663 and KIP-671 below), having no running threads no longer clearly indicates an <code>ERROR</code>, nor is it terminal. Regardless, applications can still experience fatal errors, and you still need to know when this happens. <a href="https://cwiki.apache.org/confluence/x/lCvZCQ">KIP-696</a> updates the state machine to grant <code>ERROR</code> a more specific meaning, namely that it is a terminal state indicating that the application has experienced a fatal <code>ERROR</code>. <h2>KIP-689: Extend <code>StreamJoined</code> to allow more store configs</h2> Kafka Streams offers the <code>StreamJoined</code> config object to set various configuration options for join operations. <a href="https://cwiki.apache.org/confluence/x/DyrZCQ">KIP-689</a> adds the ability to control the settings of the changelog topics that make the join state durable. The default configuration is still appropriate for most applications, but advanced operators need the ability to tune the configurations of the internal topics that support stream processing. <h2>KIP-680: <code>TopologyTestDriver</code> should not require a properties argument</h2> Kafka Streams offers the <code>TopologyTestDriver</code> runtime, which supports testing entire Streams applications in a fast, single-threaded, deterministic environment without running any extra components (like brokers or ZooKeeper). The constructor of <code>TopologyTestDriver</code> was designed to mirror the constructor of <code>KafkaStreams</code> (the main runtime): It takes the application itself (a topology) as one argument and the configuration as a second argument. Historically, <code>TopologyTestDriver</code> enforced the same required configs as <code>KafkaStreams</code>, including a broker connection string and an application identifier, even though these configurations are meaningless for <code>TopologyTestDriver</code>. Starting in 2.8.0, these boilerplate configurations are no longer required, and <a href="https://cwiki.apache.org/confluence/x/MB3ZCQ">KIP-680</a> simplifies the common case by adding new constructor overloads that do not require a configuration argument at all. The main constructor is still available, so your current tests will continue to work, and you can still use the main constructor if you need to specify extra configuration options. <h2>KIP-671: Introduce Kafka-Streams-specific uncaught exception handler</h2> Kafka Streams encapsulates complex logic, including both user- and system-defined code, I/O operations, multi-threading, etc., all of which offer any number of opportunities to encounter an unexpected exception. Before, Kafka Streams adopted the safe and simple approach of throwing the exceptions up to the top level, which would ultimately kill the relevant execution thread. For visibility, Streams exposed the native Java thread's ability to register an <code>UncaughtExceptionHandler</code>. In practice, many use cases require more than just visibility when a thread dies. <a href="https://cwiki.apache.org/confluence/x/lkN4CQ">KIP-671</a> adds a new handler (<code>StreamsUncaughtExceptionHandler</code>), which offers the same level of visibility while also providing a mechanism to replace the dead thread (if you desire more resilience) or shut down the system (either all threads in the current instance or all instances in the cluster), in case you prefer to fail fast. The handler allows the selection of different actions, depending on the actual exception. <h2>KIP-663: API to start and shut down Streams threads</h2> Kafka Streams applications are structured as a cluster of instances, each with some number of execution threads. The number of threads is configured at startup. Under heavy load, you may wish to experiment with increasing or decreasing the number of threads on an instance in order to better utilize system resources or reduce bottlenecks. Previously, this involved stopping, reconfiguring, and restarting each instance. <a href="https://cwiki.apache.org/confluence/x/FDd4CQ">KIP-663</a> adds new methods to the <code>KafkaStreams</code> interface, which allows you to individually add and remove processing threads without disrupting the other threads running on the same instance. <h2>KIP-659: Improve <code>TimeWindowedDeserializer</code> and <code>TimeWindowedSerde</code> to handle window size</h2> One of the operations that Kafka Streams provides is the ability to window an input record stream for aggregation. For example, when computing the number of updates for each key per hour, the window size is one hour. The window size is defined as part of the stream processing logic in the Streams DSL, and Kafka Streams automatically configures the serializer and deserializer necessary to store and retrieve these windows from local storage and Kafka topics. Because the window size itself is fixed and known for a particular operation, the serializer and deserializer contain a space optimization, storing only the window's start timestamp (as the end can be computed by adding the window size to the start time). Occasionally, you need to directly load serialized records, for example, when debugging an application or verifying an intermediate processing phase. To support these use cases, <a href="https://cwiki.apache.org/confluence/x/aDR4CQ">KIP-659</a> gives callers a way to directly configure the deserializer (<code>TimeWindowedDeserializer</code>) with the window size, in much the same way that Streams configures its own internal deserializer for the same data. <h2>KIP-572: Improve timeouts and retries in Kafka Streams</h2> <a href="https://cwiki.apache.org/confluence/x/5ArcC">KIP-572</a> was partially implemented in Apache Kafka 2.7.0 and completed in 2.8.0. This KIP adds a new retry behavior to fill an important resilience gap in running Kafka Streams applications. Many of Streams's functions rely on remote calls, for example, to Kafka brokers. As with any network call, these operations are subject to arbitrary errors and delays. The Kafka client libraries that Streams relies on have their own resilience settings, which can help to smooth out minor network disruptions, but setting the clients to be too resilient means that any client API call may block for a long time, which affects the overall stability of the application. On the other hand, setting these client timeouts too short would lead to applications crashing during minor network outages. KIP-572 adds a higher-level retry loop. Now, when Streams encounters a timeout exception while processing a task, it will attempt to make progress on other tasks before retrying the failed one. <h1>Conclusion</h1> Apache Kafka 2.8.0 has a lot of great fixes and improvements in addition to the KIPs listed here. For next steps: <ul> <li>See the <a href="https://dist.apache.org/repos/dist/release/kafka/2.8.0/RELEASE_NOTES.html">release notes</a> for the full list of changes </li> <li>Check out the <a href="https://youtu.be/vp-hV_li_bk">release video</a> to learn more</li> <li>Download <a href="https://kafka.apache.org/downloads">Apache Kafka 2.8.0</a> to get started with the latest release</li> </ul> This was a huge community effort, so thank you to everyone who contributed to this release, including all our users and our 151 authors and reviewers: 17hao, abc863377, Adem Efe Gencer, akumar, Alexander Iskuskov, Alexandre Dupriez, Almog Gavra, Alok Nikhil, Anastasia Vela, Andrew Choi, Andrey Bozhko, Andrey Falko, Andy Coates, Andy Wilkinson, Ankit Kumar, Anna Povzner, Anna Sophie Blee-Goldman, APaMio, Arjun Satish, ArunParthiban-ST, Attila Sasvari, Benoit Maggi, bertber, Bill Bejeck, Bob Barrett, Boyang Chen, Brajesh Kumar, Brian Byrne, Bruno Cadonna, Cheng Tan, Chia-Ping Tsai, Chris Egerton, CHUN-HAO TANG, Colin Patrick McCabe, Cyrus Vafadari, David Arthur, David Jacot, David Mao, dengziming, Dhruvil Shah, Dima Reznik, Dongjoon Hyun, Dongxu Wang, Edoardo Comar, Emre Hasegeli, Ewen Cheslack-Postava, feyman2016, fml2, Gardner Vickers, Geordie, Govinda Sakhare, Greg Harris, Guozhang Wang, Gwen Shapira, Hamza Slama, high.lee, huxi, Igor Soarez, Ilya Ganelin, Ismael Juma, Israel Ekpo, Ivan Ponomarev, Ivan Yurchenko, jackyoh, Jakob Homan, James Cheng, James Yuzawa, Jason Gustafson, Jesse Gorzinski, Jim Galasyn, John Roesler, Jorge Esteban Quilcate Otoya, José Armando García Sancio, Jose Sancio, Julien Chanaud, Julien Jean Paul Sirocchi, Jun Rao, Justine Olshan, Kengo Seki, Konstantine Karantasis, Kowshik Prakasam, leah, Leah Thomas, Lee Dongjin, Levani Kokhreidze, Lev Zemlyanov, Liju John, limengmonty, Lincong Li, Lucas Bradstreet, Luke Chen, Manikumar Reddy, Marco Aurelio Lotz, mathieu, Matthew Wong, Matthias J. Sax, Matthias Merdes, Michael Bingham, Michael G. Noll, Mickael Maison, Montyleo, mowczare, Nigel Liang, Nikhil Bhatia, Nikolay Izhikov, Ning Zhang, Nitesh Mor, notifygd, Okada Haruki, Oliver Dineen, panguncle, parafiend, Patrick Dignan, Prateek Agarwal, Prithvi, Rajini Sivaram, Raman Verma, Ramesh Krishnan M, Rameshkrishnan Muthusamy, Randall Hauch, Richard Fussenegger, Rohan Desai, Rohit Deshpande, Ron Dagostino, Ryanne Dolan, Samuel Cantero, Sanjana Kaundinya, Sanket Fajage, Satish Duggana, Scott Hendricks, Scott Sugar, Shao Yang Hong, shenwenbing, ssugar, Stanislav Kozlovski, Stanislav Vodetskyi, Taisiia Goltseva, tang7526, Thorsten Hake, Tom Bentley, vamossagar12, Victoria Xia, Viktor Somogyi-Vass, voffcheg109, Walker Carlson, wenbingshen, William Hammond, wycccccc, xakassi, Xavier Léauté, Yilong Chang, zhangyue19921010

_posts/2021-04-19-what-s-new-in-apache5.html (220 lines of code) (raw):