content/2024/08/02/announcing-the-release-of-apache-flink-1.20/index.html

<!DOCTYPE html> <html lang="en" dir=ZgotmplZ> <head> <link rel="stylesheet" href="/bootstrap/css/bootstrap.min.css"> <script src="/bootstrap/js/bootstrap.bundle.min.js"></script> <link rel="stylesheet" type="text/css" href="/font-awesome/css/font-awesome.min.css"> <script src="/js/anchor.min.js"></script> <script src="/js/flink.js"></script> <link rel="canonical" href="https://flink.apache.org/2024/08/02/announcing-the-release-of-apache-flink-1.20/"> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="description" content="The Apache Flink PMC is pleased to announce the release of Apache Flink 1.20.0. As usual, we are looking at a packed release with a wide variety of improvements and new features. Overall, 142 people contributed to this release completing 13 FLIPs and 300+ issues. Thank you! Let’s dive into the highlights. Standing on the Eve of Apache Flink 2.0 # Flink 1.0 was released eight years ago. Since several months, the community is actively planning and taking steps towards the next major release."> <meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Announcing the Release of Apache Flink 1.20" /> <meta property="og:description" content="The Apache Flink PMC is pleased to announce the release of Apache Flink 1.20.0. As usual, we are looking at a packed release with a wide variety of improvements and new features. Overall, 142 people contributed to this release completing 13 FLIPs and 300+ issues. Thank you! Let’s dive into the highlights. Standing on the Eve of Apache Flink 2.0 # Flink 1.0 was released eight years ago. Since several months, the community is actively planning and taking steps towards the next major release." /> <meta property="og:type" content="article" /> <meta property="og:url" content="https://flink.apache.org/2024/08/02/announcing-the-release-of-apache-flink-1.20/" /><meta property="article:section" content="posts" /> <meta property="article:published_time" content="2024-08-02T08:00:00+00:00" /> <meta property="article:modified_time" content="2024-08-02T08:00:00+00:00" /> <title>Announcing the Release of Apache Flink 1.20 | Apache Flink</title> <link rel="manifest" href="/manifest.json"> <link rel="icon" href="/favicon.png" type="image/x-icon"> <link rel="stylesheet" href="/book.min.22eceb4d17baa9cdc0f57345edd6f215a40474022dfee39b63befb5fb3c596b5.css" integrity="sha256-IuzrTRe6qc3A9XNF7dbyFaQEdAIt/uObY777X7PFlrU="> <script defer src="/en.search.min.6950414fdb865fa2a65062656ee52f8a1d13a0578a8866521a5e3501cdc3b475.js" integrity="sha256-aVBBT9uGX6KmUGJlbuUvih0ToFeKiGZSGl41Ac3DtHU="></script>  <meta name="generator" content="Hugo 0.124.1"> <script> var _paq = window._paq = window._paq || []; _paq.push(['disableCookies']); _paq.push(["setDomains", ["*.flink.apache.org","*.nightlies.apache.org/flink"]]); _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="//analytics.apache.org/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '1']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); </script> </head> <body dir=ZgotmplZ> <header> <nav class="navbar navbar-expand-xl"> <div class="container-fluid"> <a class="navbar-brand" href="/"> <img src="/img/logo/png/100/flink_squirrel_100_color.png" alt="Apache Flink" height="47" width="47" class="d-inline-block align-text-middle"> Apache Flink </a> <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation"> </button> <div class="collapse navbar-collapse" id="navbarSupportedContent"> <ul class="navbar-nav"> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">About</a> <ul class="dropdown-menu"> <li> <a class="dropdown-item" href="/what-is-flink/flink-architecture/">Architecture</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/flink-applications/">Applications</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/flink-operations/">Operations</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/use-cases/">Use Cases</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/powered-by/">Powered By</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/roadmap/">Roadmap</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/community/">Community & Project Info</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/security/">Security</a> </li> <li> <a class="dropdown-item" href="/what-is-flink/special-thanks/">Special Thanks</a> </li> </ul> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Getting Started</a> <ul class="dropdown-menu"> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/local_installation/">With Flink </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/docs/try-flink-kubernetes-operator/quick-start/">With Flink Kubernetes Operator </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/">With Flink CDC </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/docs/try-flink-ml/quick-start/">With Flink ML </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/getting-started/project-setup.html">With Flink Stateful Functions </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/overview/">Training Course </a> </li> </ul> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">Documentation</a> <ul class="dropdown-menu"> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-stable/">Flink 2.0 (stable) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-lts/">Flink 1.20 (LTS) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-docs-master/">Flink Master (snapshot) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/">Kubernetes Operator 1.11 (latest) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main">Kubernetes Operator Main (snapshot) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-stable">CDC 3.3 (stable) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-cdc-docs-master">CDC Master (snapshot) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-stable/">ML 2.3 (stable) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-ml-docs-master">ML Master (snapshot) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-stable/">Stateful Functions 3.3 (stable) </a> </li> <li> <a class="dropdown-item" href="https://nightlies.apache.org/flink/flink-statefun-docs-master">Stateful Functions Master (snapshot) </a> </li> </ul> </li> <li class="nav-item dropdown"> <a class="nav-link dropdown-toggle" href="#" role="button" data-bs-toggle="dropdown" aria-expanded="false">How to Contribute</a> <ul class="dropdown-menu"> <li> <a class="dropdown-item" href="/how-to-contribute/overview/">Overview</a> </li> <li> <a class="dropdown-item" href="/how-to-contribute/contribute-code/">Contribute Code</a> </li> <li> <a class="dropdown-item" href="/how-to-contribute/reviewing-prs/">Review Pull Requests</a> </li> <li> <a class="dropdown-item" href="/how-to-contribute/code-style-and-quality-preamble/">Code Style and Quality Guide</a> </li> <li> <a class="dropdown-item" href="/how-to-contribute/contribute-documentation/">Contribute Documentation</a> </li> <li> <a class="dropdown-item" href="/how-to-contribute/documentation-style-guide/">Documentation Style Guide</a> </li> <li> <a class="dropdown-item" href="/how-to-contribute/improve-website/">Contribute to the Website</a> </li> <li> <a class="dropdown-item" href="/how-to-contribute/getting-help/">Getting Help</a> </li> </ul> </li> <li class="nav-item"> <a class="nav-link" href="/posts/">Flink Blog</a> </li> <li class="nav-item"> <a class="nav-link" href="/downloads/">Downloads</a> </li> </ul> <div class="book-search"> <div class="book-search-spinner hidden"> </div> <form class="search-bar d-flex" onsubmit="return false;"su> <input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/"> </form> <div class="book-search-spinner hidden"></div> <ul id="book-search-results"></ul> </div> </div> </div> </nav> <div class="navbar-clearfix"></div> </header> <main class="flex"> <section class="container book-page"> <article class="markdown"> <h1> <a href="/2024/08/02/announcing-the-release-of-apache-flink-1.20/">Announcing the Release of Apache Flink 1.20</a> </h1> August 2, 2024 - Weijie Guo <a href="https://twitter.com/WeijieGuo12">(@WeijieGuo12)</a> Rui Fan <a href="https://twitter.com/1996fanrui">(@1996fanrui)</a> The Apache Flink PMC is pleased to announce the release of Apache Flink 1.20.0. As usual, we are looking at a packed release with a wide variety of improvements and new features. Overall, 142 people contributed to this release completing 13 FLIPs and 300+ issues. Thank you! Let’s dive into the highlights. <h1 id="standing-on-the-eve-of-apache-flink-20"> Standing on the Eve of Apache Flink 2.0 <a class="anchor" href="#standing-on-the-eve-of-apache-flink-20">#</a> </h1> Flink 1.0 was released eight years ago. Since several months, the community is actively planning and taking steps towards the next major release. The new 1.20 release is planned to be the last minor release before Flink 2.0, which is anticipated by the end of 2024. Start from Flink 1.19, the community has decided to officially deprecate multiple APIs that were approaching end of life for a while. In 1.20, we further sorted through all relevant APIs that might need to be replaced or deprecated to clear the way for the 2.0 release: <ul> <li>Configuration Improvements: As Flink moves towards version 2.0, we have revisited all runtime & Table API/SQL configuration options and identified several opportunities to enhance user-friendliness and maintainability.</li> <li>Deprecate the Legacy <a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/api/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.html">SinkFunction</a> API: Since its introduction in Flink 1.12, the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/api/java/org/apache/flink/api/connector/sink2/Sink.html">Unified Sink API</a> has undergone extensive development and testing. Over multiple release cycles, the API has demonstrated stability and robustness, aligning with the criteria set forth in <a href="https://cwiki.apache.org/confluence/x/J5eqCw">FLIP-197</a> for API stability graduation. Therefore, we promote the Unified Sink API v2 to <code>@Public</code> and deprecate the legacy <code>SinkFunction</code> interface.</li> </ul> It has been eight years since the Flink community’s last major release, and we have great expectations for Flink 2.0. We are planning to release several high-impact features in <code>2.x</code>. Some of them are already introduced in Flink 1.20 in MVP (minimum viable product) state and discussed in more detail below. <ul> <li>Introduce a New Materialized Table for Simplifying Data Pipelines: <a href="https://cwiki.apache.org/confluence/x/HYySEQ">FLIP-435</a> designed to simplify the development of data processing pipelines. With dynamic table, uniform SQL statements and freshness, users can define batch and streaming transformations to data in the same way, accelerate ETL pipeline development, and manage task scheduling automatically. See below for more details on this exciting feature.</li> <li>Unified File Merging Mechanism for Checkpoints: The unified file merging mechanism for checkpointing is introduced to Flink 1.20 as an MVP feature, which allows scattered small checkpoint files to be written into larger files, reducing the number of file creations and file deletions and alleviating the pressure of file system metadata management raised by the file flooding problem (large number of small files are created) during checkpoints. For the whole story and more details, please refer to <a href="https://cwiki.apache.org/confluence/x/DwsNDw">FLIP-306</a>.</li> </ul> <h1 id="flink-sql-improvements"> Flink SQL Improvements <a class="anchor" href="#flink-sql-improvements">#</a> </h1> <h2 id="introduce-materialized-tables"> Introduce Materialized Tables <a class="anchor" href="#introduce-materialized-tables">#</a> </h2> In Flink 1.20, We introduced Materialized Tables abstraction in Flink SQL, a new table type designed to simplify both batch and stream data pipelines, while providing a consistent development experience. Materialized tables are defined with a query and a data freshness specification. The engine automatically derives the table schema and creates a data refresh pipeline to maintain the query result with the requested freshness. Users are relieved from the burden of comprehending the concepts and differences between streaming and batch processing, and they do not have to directly maintain Flink streaming or batch jobs. All operations are done on Materialized tables, which can significantly accelerate ETL pipeline development. Here is an example to create a materialized table that is constantly refreshed with a data freshness of <code>3</code> minutes. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql">-- 1. Create table schema and data refresh pipeline CREATE MATERIALIZED TABLE dwd_orders ( PRIMARY KEY(ds, id) NOT ENFORCED ) PARTITIONED BY (ds) FRESHNESS = INTERVAL '3' MINUTE AS SELECT o.ds o.id, o.order_number, o.user_id, ... FROM orders as o LEFT JOIN products FOR SYSTEM_TIME AS OF proctime() AS prod ON o.product_id = prod.id LEFT JOIN order_pay AS pay ON o.id = pay.order_id and o.ds = pay.ds; -- 2. Pause the data refresh pipeline ALTER MATERIALIZED TABLE dwd_orders SUSPEND; -- 3. Resume the data refresh pipeline ALTER MATERIALIZED TABLE dwd_orders RESUME -- Set table option via WITH clause WITH( 'sink.parallesim' = '10' ); -- Refresh historical partition manually ALTER MATERIALIZED TABLE dwd_orders REFRESH PARTITION(ds='20231023'); </code></pre></div>More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-35187">FLINK-35187</a></li> <li><a href="https://cwiki.apache.org/confluence/x/HYySEQ">FLIP-435</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/materialized-table/overview/">Materialized Table Overview</a></li> </ul> <h2 id="introduce-catalog-related-syntax"> Introduce Catalog-Related Syntax <a class="anchor" href="#introduce-catalog-related-syntax">#</a> </h2> With the growing adoption of Flink SQL, implementations of Flink’s <code>Catalog</code> interface play an increasingly important role. Today, Flink features a JDBC and a Hive catalog implementation and other open source projects such as Apache Paimon integrate with this interface as well. Now in Flink 1.20, you can use the <code>DQL</code> syntax to obtain detailed metadata from existing catalogs, and the <code>DDL</code> syntax to modify metadata such as properties or comment in the specified catalog. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql">Flink SQL> CREATE CATALOG `cat` WITH ('type'='generic_in_memory', 'default-database'='db'); [INFO] Execute statement succeeded. Flink SQL> SHOW CREATE CATALOG `cat`; +---------------------------------------------------------------------------------------------+ | result | +---------------------------------------------------------------------------------------------+ | CREATE CATALOG `cat` WITH ( 'default-database' = 'db', 'type' = 'generic_in_memory' ) | +---------------------------------------------------------------------------------------------+ 1 row in set Flink SQL> DESCRIBE CATALOG `cat`; +-----------+-------------------+ | info name | info value | +-----------+-------------------+ | name | cat | | type | generic_in_memory | | comment | | +-----------+-------------------+ 3 rows in set Flink SQL> ALTER CATALOG `cat` SET ('default-database'='new-db'); [INFO] Execute statement succeeded. Flink SQL> SHOW CREATE CATALOG `cat`; +-------------------------------------------------------------------------------------------------+ | result | +-------------------------------------------------------------------------------------------------+ | CREATE CATALOG `cat` WITH ( 'default-database' = 'new-db', 'type' = 'generic_in_memory' ) | +-------------------------------------------------------------------------------------------------+ 1 row in set </code></pre></div>More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-34914">FLINK-34914</a></li> <li><a href="https://cwiki.apache.org/confluence/x/xAmpEQ">FLIP-436</a></li> </ul> <h2 id="add-distributed-by-clause"> Add DISTRIBUTED BY Clause <a class="anchor" href="#add-distributed-by-clause">#</a> </h2> Many SQL engines expose the concepts of <code>Partitioning</code>, <code>Bucketing</code>, or <code>Clustering</code>. We propose to introduce the concept of <code>Bucketing</code> to Flink. Buckets enable load balancing in an external storage system by splitting data into disjoint subsets. It depends heavily on the semantics of the underlying connector. However, a user can influence the bucketing behavior by specifying the number of buckets, the distribution algorithm, and (if the algorithm allows it) the columns which are used for target bucket calculation. All bucketing components (i.e. bucket number, distribution algorithm, bucket key columns) are optional in the SQL syntax. Take the following SQL statements as an example: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql">-- declares a hash function on a fixed number of 4 buckets (i.e. HASH(uid) % 4 = target bucket). CREATE TABLE MyTable (uid BIGINT, name STRING) DISTRIBUTED BY HASH(uid) INTO 4 BUCKETS; -- leaves the selection of an algorithm up to the connector. CREATE TABLE MyTable (uid BIGINT, name STRING) DISTRIBUTED BY (uid) INTO 4 BUCKETS; -- leaves the number of buckets up to the connector. CREATE TABLE MyTable (uid BIGINT, name STRING) DISTRIBUTED BY (uid); -- only defines the number of buckets. CREATE TABLE MyTable (uid BIGINT, name STRING) DISTRIBUTED INTO 4 BUCKETS; </code></pre></div>More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-33494">FLINK-33494</a></li> <li><a href="https://cwiki.apache.org/confluence/x/loxEE">FLIP-376</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/create/#distributed">Documentation</a></li> </ul> <h1 id="state--checkpoint-improvements"> State & Checkpoint Improvements <a class="anchor" href="#state--checkpoint-improvements">#</a> </h1> <h2 id="unified-file-merging-mechanism-for-checkpoints"> Unified File Merging Mechanism for Checkpoints <a class="anchor" href="#unified-file-merging-mechanism-for-checkpoints">#</a> </h2> The unified file merging mechanism for checkpointing is introduced to Flink 1.20 as an MVP (“minimum viable product”) feature. It combines multiple small checkpoint files into fewer larger files, which reduces the number of file creation and file deletion operations and alleviates the pressure of file system metadata management during checkpoints. The mechanism can be enabled by setting <code>execution.checkpointing.file-merging.enabled</code> to <code>true</code>. For more advanced options and the principles behind this feature, please refer to the Checkpointing documentation. More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-32070">FLINK-33494</a></li> <li><a href="https://cwiki.apache.org/confluence/x/DwsNDw">FLIP-306</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/checkpointing/#unify-file-merging-mechanism-for-checkpoints-experimental">Documentation</a></li> </ul> <h2 id="compaction-of-small-sst-files"> Compaction of Small SST Files <a class="anchor" href="#compaction-of-small-sst-files">#</a> </h2> In some cases, the number of files produced by the RocksDB state backend grows indefinitely. In addition to the overhead caused by many small files, this behavior can cause the task state info to exceed the RPC message size limit and therefore lead to recovery or checkpoint failures. From release 1.20 on, Flink can merge such files in the background using the RocksDB API. More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-26050">FLINK-26050</a></li> </ul> <h1 id="batch-processing-improvements"> Batch Processing Improvements <a class="anchor" href="#batch-processing-improvements">#</a> </h1> <h2 id="support-job-recovery-from-jobmaster-failures-for-batch-jobs"> Support Job Recovery from JobMaster Failures for Batch Jobs <a class="anchor" href="#support-job-recovery-from-jobmaster-failures-for-batch-jobs">#</a> </h2> In Flink 1.20, we introduced a batch job recovery mechanism to enable batch jobs to recover as much progress as possible after a <code>JobMaster</code> failover, avoiding the need to rerun tasks that have already finished. More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-33892">FLINK-33892</a></li> <li><a href="https://cwiki.apache.org/confluence/x/QwqZE">FLIP-383</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/ops/batch/recovery_from_job_master_failure/">Documentation</a></li> </ul> <h2 id="support-dynamic-parallelism-inference-for-hivesource"> Support Dynamic Parallelism Inference for HiveSource <a class="anchor" href="#support-dynamic-parallelism-inference-for-hivesource">#</a> </h2> In Flink 1.20, we added support for dynamic source parallelism inference in batch jobs for the <code>Hive</code> source connector. This allows the connector to dynamically determine parallelism based on the actual partitions with dynamic partition pruning. Additionally, we have introduced a new configuration option - <code>table.exec.hive.infer-source-parallelism.mode</code> to enable users to choose between static and dynamic inference modes for source parallelism. It should be noted that in Flink 1.20, the previous configuration option <code>table.exec.hive.infer-source-parallelism</code> has been marked as deprecated, but it will continue to serve as a switch for automatic parallelism inference until it is fully phased out. More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-35293">FLINK-35293</a></li> <li><a href="https://cwiki.apache.org/confluence/x/Hgr9EQ">FLIP-445</a></li> </ul> <h1 id="datastream-api-improvements"> DataStream API Improvements <a class="anchor" href="#datastream-api-improvements">#</a> </h1> The <code>DataSet</code> API has been already formally deprecated and will be removed in the Flink 2.0 version. Flink users are recommended to migrate from the <code>DataSet</code> API to the <code>DataStream</code> API, <code>Table</code> API and <code>SQL</code> for their data processing requirements. <h2 id="support-full-partition-processing-on-datastream-api"> Support Full Partition Processing on DataStream API <a class="anchor" href="#support-full-partition-processing-on-datastream-api">#</a> </h2> Before 1.20, the <code>DataStream</code> API did not directly support aggregations on non-keyed streams (subtask-scope aggregations). As a workaround, users could assign the subtask id to the records before turning the stream into a keyed stream which incurred additional overhead. Flink 1.20 adds built-in support for these operations via the <code>FullPartitionWindow</code> API. Suppose we want to count the number of records in each partition and output to a downstream operator. This can be done as follows: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java">inputStream.fullWindowPartition() .mapPartition( new MapPartitionFunction<Record, Long>() { @Override public void mapPartition( Iterable<Record> values, Collector<Long> out) throws Exception { long counter = 0; for (Record value : values) { counter++; } out.collect(counter)); } }) </code></pre></div>More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-34543">FLINK-34543</a></li> <li><a href="https://cwiki.apache.org/confluence/x/0gt1E">FLIP-380</a></li> </ul> <h1 id="important-configuration-changes-for-flink-20"> Important Configuration Changes for Flink 2.0 <a class="anchor" href="#important-configuration-changes-for-flink-20">#</a> </h1> As Apache Flink progresses to version 2.0, several configuration options are being changed or deprecated to improve ease-of-use and maintainability. <h2 id="using-proper-type-for-configuration-options"> Using Proper Type for Configuration Options <a class="anchor" href="#using-proper-type-for-configuration-options">#</a> </h2> Some configuration options like <code>client.heartbeat.interval</code> has been updated to the <code>Duration</code> type in a backward-compatible manner. The full list is available in <a href="https://issues.apache.org/jira/browse/FLINK-35359">FLINK-35359</a>. The following configuration options have been updated to the <code>Enum</code> type in a backward-compatible manner: <ul> <li><code>taskmanager.network.compression.codec</code></li> <li><code>table.optimizer.agg-phase-strategy</code></li> </ul> The following configuration options have been updated to the <code>Int</code> type in a backward-compatible manner: <ul> <li><code>yarn.application-attempts</code></li> </ul> More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-35359">FLINK-35359</a></li> </ul> <h2 id="deprecate-multiple-configuration-options"> Deprecate Multiple Configuration Options <a class="anchor" href="#deprecate-multiple-configuration-options">#</a> </h2> In preparation for the release of Flink 2.0, the community has decided to officially deprecate multiple configuration options that were approaching end of life for a while. The following configuration options have been deprecated and will be removed in Flink 2.0 as we are phasing out the hash-based blocking shuffle: <ul> <li><code>taskmanager.network.sort-shuffle.min-parallelism</code></li> <li><code>taskmanager.network.blocking-shuffle.type</code></li> </ul> The following configuration options have been deprecated and will be removed in Flink 2.0 as we are phasing out the legacy hybrid shuffle: <ul> <li><code>taskmanager.network.hybrid-shuffle.spill-index-region-group-size</code></li> <li><code>taskmanager.network.hybrid-shuffle.num-retained-in-memory-regions-max</code></li> <li><code>taskmanager.network.hybrid-shuffle.enable-new-mode</code></li> </ul> The following configuration options have been deprecated to simply the configuration of network buffers: <ul> <li><code>taskmanager.network.memory.buffers-per-channel</code>(will be removed in Flink 2.0)</li> <li><code>taskmanager.network.memory.floating-buffers-per-gate</code>(will be removed in Flink 2.0)</li> <li><code>taskmanager.network.memory.max-buffers-per-channel</code>(will be removed in Flink 2.0)</li> <li><code>taskmanager.network.memory.max-overdraft-buffers-per-gate</code>(will be removed in Flink 2.0)</li> <li><code>taskmanager.network.memory.exclusive-buffers-request-timeout-ms</code> (Please use <code>taskmanager.network.memory.buffers-request-timeout</code> instead)</li> </ul> The configuration option <code>taskmanager.network.batch-shuffle.compression.enabled</code> has been deprecated. Please set <code>taskmanager.network.compression.codec</code> to <code>NONE</code> to disable compression. The following Netty-related configuration options are no longer recommended for use and have been deprecated, we will remove them in Flink 2.0: <ul> <li><code>taskmanager.network.netty.num-arenas</code></li> <li><code>taskmanager.network.netty.server.numThreads</code></li> <li><code>taskmanager.network.netty.client.numThreads</code></li> <li><code>taskmanager.network.netty.server.backlog</code></li> <li><code>taskmanager.network.netty.sendReceiveBufferSize</code></li> <li><code>taskmanager.network.netty.transport</code></li> </ul> The following configuration options are unnecessary and have been deprecated: <ul> <li><code>taskmanager.network.max-num-tcp-connections</code>(will be removed and hard-coded to <code>1</code> in Flink 2.0)</li> <li><code>fine-grained.shuffle-mode.all-blocking</code>(will be removed in Flink 2.0)</li> </ul> These options were previously used for fine-tuning TPC testing but are no longer needed by the current Flink planner and will be removed in Flink 2.0: <ul> <li><code>table.exec.range-sort.enabled</code></li> <li><code>table.optimizer.rows-per-local-agg</code></li> <li><code>table.optimizer.join.null-filter-threshold</code></li> <li><code>table.optimizer.semi-anti-join.build-distinct.ndv-ratio</code></li> <li><code>table.optimizer.shuffle-by-partial-key-enabled</code></li> <li><code>table.optimizer.smj.remove-sort-enabled</code></li> <li><code>table.optimizer.cnf-nodes-limit</code></li> </ul> These options were introduced for the now-obsolete <code>FilterableTableSource</code> interface and will be removed in Flink 2.0: <ul> <li><code>table.optimizer.source.aggregate-pushdown-enabled</code></li> <li><code>table.optimizer.source.predicate-pushdown-enabled</code></li> </ul> The configuration option <code>sql-client.display.max-column-width</code> has been deprecated. Please use <code>table.display.max-column-width</code> instead. More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-35461">FLINK-35461</a></li> <li><a href="https://issues.apache.org/jira/browse/FLINK-35473">FLINK-35473</a></li> </ul> <h2 id="new-and-updated-configuration-options"> New and Updated Configuration Options <a class="anchor" href="#new-and-updated-configuration-options">#</a> </h2> <h3 id="state--checkpointing-options"> State & Checkpointing Options <a class="anchor" href="#state--checkpointing-options">#</a> </h3> In Flink 1.20, all the options about state and checkpointing are reorganized and categorized by prefixes as listed below: <ol> <li><code>execution.checkpointing.*</code>: all configuration options associated with checkpointing and savepoints.</li> <li><code>execution.state-recovery.*</code>: all configuration options related to state recovery.</li> <li><code>state.*</code>: all configuration options related to the state accessing. a. <code>state.backend.*</code>: configuration options for individual state backends, such as RocksDB. b. <code>state.changelog.*</code>: configuration options for the state changelog, as outlined in FLIP-158, including the options for the “Durable Short-term Log” (DSTL). c. <code>state.latency-track.*</code>: configuration options related to the latency tracking of state access.</li> </ol> <h3 id="batch-execution-options"> Batch Execution Options <a class="anchor" href="#batch-execution-options">#</a> </h3> The following options have been moved from <code>org.apache.flink.table.planner.codegen.agg.batch.HashAggCodeGenerator</code> to <code>org.apache.flink.table.api.config</code> and promoted to <code>@PublicEvolving</code>. <ul> <li><code>table.exec.local-hash-agg.adaptive.enabled</code></li> <li><code>table.exec.local-hash-agg.adaptive.sampling-threshold</code></li> <li><code>table.exec.local-hash-agg.adaptive.distinct-value-rate-threshold</code></li> </ul> <h3 id="lookup-hint-options"> Lookup Hint Options <a class="anchor" href="#lookup-hint-options">#</a> </h3> The following options have been moved from <code>org.apache.flink.table.planner.hint.LookupJoinHintOptions</code> to <code>org.apache.flink.table.api.config.LookupJoinHintOptions</code> and promoted to <code>@PublicEvolving</code>. <ul> <li><code>table</code></li> <li><code>async</code></li> <li><code>output-mode</code></li> <li><code>capacity</code></li> <li><code>timeout</code></li> <li><code>retry-predicate</code></li> <li><code>retry-strategy</code></li> <li><code>fixed-delay</code></li> <li><code>max-attempts</code></li> </ul> <h3 id="optimizer-options"> Optimizer Options <a class="anchor" href="#optimizer-options">#</a> </h3> The following options have been moved from <code>org.apache.flink.table.planner.plan.optimize.RelNodeBlock</code> to <code>org.apache.flink.table.api.config.OptimizerConfigOptions</code> and promoted to <code>@PublicEvolving</code> <ul> <li><code>table.optimizer.union-all-as-breakpoint-enabled</code></li> <li><code>table.optimizer.reuse-optimize-block-with-digest-enabled</code></li> </ul> <h3 id="aggregate-optimizer-option"> Aggregate Optimizer Option <a class="anchor" href="#aggregate-optimizer-option">#</a> </h3> <code>table.optimizer.incremental-agg-enabled</code> has been moved from <code>org.apache.flink.table.planner.plan.rules.physical.stream.IncrementalAggregateRule</code> to <code>org.apache.flink.table.api.config.OptimizerConfigOptions</code> and promoted to <code>@PublicEvolving</code> More Information <ul> <li><a href="https://issues.apache.org/jira/browse/FLINK-35461">FLINK-35461</a></li> <li><a href="https://issues.apache.org/jira/browse/FLINK-35473">FLINK-35473</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/#checkpointing">Checkpointing Options</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/#recovery">Recovery Options</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/#state-backends">State Backend Options</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/#state-changelog-options">State Changelog Options</a></li> <li><a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/#state-latency-tracking-options">Latency-track Options</a></li> </ul> <h1 id="upgrade-notes"> Upgrade Notes <a class="anchor" href="#upgrade-notes">#</a> </h1> The Flink community tries to ensure that upgrades are as seamless as possible. However, certain changes may require users to make adjustments to certain parts of the program when upgrading to version 1.20. Please refer to the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.20/release-notes/flink-1.20/">release notes</a> for a comprehensive list of adjustments to make and issues to check during the upgrading process. <h1 id="list-of-contributors"> List of Contributors <a class="anchor" href="#list-of-contributors">#</a> </h1> The Apache Flink community would like to express gratitude to all the contributors who made this release possible: Ahmed Hamdy, Alan Sheinberg, Aleksandr Pilipenko, Alexander Fedulov, Andrey Gaskov, Antonio Vespoli, Anupam Aggarwal, Barak Ben-Nathan, Benchao Li, Brad, Cheng Pan, Chesnay Schepler, DamonXue, Danny Cranmer, David Christle, David Moravek, David Schlosnagle, Dawid Wysakowicz, Dian Fu, Dmitriy Linevich, Elphas Toringepi, Emre Kartoglu, Fang Yong, Feng Jin, Ferenc Csaky, Frank Yin, Gabor Somogyi, Gyula Fora, HCTommy, Hangxiang Yu, Hanyu Zheng, Hao Li, Hong Liang Teoh, Hong Teoh, HuangXingBo, Jacky Lau, James Hughes, Jane Chan, Jeyhun Karimov, Jiabao Sun, Jim Hughes, Jing Ge, Jinzhong Li, JunRuiLee, Juntao Hu, JustinLee, Kartikey Pant, Kumar Mallikarjuna, Leonard Xu, Lorenzo Affetti, Luke Chen, Martijn Visser, Mason Chen, Matthias Pohl, Mingliang Liu, Panagiotis Garefalakis, Peter Huang, Peter Vary, Piotr Nowojski, Puneet Duggal, Qinghui Xu, Qingsheng Ren, Ravi Dutt Singh, Robert Metzger, Robert Young, Roc Marshal, Roman, Roman Boyko, Roman Khachatryan, Ron, Rui Fan, Ryan Skraba, Samrat, Sergey Nuyanzin, Shilun Fan, Stefan Richter, SuDewei, Timo Walther, Ufuk Celebi, Vincent Woo, Wang FeiFan, Weijie Guo, Wencong Liu, Wouter Zorgdrager, Xiangyu Feng, Xintong Song, Xuyang, Yanfei Lei, Yangze Guo, Yu Chen, Yubin Li, Yuepeng Pan, Yun Tang, Yuxin Tan, Zakelly, Zhanghao Chen, Zhen Wang, Zhenqiu Huang, Zhu Zhu, Zmm, ammar-master, anupamaggarwal, bvarghese1, caicancai, caodizhou, chenzihao, drymatini, dsaisharath, eason.qin, elon-X, fengli, gongzhongqiang, hejufang, jectpro7, jiangxin, liming.1018, lincoln lee, liuyongvs, lxliyou001, oleksandr.nitavskyi, plugatarev, rmoff, slfan1989, spoon-lz, sunxia, sxnan, sychen, wforget, xiaogang, xingbo, yebukong, yunfengzhou-hub, yunhong, zhouyisha, 马越 </article> <div class="edit-this-page"> <a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications">Want to contribute translation?</a> <a href="//github.com/apache/flink-web/edit/asf-site/docs/content/posts/2024-08-02-release-1.20.0.md"> Edit This Page </a> </div> </section> <aside class="book-toc"> <nav id="TableOfContents"><h3>On This Page <a href="javascript:void(0)" class="toc" onclick="collapseToc()"></a></h3> <ul> <li><a href="#standing-on-the-eve-of-apache-flink-20">Standing on the Eve of Apache Flink 2.0</a></li> <li><a href="#flink-sql-improvements">Flink SQL Improvements</a> <ul> <li><a href="#introduce-materialized-tables">Introduce Materialized Tables</a></li> <li><a href="#introduce-catalog-related-syntax">Introduce Catalog-Related Syntax</a></li> <li><a href="#add-distributed-by-clause">Add DISTRIBUTED BY Clause</a></li> </ul> </li> <li><a href="#state--checkpoint-improvements">State & Checkpoint Improvements</a> <ul> <li><a href="#unified-file-merging-mechanism-for-checkpoints">Unified File Merging Mechanism for Checkpoints</a></li> <li><a href="#compaction-of-small-sst-files">Compaction of Small SST Files</a></li> </ul> </li> <li><a href="#batch-processing-improvements">Batch Processing Improvements</a> <ul> <li><a href="#support-job-recovery-from-jobmaster-failures-for-batch-jobs">Support Job Recovery from JobMaster Failures for Batch Jobs</a></li> <li><a href="#support-dynamic-parallelism-inference-for-hivesource">Support Dynamic Parallelism Inference for HiveSource</a></li> </ul> </li> <li><a href="#datastream-api-improvements">DataStream API Improvements</a> <ul> <li><a href="#support-full-partition-processing-on-datastream-api">Support Full Partition Processing on DataStream API</a></li> </ul> </li> <li><a href="#important-configuration-changes-for-flink-20">Important Configuration Changes for Flink 2.0</a> <ul> <li><a href="#using-proper-type-for-configuration-options">Using Proper Type for Configuration Options</a></li> <li><a href="#deprecate-multiple-configuration-options">Deprecate Multiple Configuration Options</a></li> <li><a href="#new-and-updated-configuration-options">New and Updated Configuration Options</a> <ul> <li><a href="#state--checkpointing-options">State & Checkpointing Options</a></li> <li><a href="#batch-execution-options">Batch Execution Options</a></li> <li><a href="#lookup-hint-options">Lookup Hint Options</a></li> <li><a href="#optimizer-options">Optimizer Options</a></li> <li><a href="#aggregate-optimizer-option">Aggregate Optimizer Option</a></li> </ul> </li> </ul> </li> <li><a href="#upgrade-notes">Upgrade Notes</a></li> <li><a href="#list-of-contributors">List of Contributors</a></li> </ul> </nav> </aside> <aside class="expand-toc hidden"> <a class="toc" onclick="expandToc()" href="javascript:void(0)"> </a> </aside> </main> <footer> <div class="separator"></div> <div class="panels"> <div class="wrapper"> <div class="panel"> <ul> <li> <a href="https://flink-packages.org/">flink-packages.org</a> </li> <li> <a href="https://www.apache.org/">Apache Software Foundation</a> </li> <li> <a href="https://www.apache.org/licenses/">License</a> </li> <li> <a href="/zh/">  中文版 </a> </li> </ul> </div> <div class="panel"> <ul> <li> <a href="/what-is-flink/security">Security</a--> </li> <li> <a href="https://www.apache.org/foundation/sponsorship.html">Donate</a> </li> <li> <a href="https://www.apache.org/foundation/thanks.html">Thanks</a> </li> </ul> </div> <div class="panel icons"> <div> <a href="/posts"> <div class="icon flink-blog-icon"></div> Flink blog </a> </div> <div> <a href="https://github.com/apache/flink"> <div class="icon flink-github-icon"></div> Github </a> </div> <div> <a href="https://twitter.com/apacheflink"> <div class="icon flink-twitter-icon"></div> Twitter </a> </div> </div> </div> </div> <hr/> <div class="container disclaimer"> The contents of this website are © 2024 Apache Software Foundation under the terms of the Apache License v2. Apache Flink, Flink, and the Flink logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. </div> </footer> </body> </html>

content/2024/08/02/announcing-the-release-of-apache-flink-1.20/index.html (931 lines of code) (raw):