_posts/2019-08-22-hadoop-community-meetup-beijing-aug.html

--- layout: post status: PUBLISHED published: true title: Hadoop Community Meetup @ Beijing, Aug 2019 id: 5b18f0f0-6985-41b9-8357-cff113cb763d date: '2019-08-22 09:50:18 -0400' categories: hadoop tags: - meetup - hadoop permalink: hadoop/entry/hadoop-community-meetup-beijing-aug --- <html> <head> <meta http-equiv=Content-Type content="text/html; charset=utf-8"> <meta name=Generator content="Microsoft Word 15 (filtered)"> <style>  </style> </head> <body lang=EN-US> <div class=WordSection1> Hadoop Community Meetup @ Beijing <h4> Author: Junping Du (Tencent) & Wangda Tan (Cloudera) </h4> <a href="https://blogs.apache.org/hadoop/mediaresource/fdaef52f-146c-46c3-b780-b501a7302d94"><img src="https://blogs.apache.org/hadoop/mediaresource/fdaef52f-146c-46c3-b780-b501a7302d94" alt="Picture01.jpg"></img></a> <a href="https://blogs.apache.org/hadoop/mediaresource/581538b6-3a43-49c0-b092-eaede2739c03"><img src="https://blogs.apache.org/hadoop/mediaresource/581538b6-3a43-49c0-b092-eaede2739c03" alt="Picture02.jpg"></img></a> <h1>Overview</h1> On Aug 11th 2019, Hadoop developers/users gathered together at Tencent’s Sigma Center office in Beijing to share their latest works, with 12 presentations by engineers from Tencent, Cloudera, Alibaba, Didi, Xiaomi, Meituan, ByteDance (Parent company of TikTok, Toutiao, etc.), JD.com, Huawei. This is also first Hadoop community meetup hosted by Apache Hadoop PMC members. <h2>Attendees</h2> We received tremendous numbers of participations to the meetup. There’re 200 spots available for registration to attend this meetup in-person, and spots got fully booked in 10 mins. We got 150+ attendees in-person, and 3000+ attendees participated online live sessions. We have participants from dozens of different companies and universities in person, many of them are flying from Shanghai, Hangzhou, Shenzhen and even San Francisco Bay Area! <h1>Sessions</h1> <h2>1. Hadoop Community Update And Roadmaps</h2> Junping Du @ Tencent and Wangda Tan @ Cloudera talked about Hadoop community updates and roadmaps. <a href="https://blogs.apache.org/hadoop/mediaresource/f7f923fb-95a2-4da9-ab67-0647152a1532"><img src="https://blogs.apache.org/hadoop/mediaresource/f7f923fb-95a2-4da9-ab67-0647152a1532" alt="Picture03.jpg"></img></a> <span style='font-family:-webkit-standard;color:black'>Junping Du @ Tencent   <a href="https://blogs.apache.org/hadoop/mediaresource/46a5a5e5-285b-487e-b7d1-4dbc8f055d7c"><img src="https://blogs.apache.org/hadoop/mediaresource/46a5a5e5-285b-487e-b7d1-4dbc8f055d7c" alt="Picture04.jpg"></img></a> <span style='font-family:"Times New Roman",serif'>Wangda Tan @ Cloudera Junping introduced recent trends in the storage field, such as better scalability and moving to cloud. He talked about features like RBF (Router Based Federation), improvements of NameNode scalability, Improvements of cloud connectors and Ozone. Wangda talked about recent trends in the compute field, such as better scalability, moving to clkoud-native environment, containerization works and support of Machine-Learning use cases. He talked about global scheduling framework for better scheduling throughput and placement quality. Recent containerization works in YARN such as runc, interactive docker shell. And YARN-on-cloud initiatives from community such as autoscaling, graceful decommissions, etc. Wangda also talked about Submarine and its release plans. At last, Wangda looked back at releases in 2018/2019, and shared tentative release plan of Hadoop in 2019. Such as 3.1.3, 3.2.1 and what’s new coming to 3.3.0. <h2>2. Ozone: Hadoop native object store </h2> Sammi (Yi) Chen @ Tencent talked about native object store project from Hadoop community. <a href="https://blogs.apache.org/hadoop/mediaresource/7b5c8dc2-d493-4883-ac0a-b2330e3a4314"><img src="https://blogs.apache.org/hadoop/mediaresource/7b5c8dc2-d493-4883-ac0a-b2330e3a4314" alt="Picture05.jpg"></img></a> Ozone is a strong-consistent distributed object store service. Like HDFS, Ozone has same level of reliability, consistency and usability. It supports S3 interface, so it is not only useful to on-prem big-data workload. It is also a good option to move big data to cloud. Sammi talked about architecture of Ozone, and what’s new in Ozone 0.5 release. <h2>3. YARN 3.x in Alibaba </h2> Tao Yang from Alibaba talked about Hadoop use cases in Alibaba. He also talked about how new features in YARN 3.x being used to solve use cases. Tao talked about features like preemption, scheduling, resource over-commitment, scheduling diagnostic, mixed deployment of online/offline workload. Tao also talked about how new features in YARN help to better run Apache Flink on YARN. <a href="https://blogs.apache.org/hadoop/mediaresource/c4b7a61e-4b32-40cd-b4a0-4e7b3a6693b6"><img src="https://blogs.apache.org/hadoop/mediaresource/c4b7a61e-4b32-40cd-b4a0-4e7b3a6693b6" alt="Picture06.jpg"></img></a> Tao talked about many interesting features such as MultiNodeLookupPolicy, which can help schedule jobs on a pluggable node sorter. <h2>4. HDFS Best Practices learned from Didi’s production environment. </h2> Hui Fei from Didi talked about HDFS best practices learned from Didi’s large scale (hundreds of PBs) production environment. Hui first talked about storage use cases and scale in Didi’s environment. Then Hui talked about functionalities and improvements Didi’s Hadoop team built on top of Hadoop HDFS 2.7.2 such as: Security, NameNode Federation, Balancer, etc. <a href="https://blogs.apache.org/hadoop/mediaresource/57b9a051-9e6b-4616-b1ca-dcdab17cd42f"><img src="https://blogs.apache.org/hadoop/mediaresource/57b9a051-9e6b-4616-b1ca-dcdab17cd42f" alt="Picture07.jpg"></img></a> Hui also talked about the status of upgrading production cluster based on Hadoop 2.7.2 to Hadoop 3.2.0. The primary driver of upgrade is to save storage spaces. Didi wants to use features like Erasure Coding in Hadoop 3.x. Didi has upgraded a test cluster (100+ nodes) from 2.7.2 to 3.2.0, has a backup cluster with 2k+ nodes run Hadoop 3.1.1 and will rolling upgrade it to 3.2.0. There’s a primary cluster with 10K+ nodes (with 5 namespaces), will start to upgrade to 3.2.0 starting Oct <h2>5. Submarine: A one-stop, cross-platform machine learning platform</h2> Xun Liu @ NetEase and Zhankun Tang @ Cloudera talked about background, existing status and future of Submarine project. <a href="https://blogs.apache.org/hadoop/mediaresource/06051afb-2491-45d3-bade-4c3a8fdbcc4e"><img src="https://blogs.apache.org/hadoop/mediaresource/06051afb-2491-45d3-bade-4c3a8fdbcc4e" alt="Picture08.jpg"></img></a> <span style='font-family:-webkit-standard;color:black'>Zhankun Tang @ Cloudera <span style='font-family:"Times New Roman",serif'>  <a href="https://blogs.apache.org/hadoop/mediaresource/a0087efe-b086-4b75-bb36-03518dcaeb3d"><img src="https://blogs.apache.org/hadoop/mediaresource/a0087efe-b086-4b75-bb36-03518dcaeb3d" alt="Picture09.jpg"></img></a> <span style='font-family:"Times New Roman",serif'>Xun Liu @ Netease Machine learning includes many components like data-preprocessing, feature extraction, model training/serving/management, distributed workload management. Submarine project started by Hadoop community is targeted to achieve these goals by focusing on Notebook experiences. With Submarine, data scientists or machine learning engineer don’t need to understand lower-level platform such as YARN, K8s, Docker container. Zhankun showed a new feature called mini-submarine which allows developers try Submarine locally without installing a YARN cluster. Xun did demos for: <span style='font-family:"Noto Sans Symbols"'>●      Integration of Submarine + Zeppelin notebook. <span style='font-family:"Noto Sans Symbols"'>●      New Submarine web UI to allow data scientists<span style='color:black'> to run jobs and manage models, etc. in the unified user experiences. Xun also talked about companies which are reported using Submarine in production. Such as NetEase, Linkedin, Dahua, Ke.com, JD.com. <h2>6. Hadoop Improvements in Xiaomi</h2> Chen Zhang and Kang Zhou from Xiaomi talked about how Hadoop is being used in Xiaomi. They talked about improvements of HDFS’s performance and scalability; Problems/Solutions when trying to platformize YARN. For HDFS side, Chen talked about their improvements of HDFS federation, such as lower the business impact when upgrading single NameNode to federated NameNode. They have also improved NameNode Performance, which now allows supporting 600 millions of objects (files + blocks) in a single NameNode. <a href="https://blogs.apache.org/hadoop/mediaresource/8e597b99-fb73-4e6a-9f1d-4211e1e58794"><img src="https://blogs.apache.org/hadoop/mediaresource/8e597b99-fb73-4e6a-9f1d-4211e1e58794" alt="Picture10.jpg"></img></a> In YARN, Kang talked about usability improvements in YARN. Such as RMStateStore/History Server, etc. Also, he talked about multi-cluster management tools such as a unified client/RM-UI for multiple clusters. Kang also talked about improvements they have done for scheduling optimization like cache Resource Usage, improvements of utilization and preemption, etc. <h2>7. Key Customizations of YARN @ ByteDance </h2> Yakun Li from ByteDance talked customizations of their YARN cluster to handle extra large scale, multi-clusters environment, Including: utilization improvements, stabilization, optimizations for streaming/model-training environment, and multi datacenter issues, etc. <a href="https://blogs.apache.org/hadoop/mediaresource/f217e756-cb1c-4e0c-955b-434e0b4d5dae"><img src="https://blogs.apache.org/hadoop/mediaresource/f217e756-cb1c-4e0c-955b-434e0b4d5dae" alt="Picture11.jpg"></img></a> For scheduling, Yakun also talked about how they implement Gang Scheduling in YARN, which do scheduling for application instead of node. And it can achieve low-latency, hard/soft constraints. He also talked about implementation of multi-thread version FairScheduler which can push number of container allocation per second up to 3k. In mixed-workloads (Batch, Streaming, ML) deployment part, Yakun talked about they have adopted Docker on YARN support to isolate dependencies. Support CPUSET/NUMA, temporarily skip nodes which have too high physical utilizations, etc. All these efforts can help mixed workload runs well in same cluster. <h2>8. YuniKorn: A New Unified Scheduler for Both YARN and K8s </h2> Weiwei Yang and Wangda Tan from Cloudera talked about their works about a new scheduler named YuniKorn (<a href="https://github.com/cloudera/yunikorn-core">https://github.com/cloudera/yunikorn-core</a>) and how it can benefit both YARN and K8s community. <a href="https://blogs.apache.org/hadoop/mediaresource/22b6c49d-62eb-4361-88fc-383f4c0dd430"><img src="https://blogs.apache.org/hadoop/mediaresource/22b6c49d-62eb-4361-88fc-383f4c0dd430" alt="Picture12.jpg"></img></a> <span style='font-family:"Times New Roman",serif'>Weiwei Yang (Right) and Wangda Tan (Left) from Cloudera Scheduler of a container orchestration system, such as YARN and Kubernetes, is a critical component that users rely on to plan resources and manage applications. They have different characters to support different workloads. YARN schedulers are optimized for high-throughput, multi-tenant batch workloads. It can scale up to 50k nodes per cluster, and schedule 20k containers per second; On the other side, Kubernetes schedulers are optimized for long-running services, but many features like hierarchical queues, fairness resource sharing, and preemption etc, are either missing or not mature enough at this point of time. However, underneath they are responsible for one same job: the decision maker for resource allocations. They mentioned the need to run services on YARN as well as run jobs on Kubernetes. This motivates them to create a universal scheduler which can work for both YARN and Kubernetes and configured in the same way. In this talk, Weiwei and Wangda talked about their efforts of design and implement the universal scheduler. They have integrated it with to Kubernetes already and YARN integration is working-in-progress. This scheduler brings long-wanted features such as hierarchical queues, fairness between users/jobs/queues, preemption to Kubernetes; and it brings service scheduling enhancements to YARN. Most importantly, it provides the opportunity to let YARN and Kubernetes share the same user experience on scheduling big data workloads. And any improvements of this universal scheduler can benefit both Kubernetes and YARN community. <h2>9. HDFS cluster improvements and optimization practices in Meituan Dianping </h2> Xiaoqiao He from Meituan Dianping talked about Hadoop cluster scalabilities now. Their Hadoop cluster keep growing since 2015. By far, there’re more than 30k nodes in the Hadoop clusters. <a href="https://blogs.apache.org/hadoop/mediaresource/ef6801e9-77ae-4bd1-b32c-ee889a16a7ca"><img src="https://blogs.apache.org/hadoop/mediaresource/ef6801e9-77ae-4bd1-b32c-ee889a16a7ca" alt="Picture13.jpg"></img></a> He shared many details and practice about the infrastructure of physical deployments, especially on solution for cluster across multiple regions. In the last part, Xiaoqiao shows some practices for optimizing HDFS cluster, such as: improve the Namenode restart process and rebalance for Namenode workload, etc. <h2><a name="_heading=h.aldjt4xtlgc"></a>10. Evolution of YARN in JD.com</h2> Wanqiang Ji from JD.com talked about how YARN evolves to support JD.com’s business needs. <a href="https://blogs.apache.org/hadoop/mediaresource/5513cff6-0674-4215-8a2f-12ade3d9b04d"><img src="https://blogs.apache.org/hadoop/mediaresource/5513cff6-0674-4215-8a2f-12ade3d9b04d" alt="Picture14.jpg"></img></a> In the last 3 years, maximum number of nodes in a single YARN cluster scales from 3k, 5k, 10k to 16k. Internally there’re works to balance resources between YARN/K8s cluster. Also there are improvements of container eviction policies to make sure nodes won’t crash or restart when machine’s physical utilization grows above a certain level. <h2><a name="_heading=h.o4jjvh6qs4pp"></a>11. Lessons learned from large scale YARN cluster operation @ Tencent</h2> Jun Gong and Dongdong Chen from Tencent talked about their works to support large scale YARN cluster inside tencent. <a href="https://blogs.apache.org/hadoop/mediaresource/32c57607-a28c-41de-96a5-f0371a5a6748"><img src="https://blogs.apache.org/hadoop/mediaresource/32c57607-a28c-41de-96a5-f0371a5a6748" alt="Picture15.jpg"></img></a> Gong Jun @ Tencent <a href="https://blogs.apache.org/hadoop/mediaresource/d65796d7-4740-41c5-9e36-ade2a9d5573b"><img src="https://blogs.apache.org/hadoop/mediaresource/d65796d7-4740-41c5-9e36-ade2a9d5573b" alt="Picture16.jpg"></img></a> Dongdong Chen @ Tencent Jun and Dongdong shared inside Tencent, they widely used SLS to figure out bottleneck of scheduler, many of the scheduler improvements have contributed back to the community. After optimization, in their production cluster, they have 2k+ queues, 8K+ nodes, 5k+ concurrent jobs. And they can achieve 3k+ container allocations per second, and more than 100 millions container allocations per day. Also, Jun and Dongdong shared how they uses YARN CGroups parameters to fine-tune CPU/Memory/Network shares for launched YARN containers in a multi-tenant cluster. <h2><a name="_heading=h.sxlj482o0isz"></a>12. Run Spark and Hadoop on ARM</h2> Rui Chen and Sheng Liu from Huawei shared their works to run Spark and Hadoop on ARM. <a href="https://blogs.apache.org/hadoop/mediaresource/b6a72eef-5158-4abd-bb83-33eb867f33f3"><img src="https://blogs.apache.org/hadoop/mediaresource/b6a72eef-5158-4abd-bb83-33eb867f33f3" alt="Picture17.jpg"></img></a> Rui Chen   <a href="https://blogs.apache.org/hadoop/mediaresource/88c34ec0-d4fd-47c8-8ed9-47724d6c715d"><img src="https://blogs.apache.org/hadoop/mediaresource/88c34ec0-d4fd-47c8-8ed9-47724d6c715d" alt="Picture18.jpg"></img></a> Sheng Liu   Rui and Sheng shared the motivation of running hadoop and spark on ARM platform which is for high performance and power efficiency. After that, they went ahead to share status of ARM support for hadoop and spark and details of building release Tarball on ARM platform include parameters, and issues. In the last part, they introduced how hadoop/spark release work can make sure proper testing for arm platform and they were building a community called OpenLab to make sure the process more smoothly.   <h1>Acknowledges</h1> Thanks everyone for contributing this successful event in one way or another, such as following speakers: Sammi Chen, Jun Gong and Dongdong Chen from Tencent, Weiwei Yang, Zhankun Tang from Cloudera, Wanqiang Ji from Jingdong, Tao Yang from Alibaba, Chen Zhang and Kang Zhou from Xiaomi, Hui Fei from Didi, Rui Chen and Sheng Liu from Huawei, Xiaoqiao He from Meituan Dianping , Yakun Li from ByteDance, and Xun Liu from Netease. And especially thanks Chunyu Wang, Summer Xia, Katty Ma for organizing the meetup! </div> </body> </html>

_posts/2019-08-22-hadoop-community-meetup-beijing-aug.html (275 lines of code) (raw):