_posts/2015-12-17-offheaping_the_read_path

--- layout: post status: PUBLISHED published: true title: 'Offheaping the Read Path in Apache HBase: Part 2 of 2' id: cf570a23-a9bc-4827-a058-4b8ae3e782d1 date: '2015-12-17 23:58:18 -0500' categories: hbase tags: - offheap - apache - part2of2 - base - jam permalink: hbase/entry/offheaping_the_read_path_in1 --- by HBase Committers Anoop Sam John, Ramkrishna S Vasudevan, and Michael Stack This is part two of a two part blog. Herein we compare before and after off heaping. See <a href="https://blogs.apache.org/hbase/entry/offheaping_the_read_path_in">part one</a> for preamble and detail on work done. <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">Performance Results</h2> There were two parts to our performance measurement.   <ol style="line-height: normal; widows: 1; margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: decimal; vertical-align: baseline; background-color: transparent;"> Using HBase’s built-in Performance Evaluation (PE) tool.   </li> <li dir="ltr" style="list-style-type: decimal; vertical-align: baseline; background-color: transparent;"> Using <a href="https://github.com/brianfrankcooper/YCSB">YCSB</a> to measure the throughput. </li> </ol> The PE test was conducted on a single node machine.  Table is created and loaded with 100 GB of data.  Table has one CF and one column per row. Each cell value size is 1K. Configuration of the node : <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">System configuration</h2> <pre style="widows: 1; line-height: 1.295; margin-top: 0pt; margin-bottom: 8pt;">CPU : Intel(R) Xeon(R) CPU with 8 cores. RAM : 150 GB JDK : 1.8</pre> <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">HBase configuration</h2> <pre style="widows: 1; line-height: 1.295; margin-top: 0pt; margin-bottom: 8pt;">HBASE_HEAPSIZE = 9 GB HBASE_OFFHEAPSIZE = 105 GB <property>  <name>hbase.bucketcache.ioengine</name>  <value>offheap</value> </property> <property>  <name>hfile.block.cache.size</name>  <value>0.2</value> </property> <property>  <name>hbase.bucketcache.size</name>  <value> 104448 </value> </property></pre> 20% of the heap is allocated to the L1 cache (LRU cache). When L2 is enabled, L1 holds no data, just index and bloom filter blocks. 102 GB off heap space is allocated to the L2 cache (BucketCache) Before performing the read tests, we have made sure that all data is loaded into BucketCache so there is no i/o.  The read workloads of PE tool run with multiple threads.  We considerthe average completion time for each of the threads to do the required reads. 1. Each thread does 100 row multi get operations for 1000000 times. We can see that there is a 55 – 82 % reduction in average run time. See the graph below for the test for 5 to 75 threads reading. Y axis shows average completion time, in seconds, for one thread. <img src="https://lh4.googleusercontent.com/fFeJlNKx--Z62BZebqnLOo9gxNpkONEsj-tlzOZUNbG2FKM7ApNujdxtNv55YUd-jCp5fMQyO14iw17Mj6R-kWykexi4L91nKQDNTJ-HyZL8OPlUCyQDR7BUVwcylh3p-ZYbzw-A" width="624" height="333" style="transform: rotate(0rad);" /> Here each thread is doing 100000000 row get and converting this to throughput numbers we can see <div dir="ltr" style="line-height: normal; widows: 1; margin-left: -1.1pt;"> <table style="border: none; border-collapse: collapse;"> <colgroup> <col width="168" /> <col width="92" /> <col width="85" /> <col width="85" /> <col width="85" /> <col width="85" /> <col width="85" /></colgroup> <tbody> <tr style="height: 20px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 5 Threads </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 10 Threads </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 20 Threads </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 25 Threads </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 50 Threads </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 75 Threads </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> Throughput Without Patch </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 5594092.638 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 7152564.19 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 7001330.25 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 6920798.38 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 6113142.03 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 5463246.92 </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> Throughput With HBASE-11425 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 11353315.17 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 19782393.7 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 28477858.5 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 28216704.3 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 30229746.1 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 30647270.3 </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> Throughput gain </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 2x </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 2.7x </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 4x </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 4x </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 4.9x </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 0px 8px;"> 5.6x </td> </tr> </tbody> </table></div> So without the patch case, at the 20 threads level, the system goes to peak load situation and throughput starts to fall off. But with HBASE-11425 this is not the case and even with 75 threads. It is mostly linear scaling with more loading.  The major factor which helps us here is reduced GC activity. 2. Each thread is doing a range scan of 10000 rows with filtering of all the data on the server side. The filtering is done to see the server side gain alone and avoid any impact of network and/or client app side bottleneck. Each thread is doing the scan operation 10000 times. We can see that there is 55 – 74 % reduction in average run time of each thread. See below the graph for the test for 5 to 75 threads reading. Y axis shows average completion time, in seconds, for one thread. <img src="https://lh6.googleusercontent.com/4vqxdMht1iVJyySN93yU301FzD2JeadhRbs2ZhtdNb-SCUZTlbhhiVErWqDljhf_G5-pMk3yNGNf0ERdC2eVGTEg9uF4RLOUvdaBnw7iKvtt4AxonN2vaAgf7O_3TO2pmJ5AxISa" width="608" height="296" style="transform: rotate(0rad);" /> 3.  Another range scan test is performed with part of data returned back to client.  The test will return 10% of total rows back to the client and the remaining rows are filtered out at the server. The below graph is for a test with 10, 20, 25 and 50 threads. Again Y axis gives the average thread completion time, in seconds. The gain is  28 – 39% latency. <img src="https://lh3.googleusercontent.com/Xp8ygro3hY1DgmBAOjiUd23ABnCO22Pl0__a0vrEIUpvrHzEuLudkZ7lEd92nBtnVbPA4wrhwVvm2Pg-kkQmoYJT5aSwMRjEOzca-QXS2aiJwCZJIKHrdhQnF07UQoZvwP_EQbi5" width="480" height="288" style="transform: rotate(0rad);" /> The YCSB test is done in same cluster with 90 GB of data. We had similar system configuration and HBase configuration as for the PE tests. The YCSB setup involves creating a table with a single column family with around 90 GB of data. There are 10 columns each with 100 bytes of data (each row having 1k of data). All the readings are taken after ensuring that the entire data set is in the BucketCache. The multi get test includes each thread doing 100 row gets and 5000000 such operations. Range scan does random range scans with 1000000 operations. <img src="https://lh3.googleusercontent.com/0JPUmi_pFKSGAIjUvtXGoIchv7wSI0nPghlq8al78nnOiwGtbpmlTVfuSRzTVHe2k5VpW6Pg1_eiCUFEsRj5ukJMH1XvZUmXgSPCpE-9nxFZO4_bA3UnOwYJXjxSgFbSaXtrBMuJ" width="483" height="290" style="transform: rotate(0rad);" /> <div dir="ltr" style="line-height: normal; widows: 1; margin-left: 0pt;"> <table style="border: none; border-collapse: collapse;"> <colgroup> <col width="149" /> <col width="177" /> <col width="176" /></colgroup> <tbody> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> Threads </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> With HBASE-11425(Ops/sec) </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> Withoutpatch(Ops/sec) </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 10 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 28045.53 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 23277.97 </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 25 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 45767.99 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 25922.18 </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 50 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 58904.03 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 24558.72 </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 75 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 63280.86 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 24316.74 </td> </tr> </tbody> </table></div> <img src="https://lh6.googleusercontent.com/EIfwWcviZIWcPo8PtBWW790LxGFF4of5viEu5oKqoVQ-mvo7DHcCi6eX2PqwhdCf1x8WbksMGRDv-34zy33nornZ2NG9VyF9Yz57EvZEQbYL-iPCmG9D1RWdl13c45oB9ZQp7Pia" width="449" height="290" style="transform: rotate(0rad);" /> <div dir="ltr" style="line-height: normal; widows: 1; margin-left: 0pt;"> <table style="border: none; border-collapse: collapse;"> <colgroup> <col width="107" /> <col width="152" /> <col width="182" /></colgroup> <tbody> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> Threads </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> With HBASE-11425(Ops/Sec) </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> Withoutpatch(Ops/sec) </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 10 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 5332.451701 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 4139.416 </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 25 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 8685.456204 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 5340.796 </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 50 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 9180.235565 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 5149 </td> </tr> <tr style="height: 0px;"> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 75 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 9019.192842 </td> <td style="border: 1px solid #000000; vertical-align: bottom; padding: 7px;"> 4981.8 </td> </tr> </tbody> </table></div> For multi get there is 20 – 160 % throughput gain whereas for range scan it is 20 - 80%. With HBASE-11425 we can see there is linear throughput increase with more threads whereas old code starts performing badly when more threads (See when 50 and 75 threads) <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">GC graphs</h2> With HBASE-11425, we serve data directly from the off heap cache rather than copy each of the blocks on heap for each of the reads.  So we should be doing much better with respect to GC on the  RegionServer side.  Below are the GC graph samples taken on one RegionServer during the PE test run.  We can clearly notice that with the changes in HBASE-11425, we are doing much less (better) GC. <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">MultiGets – without HBASE-11425 (25 threads)</h2> <img src="https://lh6.googleusercontent.com/Kthbru6VNH9f2p_4jVe_uoQBxvVMVfrXDzemkG96xuyYSzmIOAoUwDlkT8d3TmDd7KuMn9oItHfsFl8ZgPz9cqdqUhPB_pkJTo_T4nVPfgkWUJQiM8n-S4L1G7pgKcrhsOiNicYrFjMvAF4c" width="624" height="337" alt="multiGets_69machine_withoutpatch" style="transform: rotate(0rad);" /> <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">Multigets – with HBASE-11425(25 threads)</h2> <img src="https://lh5.googleusercontent.com/EI1dHUMRqP0CrTA39dRUvSi64RAYY5jI1EsTN5PLuphBxz84YrDecaBJ1AJ4StngsOVCMIIBZJCDI82SzecVJ7WoOtnoWhfkNmC7bdhPZZIL6LLOXxgBmpIvlpJgUvvvmKjp_Kdw5IyP3IDq" width="623" height="336" alt="multiGets_69machine_withpatch" style="transform: rotate(0rad);" /> <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">ScanRange10000 – without HBASE-11425 (20 threads)</h2> <img src="https://lh4.googleusercontent.com/xjJ4UCHDBxQXNYdmh3s5Vkd390qBhALWTrmYOU7bDP1dzlexrZDRghE-r97wths4Dqpmoxm7BxW3dz3scBfacD28TcmsMm4HbUCyaBb4AGOca2GlV0CfcVkYNwuc89pYZbX-75RUlhv416AA" width="623" height="341" alt="scanRange_1k_50threads_withoutpatch" style="transform: rotate(0rad);" /> <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">ScanRange10000 – with HBASE-11425 (20 threads)</h2> <img src="https://lh4.googleusercontent.com/nDoeLsapfjtyif2AhC3doN73QhhI0puzYt-XwEzTnDIPHAU4jy9xyfZcoLyfMD8WXYoQ80z9bGIcp8NChPkpFqKPXoEVhTR6mWbDFdIkaJkexsW2UylNhomhy3ymnJDI-DesVB3XTRozgjFF" width="623" height="348" alt="scanRange_1k_50threads_withpatch" style="transform: rotate(0rad);" /> <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">Future Work</h2> One implication of the findings above is that we should run with off heap Cache on always. We were reluctant to do this in the past when reads from the off heap cache took longer. This is no longer the case. We will look into making this option on by default. Also, this posting has described our conversion of the read pipeline to make it run with offheap buffers. Next up, naturally, would be making the write pipeline offheap. <h2 dir="ltr" style="margin-top: 2pt; margin-bottom: 0pt; widows: 1; line-height: 1.295;">Conclusion</h2> Some parts of this work made it into branch-1 but to run with a fully off heap read path, you will have to wait on the HBase 2.0 release which should be available early next year (2016). Enabling L2 (BucketCache) in off heap mode will automatically turn on the off heap mechanisms described above. We hope you enjoy these improvements made to the Apache HBase read path.

_posts/2015-12-17-offheaping_the_read_path_in1.html (322 lines of code) (raw):