_posts/2016-04-22-hdfs_hsm_and_hbase

--- layout: post status: PUBLISHED published: true title: 'HDFS HSM and HBase: Tuning (Part 3 of 7)' id: 855e5449-b803-4888-b2da-e326c554b161 date: '2016-04-22 23:49:14 -0400' categories: hbase tags: - evaluation - performance - part3of7 - tuning permalink: hbase/entry/hdfs_hsm_and_hbase_tuning --- This is part 3 of a 7 part report by HBase Contributor, Jingcheng Du and HDFS contributor, Wei Zhou (Jingcheng and Wei are both Software Engineers at Intel)  <ol> <li><a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_part">Introduction</a></li> <li><a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_cluster">Cluster Setup</a></li> <li><a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_tuning">Tuning</a></li> <li><a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_cluster1">Experiment</a></li> <li><a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_cluster2">Experiment (continued)</a></li> <li><a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_cluster3">Issues</a></li> <li><a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_cluster4">Conclusions</a></li> </ol> <h1 dir="ltr" style="line-height: 1.38; margin-top: 24pt; margin-bottom: 6pt;" id="docs-internal-guid-894f636c-4051-61e6-cd5a-f52654a3a963">Stack Enhancement and Parameter Tuning</h1> <h2 dir="ltr" style="line-height: 1.38; margin-top: 18pt; margin-bottom: 4pt;">Stack Enhancement</h2> To perform the study, we made a set of enhancements in the software stack: <ul style="margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: disc; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> HDFS: </li> <ul style="margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: circle; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> Support a new storage RAMDISK </li> <li dir="ltr" style="list-style-type: circle; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> Add file level mover support, a user can move blocks per file without scanning all metadata in NameNode </li> </ul> <li dir="ltr" style="list-style-type: disc; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> HBase: </li> <ul style="margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: circle; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> WAL, flushed HFiles, HFiles generated in compactions, and archived HFiles can be stored in different storage </li> <li dir="ltr" style="list-style-type: circle; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> When renaming HFiles across storage, the blocks of that file would be moved to the target storage asynchronously </li> </ul> </ul> <h2 dir="ltr" style="line-height: 1.38; margin-top: 18pt; margin-bottom: 4pt;">HDFS/HBase Tuning</h2> This step is to find the best configurations for HDFS and HBase. <h3 dir="ltr" style="line-height: 1.38; margin-top: 14pt; margin-bottom: 4pt;">Known Key Performance Factors in HBase</h3> These are the key performance factors in HBase: <ol style="margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: decimal; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> WAL: write ahead log to guarantee the non-volatility and consistency of the data. Each record that is inserted to HBase must be written to WAL which can slow down user operations. It’s latency-sensitive. </li> <li dir="ltr" style="list-style-type: decimal; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> Memstore and Flush: The records inserted into HBase are cached in memstore, and when reaches a threshold the memstore is flushed to a store file. Slow flush can lead to high GC (Garbage Collection) pause, and make memory usage reach the thresholds in regions and region server, which can block the user operations. </li> <li dir="ltr" style="list-style-type: decimal; font-size: 14.6667px; font-family: Calibri; vertical-align: baseline; background-color: transparent;"> Compaction and Number of Store Files: HBase compaction compacts small store files to a larger one which can reduce the number of store files and accelerate the reading, but it can generate heavy I/O and consume the disk bandwidth in runtime. Less compaction can accelerate the writing but generates too many store files, which slow down the reading. When there are too many store files, the memstore flush can be slowed down which can lead to a large memstore and further slow the user operations. </li> </ol> Based on this understanding, the following are the tuned parameters we finally used. <div dir="ltr" style="margin-left: -11.5pt;"> <table style="border: none; border-collapse: collapse;"> <colgroup> <col width="306" /> <col width="333" /> <col width="17" /> <col width="92" /></colgroup> <tbody> <tr style="height: 20px;"> <td style="border-style: solid; border-color: #5b9bd5 #000000 #5b9bd5 #5b9bd5; border-width: 1px 0px 1px 1px; vertical-align: top; padding: 0px 8px; background-color: #5b9bd5;"> Property </td> <td style="border-style: solid; border-color: #5b9bd5 #5b9bd5 #5b9bd5 #000000; border-width: 1px 1px 1px 0px; vertical-align: top; padding: 0px 8px; background-color: #5b9bd5;"> Value </td> </tr> <tr style="height: 20px;"> <td style="border-style: solid; border-color: #5b9bd5 #9cc3e5 #9cc3e5; border-width: 1px; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> dfs.datanode.handler.count </td> <td style="border-style: solid; border-color: #5b9bd5 #9cc3e5 #9cc3e5; border-width: 1px; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> 64 </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> dfs.namenode.handler.count </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> 100 </td> </tr> </tbody> </table></div> Table 8. HDFS configuration <div dir="ltr" style="margin-left: -11.5pt;"> <table style="border: none; border-collapse: collapse;"> <colgroup> <col width="307" /> <col width="330" /> <col width="15" /> <col width="77" /></colgroup> <tbody> <tr style="height: 20px;"> <td style="border-style: solid; border-color: #5b9bd5 #000000 #5b9bd5 #5b9bd5; border-width: 1px 0px 1px 1px; vertical-align: top; padding: 0px 8px; background-color: #5b9bd5;"> Property </td> <td style="border-style: solid; border-color: #5b9bd5 #5b9bd5 #5b9bd5 #000000; border-width: 1px 1px 1px 0px; vertical-align: top; padding: 0px 8px; background-color: #5b9bd5;"> Value </td> </tr> <tr style="height: 20px;"> <td style="border-style: solid; border-color: #5b9bd5 #9cc3e5 #9cc3e5; border-width: 1px; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> hbase.regionserver.thread.compaction.small </td> <td style="border-style: solid; border-color: #5b9bd5 #9cc3e5 #9cc3e5; border-width: 1px; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> 3 for non-SSD test cases. 8 for all SSD related test cases. </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> hbase.hstore.flusher.count </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> 5 for non-SSD test cases. 15 for all SSD related test cases. </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> hbase.wal.regiongrouping.numgroups </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> 4 </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> hbase.wal.provider </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> multiwal </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> hbase.hstore.blockingStoreFiles </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> 15 </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> hbase.regionserver.handler.count </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> 200 </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> hbase.hregion.memstore.chunkpool.maxsize </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px; background-color: #deebf6;"> 1 </td> </tr> <tr style="height: 20px;"> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> hbase.hregion.memstore.chunkpool.initialsize </td> <td style="border: 1px solid #9cc3e5; vertical-align: top; padding: 0px 8px;"> 0.5 </td> </tr> </tbody> </table></div> Table 9. HBase configuration Go to part 4, <a href="https://blogs.apache.org/hbase/entry/hdfs_hsm_and_hbase_cluster1">Experiment</a>

_posts/2016-04-22-hdfs_hsm_and_hbase_tuning.html (192 lines of code) (raw):