_posts/2017-03-27-hbase-on-azure-import-export.html

--- layout: post status: PUBLISHED published: true title: 'HBase on Azure: Import/Export snapshots from/to ADLS' id: 9d0c5176-584f-41ab-912b-c65ec5ebad4e date: '2017-03-27 23:04:06 -0400' categories: hbase tags: - adls - hbase - azure permalink: hbase/entry/hbase-on-azure-import-export --- by Apekshit Sharma, HBase Committer. <h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;">Overview</h1> Azure Data Lake Store (ADLS) is Microsoft’s cloud alternative for Apache HDFS. In this blog, we’ll see how to use it as backup for storing snapshots of Apache HBase tables. You can export snapshots to ADLS for backup; and for recovery, import the snapshot back to HDFS and use it to clone/restore the table. In this post, we’ll go over the configuration changes needed to make HDFS client talk to ADLS, and commands to copy HBase table snapshots from HDFS to ADLS and vice-versa. <h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;">Introduction</h1> “The Azure Data Lake store is an Apache Hadoop file system compatible with Hadoop Distributed File System (HDFS) and works with the Hadoop ecosystem.” <ul style="margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: disc; font-size: 12pt; font-family: Arial; font-style: italic; vertical-align: baseline;"> <a href="https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview">ADLS documentation</a> </li> </ul> ADLS can be treated as any HDFS service, except that it’s in the cloud. But then how do applications talk to it? That’s where the <a href="https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html">hadoop-azure-datalake module</a> comes into the picture. It enables an HDFS client to talk to ADLS whenever the following access path syntax is used: <div dir="ltr" style="margin-left: 0pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> adl://<Account Name>.azuredatalakestore.net/ For eg. hdfs dfs -mkdir adl://<Account Name>.azuredatalakestore.net/test_dir </td> </tr> </tbody> </table></div> However, before it can access any data in ADLS, the module needs to be able to authenticate to Azure. That requires a few configuration changes. These we describe in the next section. <h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;">Configuration changes</h1> ADLS requires an OAuth2 bearer token to be present as part of request’s HTTPS header. Users who have access to an ADLS account can obtain this token from the Azure Active Directory (Azure AD) service. To allow an HDFS client to authenticate to ADLS and access data, you’ll need to specify these tokens in core-site.xml using the following four configurations: <div dir="ltr" style="margin-left: 0pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> <property><name>dfs.adls.oauth2.access.token.provider.type</name><value>ClientCredential</value></property> <property><name>dfs.adls.oauth2.refresh.url</name><value>xxx</value></property> <property><name>dfs.adls.oauth2.client.id</name><value>xxx</value></property> <property><name>dfs.adls.oauth2.credential</name><value>xxx</value></property> </td> </tr> </tbody> </table></div> To find the values for dfs.adls.oauth2.* configurations, refer to <a href="https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html">this</a> document. Since all files/folders in ADLS are owned by the account owner, it’s ACL environment works well with that of HDFS which can have multiple users. Since the user issuing commands using the HDFS client will be different than what’s in Azure’s AD, any operation which checks for ACL will fail. To workaround this issue, use the following configuration which will tell the HDFS client that in case of ADLS requests, assume that the current user owns all files. <div dir="ltr" style="margin-left: 0pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> <property><name>adl.debug.override.localuserasfileowner</name><value>true</value></property> </td> </tr> </tbody> </table></div> Make sure to deploy the above configuration changes to the cluster. <h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;">Export snapshot to ADLS</h1> Here are the steps to export a snapshot from HDFS to ADLS. <ol style="margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> Create a new directory in ADLS to store snapshots. </li> </ol> <div dir="ltr" style="margin-left: 36pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> $ hdfs dfs -mkdir adl://appy.azuredatalakestore.net/hbase $ hdfs dfs -ls adl://appy.azuredatalakestore.net/ Found 1 items drwxr-xr-x   - systest hdfs          0 2017-03-21 23:43 adl://appy.azuredatalakestore.net/hbase </td> </tr> </tbody> </table></div> <ol style="margin-top: 0pt; margin-bottom: 0pt;" start="2"> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> Create the snapshot. To know more about this feature and how to create/list/restore snapshots, refer to <a href="http://hbase.apache.org/book.html#ops.snapshots">HBase Snapshots</a> section in the <a href="http://hbase.apache.org/book.html">HBase reference guide</a>. </li> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> Export snapshot to ADLS </li> </ol> <div dir="ltr" style="margin-left: 36pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> $ sudo -u hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot <snapshot_name> -copy-to adl://appy.azuredatalakestore.net/hbase [Output] 17/03/21 23:50:24 INFO snapshot.ExportSnapshot: Copy Snapshot Manifest … … 17/03/21 23:50:48 INFO snapshot.ExportSnapshot: Export Completed: snapshot_1 </td> </tr> </tbody> </table></div> <ol style="margin-top: 0pt; margin-bottom: 0pt;" start="4"> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> Verify that the snapshot was copied to ADLS. </li> </ol> <div dir="ltr" style="margin-left: 36pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> $ hbase snapshotinfo -snapshot <snapshot_name> -remote-dir adl://appy.azuredatalakestore.net/hbase Snapshot Info ----------------------------------------   Name: snapshot_1   Type: FLUSH  Table: t Format: 2 Created: 2017-03-21T23:42:56 </td> </tr> </tbody> </table></div> <ol style="margin-top: 0pt; margin-bottom: 0pt;" start="5"> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> It’s now safe to <a href="http://hbase.apache.org/book.html#ops.snapshots.delete">delete</a> the local snapshot (one in HDFS). </li> </ol> <h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;">Restore/Clone table from a snapshot in ADLS</h1> If you have a snapshot in ADLS which you want to use either to restore an original table to a previous state, or create a new table by cloning, follow the steps below. <ol style="margin-top: 0pt; margin-bottom: 0pt;"> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> Copy the snapshot back from ADLS to HDFS. Make sure to copy to ‘hbase’ directory on HDFS, because that’s where HBase service will look for snapshots. </li> </ol> <div dir="ltr" style="margin-left: 36pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> $ sudo -u hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot <snapshot_name> -copy-from adl://appy.azuredatalakestore.net/hbase -copy-to hdfs:///hbase </td> </tr> </tbody> </table></div> <ol style="margin-top: 0pt; margin-bottom: 0pt;" start="2"> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> Verify that the snapshot exists in HDFS. (Note that there is no -remote-dir parameter) </li> </ol> <div dir="ltr" style="margin-left: 36pt;"> <table style="border: medium none; border-collapse: collapse; width: 468pt;"> <colgroup> <col width="*" /></colgroup> <tbody> <tr style="height: 0pt;"> <td style="border-width: 1pt; border-style: solid; border-color: #000000; vertical-align: top; padding: 5pt;"> $ hbase snapshotinfo -snapshot snapshot_1 Snapshot Info ----------------------------------------   Name: snapshot_1   Type: FLUSH  Table: t Format: 2 Created: 2017-03-21T23:42:56 </td> </tr> </tbody> </table></div> <ol style="margin-top: 0pt; margin-bottom: 0pt;" start="3"> <li dir="ltr" style="list-style-type: decimal; font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline;"> Follow the instructions in <a href="http://hbase.apache.org/book.html#ops.snapshots">HBase Snapshots</a> section of <a href="http://hbase.apache.org/book.html">HBase reference guide</a> to restore/clone from the snapshot. </li> </ol> <h1 dir="ltr" style="line-height: 1.38; margin-top: 20pt; margin-bottom: 6pt;">Summary</h1> The Azure module in HDFS makes it easy to interact with ADLS. We can keep using the commands we are already know and our applications that use the HDFS client just need a few configuration changes. What what a seamless integration! In this blog, we got a glimpse of the HBase integration with Azure - Using ADLS as a backup for storing snapshots. Let’s see what the future has in store for us. Maybe, a HBase cluster fully backed by ADLS!

_posts/2017-03-27-hbase-on-azure-import-export.html (214 lines of code) (raw):