quickstart.html

<!DOCTYPE html>  <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="Date-Revision-yyyymmdd" content="20160415" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Omid – Quickstart</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.5.min.css" /> <link rel="stylesheet" href="./css/site.css" /> <link rel="stylesheet" href="./css/print.css" media="print" /> <script type="text/javascript" src="./js/apache-maven-fluido-1.5.min.js"></script> </head> <body class="topBarEnabled"> <div id="topbar" class="navbar navbar-fixed-top "> <div class="navbar-inner"> <div class="container" style="width: 100%;"><div class="nav-collapse"> <ul class="nav"> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Home <b class="caret"></b></a> <ul class="dropdown-menu"> <li> <a href="index.html" title="Overview">Overview</a> </li> <li> <a href="license.html" title="License">License</a> </li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Download <b class="caret"></b></a> <ul class="dropdown-menu"> <li> <a href="https://bintray.com/yahoo/maven/omid" title="Download Omid">Download Omid</a> </li> <li> <a href="https://bintray.com/yahoo/maven/omid" title="Bintray Repository">Bintray Repository</a> </li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">User Guide & API <b class="caret"></b></a> <ul class="dropdown-menu"> <li> <a href="quickstart.html" title="Quickstart">Quickstart</a> </li> <li> <a href="basic-examples.html" title="API and Code Examples">API and Code Examples</a> </li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Technical Docs <b class="caret"></b></a> <ul class="dropdown-menu"> <li> <a href="basic-concepts.html" title="Basic Concepts">Basic Concepts</a> </li> <li> <a href="omid-components.html" title="Omid Components">Omid Components</a> </li> <li> <a href="basic-algorithm.html" title="Basic Algorithm">Basic Algorithm</a> </li> <li> <a href="client-failure-management.html" title="Management of Client Failures">Management of Client Failures</a> </li> <li> <a href="http://yahoohadoop.tumblr.com/tagged/HBase" title="Blog Entries">Blog Entries</a> </li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribute <b class="caret"></b></a> <ul class="dropdown-menu"> <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-omid.git" title="Source Code">Source Code</a> </li> <li> <a href="https://issues.apache.org/jira/browse/Omid" title="JIRA">JIRA</a> </li> <li> <a href="mailing-lists.html" title="Mailing Lists">Mailing Lists</a> </li> <li> <a href="coding-guide-and-style.html" title="Coding Guide and Style">Coding Guide and Style</a> </li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Reports <b class="caret"></b></a> <ul class="dropdown-menu"> <li class="dropdown-submenu"> <a href="project-info.html" title="Project Information">Project Information</a> <ul class="dropdown-menu"> <li> <a href="dependency-convergence.html" title="Dependency Convergence">Dependency Convergence</a> </li> <li> <a href="dependency-info.html" title="Dependency Information">Dependency Information</a> </li> <li> <a href="dependency-management.html" title="Dependency Management">Dependency Management</a> </li> <li> <a href="distribution-management.html" title="Distribution Management">Distribution Management</a> </li> <li> <a href="index.html" title="About">About</a> </li> <li> <a href="license.html" title="Licenses">Licenses</a> </li> <li> <a href="modules.html" title="Project Modules">Project Modules</a> </li> <li> <a href="plugin-management.html" title="Plugin Management">Plugin Management</a> </li> <li> <a href="plugins.html" title="Plugins">Plugins</a> </li> <li> <a href="team-list.html" title="Team">Team</a> </li> <li> <a href="source-repository.html" title="Source Code Management">Source Code Management</a> </li> <li> <a href="project-summary.html" title="Summary">Summary</a> </li> </ul> </li> </ul> </li> </ul> <form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" > <input value="omid.incubator.apache.org" name="sitesearch" type="hidden"/> <input class="search-query" name="q" id="query" type="text" /> </form> <script type="text/javascript" src="https://cse.google.com/brand?form=search-form"></script> <ul class="nav pull-right"><li> <a href="https://twitter.com/apacheomid" class="twitter-follow-button" data-show-count="true" data-align="right" data-size="large" data-show-screen-name="true" data-lang="en">Follow apacheomid</a> <script type="text/javascript">!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script> </li></ul> </div> </div> </div> </div> <div class="container"> <div id="banner"> <div class="pull-left"> <a href="./" id="bannerLeft"> <img src="images/omid-logo.png" alt="Omid" width="200"/> </a> </div> <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight"> <img src="http://incubator.apache.org/images/egg-logo2.png" alt="Apache Incubator" width="200"/> </a> </div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> <li id="projectVersion">Version: 0.8.1.37-SNAPSHOT </li> <li id="publishDate" class="pull-right">Last Published: 2016-04-15</li> </ul> </div> <div id="bodyColumn" > <h1>Quickstart</h1> <p>Below are instructions to quickly set up an environment to test Omid in your local machine.</p> <div class="section"> <h2><a name="Requirements"></a>Requirements</h2> <ol style="list-style-type: decimal"> <li>Maven 3.x</li> <li>Java 7</li> <li>HBase 0.98</li> <li>Protobuf 2.5.0</li> </ol></div> <div class="section"> <h2><a name="TSO_Setup"></a>TSO Setup</h2> <div class="section"> <h3><a name="a1._Download_and_Install_the_Required_HBase_Version"></a>1. Download and Install the Required HBase Version</h3> <p>You can find HBase distributions in <a class="externalLink" href="http://www.apache.org/dyn/closer.cgi/hbase/">this page</a>. Then start HBase in <a class="externalLink" href="https://hbase.apache.org/book.html#quickstart">standalone mode</a>.</p></div> <div class="section"> <h3><a name="a2._Clone_the_Omid_repository_and_Build_the_TSO_Package:"></a>2. Clone the <a class="externalLink" href="https://github.com/francisco-perez-sorrosal/omid">Omid repository</a> and Build the TSO Package:</h3> <div class="source"> <div class="source"><pre class="prettyprint">$ git clone git@github.com:yahoo/omid.git $ cd omid $ mvn clean install </pre></div></div> <p>This will generate a binary package containing all dependencies for the TSO in tso-server/target/tso-server-<VERSION>-bin.tar.gz. </p> <p><b>Be aware</b> Unit tests use HBase mini cluster, it typically fails to start if you are on VPN, thus unit test fail. Unit tests coverage is also quite extensive and take a while to run on each build (~15min at the moment of writing). So, consider using <tt>mvn clean install -DskipTests</tt> to speed temporal builds. Note that <tt>-Dmaven.test.skip=true</tt> <a class="externalLink" href="http://ericlefevre.net/wordpress/2008/02/21/skipping-tests-with-maven/">is NOT an equivalent</a>.</p> <p>As an alternative to clone the project, you can download the required version for the TSO tar.gz package from the <a class="externalLink" href="https://bintray.com/yahoo/maven/omid/view">release repository</a>.</p> <p>You can also see the <a class="externalLink" href="https://github.com/yahoo/omid/tags">build history here</a>.</p></div> <div class="section"> <h3><a name="a3._Extract_the_TSO_Package"></a>3. Extract the TSO Package</h3> <div class="source"> <div class="source"><pre class="prettyprint">$ tar zxvf tso-server-<VERSION>-bin.tar.gz $ cd tso-server-<VERSION> </pre></div></div></div> <div class="section"> <h3><a name="a4._Create_Omid_Tables"></a>4. Create Omid Tables</h3> <p>Ensure that the setting for hbase.zookeeper.quorum in conf/hbase-site.xml points to your zookeeper instance, and create the Timestamp Table and the Commit Table using the omid.sh script included in the bin directory of the tso server:</p> <div class="source"> <div class="source"><pre class="prettyprint">$ bin/omid.sh create-hbase-commit-table -numRegions 16 $ bin/omid.sh create-hbase-timestamp-table </pre></div></div> <p>These two tables are required by Omid and they must not be accessed by client applications.</p></div> <div class="section"> <h3><a name="a5._Start_the_TSO_Server"></a>5. Start the TSO Server</h3> <div class="source"> <div class="source"><pre class="prettyprint">$ bin/omid.sh tso </pre></div></div> <p>This starts the TSO server that in turn will connect to HBase to store information in HBase. By default the TSO listens on port 54758. If you want to change the TSO configuration, you can modify the contents in the conf/omid.conf file.</p></div></div> <div class="section"> <h2><a name="HBase_Client_Usage"></a>HBase Client Usage</h2> <div class="section"> <h3><a name="a1._Create_a_New_Java_Application"></a>1. Create a New Java Application</h3> <p>Use your favorite IDE an create a new project.</p></div> <div class="section"> <h3><a name="a2._Add_hbase-client_Dependency"></a>2. Add hbase-client Dependency</h3> <p>Choose the right version of the hbase-client jar. For example, in a Maven-based app add the following dependency in the pom.xml file:</p> <div class="source"> <div class="source"><pre class="prettyprint"><dependency> <groupId>com.yahoo.omid</groupId> <artifactId>hbase-client</artifactId> <version>${hbase_client.version}</version> </dependency> </pre></div></div></div> <div class="section"> <h3><a name="a3._Start_Coding_Using_the_Omid_Client_Interfaces"></a>3. Start Coding Using the Omid Client Interfaces</h3> <p>In Omid there are two client interfaces: <tt>TTable</tt> and <tt>TransactionManager</tt> (<i>These interfaces will likely change slightly in future.</i>):</p> <ol style="list-style-type: decimal"> <li> <p>The <tt>TransactionManager</tt> is used for creating transactional contexts, that is, transactions. A builder is provided in the <tt>HBaseTransactionManager</tt> class in order to get the TransactionManager interface.</p></li> <li> <p><tt>TTable</tt> is used for putting, getting and scanning entries in a HBase table. TTable’s interface is similar to the standard <tt>HTableInterface</tt>, and only requires passing the transactional context as a first parameter in the transactional aware methods (e.g. <tt>put(Transaction tx, Put put)</tt>)</p></li> </ol></div></div> <div class="section"> <h2><a name="Example_Application"></a>Example Application</h2> <p>Below is provided a sample application accessing data transactionally. Its a dummy application that writes two cells in two different rows of a table in a transactional context, but is enough to show how the different Omid client APIs are used. A detailed explanation of the client interfaces can be found in the [Basic Examples] section.</p> <div class="source"> <div class="source"><pre class="prettyprint">import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.util.Bytes; import com.yahoo.omid.transaction.HBaseTransactionManager; import com.yahoo.omid.transaction.TTable; import com.yahoo.omid.transaction.Transaction; import com.yahoo.omid.transaction.TransactionManager; public class OmidExample { public static final byte[] family = Bytes.toBytes("MY_CF"); public static final byte[] qualifier = Bytes.toBytes("MY_Q"); public static void main(String[] args) throws Exception { try (TransactionManager tm = HBaseTransactionManager.newInstance(); TTable txTable = new TTable("MY_TX_TABLE")) { Transaction tx = tm.begin(); Put row1 = new Put(Bytes.toBytes("EXAMPLE_ROW1")); row1.add(family, qualifier, Bytes.toBytes("val1")); txTable.put(tx, row1); Put row2 = new Put(Bytes.toBytes("EXAMPLE_ROW2")); row2.add(family, qualifier, Bytes.toBytes("val2")); txTable.put(tx, row2); tm.commit(tx); } } } </pre></div></div> <p>To run the application, make sure <tt>core-site.xml</tt> and <tt>hbase-site.xml</tt> for your HBase cluster are present in your CLASSPATH. The default configuration settings for the Omid client are loaded from the <tt>default-hbase-omid-client-config.yml</tt> file. In order to change the client default settings, you can either; 1) put your specific configuration settings in a file named <tt>hbase-omid-client-config.yml</tt> and include it in the CLASSPATH; or 2) do it programmatically in the application code by creating an instance of the <tt>HBaseOmidClientConfiguration</tt> class and passing it in the creation of the transaction manager:</p> <div class="source"> <div class="source"><pre class="prettyprint"> import com.yahoo.omid.transaction.HBaseOmidClientConfiguration; ... HBaseOmidClientConfiguration omidClientConfiguration = new HBaseOmidClientConfiguration(); omidClientConfiguration.setConnectionType(DIRECT); omidClientConfiguration.setConnectionString("my_tso_server_host:54758"); omidClientConfiguration.setRetryDelayMs(3000); try (TransactionManager tm = HBaseTransactionManager.newInstance(omidClientConfiguration); TTable txTable = new TTable("MY_TX_TABLE")) { ... </pre></div></div> <p>Also, you will need to create a HBase table “MY_TX_TABLE”, with column family “MY_CF”, and with <tt>TTL</tt> disabled and <tt>VERSIONS</tt> set to <tt>Integer.MAX_VALUE</tt>. For example using the HBase shell:</p> <div class="source"> <div class="source"><pre class="prettyprint">create 'MY_TX_TABLE', {NAME => 'MY_CF', VERSIONS => '2147483647', TTL => '2147483647'} </pre></div></div> <p>This example assumes non-secure communication with HBase. If your HBase cluster is secured with Kerberos, you will need to use the <tt>UserGroupInformation</tt> API to log in securely.</p></div> <div class="section"> <h2><a name="The_Omid_Compactor_Coprocessor"></a>The Omid Compactor Coprocessor</h2> <p>Omid includes a jar with an HBase coprocessor for performing data cleanup that operates during compactions, both minor and major. Specifically, it does the following:</p> <ul> <li>Cleans up garbage data from aborted transactions</li> <li>Purges deleted cells. Omid deletes work by placing a special tombstone marker in cells. The compactor detects these and actually purges data when it is safe to do so (i.e. when there are no committable transactions that may read the data).</li> <li>‘Heals’ committed cells for which the writer failed to write shadow cells.</li> </ul> <p>To deploy the coprocessor, the coprocessor jar must be placed in a location (typically on HDFS) that is accessible by HBase region servers. The coprocessor may then be enabled on a transactional table by the following steps in the HBase shell:</p> <p><b>1) Disable the table</b></p> <div class="source"> <div class="source"><pre class="prettyprint">disable 'MY_TX_TABLE' </pre></div></div> <p><b>2) Add a coprocessor specification to the table via a “coprocessor” attribute. The coprocessor spec may (and usually will) also include the name of the Omid commit table</b></p> <div class="source"> <div class="source"><pre class="prettyprint">alter 'MY_TX_TABLE', METHOD => 'table_att', 'coprocessor'=>'<path_to_omid_coprocessor>/omid-hbase-coprocessor-<coprocessor_version>.jar|com.yahoo.omid.transaction.OmidCompactor|1001|omid.committable.tablename=OMID_COMMIT_TABLE' </pre></div></div> <p><b>3) Add an “OMID_ENABLED => true” flag to any column families which the co-processor should work on</b></p> <div class="source"> <div class="source"><pre class="prettyprint">alter 'MY_TX_TABLE', { NAME => 'MY_CF', METADATA => {'OMID_ENABLED' => 'true'}} </pre></div></div> <p><b>4) Re-enable the table</b></p> <div class="source"> <div class="source"><pre class="prettyprint">enable 'MY_TX_TABLE' </pre></div></div></div> </div> </div> <hr/> <footer> <div class="container"> <div class="row"> <p >Copyright © 2011–2016 <a href="http://www.apache.org">Apache Software Foundation</a>. All rights reserved. </p> </div> <p id="poweredBy" class="pull-right"> <a href="http://maven.apache.org/" title="Maven" class="builtBy"> <img class="builtBy" alt="Maven" src="http://maven.apache.org/images/logos/maven-feather.png" /> </a> </p> </div> </footer> </body> </html>

quickstart.html (302 lines of code) (raw):