blog/2014/04/15/intro-to-pydruid.html

<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="description" content="Apache Druid"> <meta name="keywords" content="druid,kafka,database,analytics,streaming,real-time,real time,apache,open source"> <meta name="author" content="Apache Software Foundation"> <title>Druid | Introduction to pydruid</title> <link rel="alternate" type="application/atom+xml" href="/feed"> <link rel="shortcut icon" href="/img/favicon.png"> <link rel="stylesheet" href="/assets/css/font-awesome-5.css"> <link href='//fonts.googleapis.com/css?family=Open+Sans+Condensed:300,700,300italic|Open+Sans:300italic,400italic,600italic,400,300,600,700' rel='stylesheet' type='text/css'> <link rel="stylesheet" href="/css/bootstrap-pure.css?v=1.1"> <link rel="stylesheet" href="/css/base.css?v=1.1"> <link rel="stylesheet" href="/css/header.css?v=1.1"> <link rel="stylesheet" href="/css/footer.css?v=1.1"> <link rel="stylesheet" href="/css/syntax.css?v=1.1"> <link rel="stylesheet" href="/css/docs.css?v=1.1"> <script> (function() { var cx = '000162378814775985090:molvbm0vggm'; var gcse = document.createElement('script'); gcse.type = 'text/javascript'; gcse.async = true; gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + '//cse.google.com/cse.js?cx=' + cx; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(gcse, s); })(); </script> </head> <body>  <script src="//ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script> <div class="top-navigator"> <div class="container"> <div class="left-cont"> <a class="logo" href="/"></a> </div> <div class="right-cont"> <ul class="links"> <li class=""><a href="/technology">Technology</a></li> <li class=""><a href="/use-cases">Use Cases</a></li> <li class=""><a href="/druid-powered">Powered By</a></li> <li class=""><a href="/docs/latest/design/">Docs</a></li> <li class=""><a href="/community/">Community</a></li> <li class="header-dropdown"> <a>Apache</a> <div class="header-dropdown-menu"> <a href="https://www.apache.org/" target="_blank">Foundation</a> <a href="https://www.apache.org/events/current-event" target="_blank">Events</a> <a href="https://www.apache.org/licenses/" target="_blank">License</a> <a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <a href="https://www.apache.org/security/" target="_blank">Security</a> <a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a> </div> </li> <li class=" button-link"><a href="/downloads.html">Download</a></li> </ul> </div> </div> <div class="action-button menu-icon"> MENU </div> <div class="action-button menu-icon-close"> MENU </div> </div> <script type="text/javascript"> var $menu = $('.right-cont'); var $menuIcon = $('.menu-icon'); var $menuIconClose = $('.menu-icon-close'); function showMenu() { $menu.fadeIn(100); $menuIcon.fadeOut(100); $menuIconClose.fadeIn(100); } $menuIcon.click(showMenu); function hideMenu() { $menu.fadeOut(100); $menuIconClose.fadeOut(100); $menuIcon.fadeIn(100); } $menuIconClose.click(hideMenu); $(window).resize(function() { if ($(window).width() >= 840) { $menu.fadeIn(100); $menuIcon.fadeOut(100); $menuIconClose.fadeOut(100); } else { $menu.fadeOut(100); $menuIcon.fadeIn(100); $menuIconClose.fadeOut(100); } }); </script>  <link rel="stylesheet" href="/css/blogs.css"> <div class="blog druid-header"> <div class="row"> <div class="col-md-8 col-md-offset-2"> <div class="title-image-wrap"> </div> </div> </div> </div> <div class="container blog"> <div class="row"> <div class="col-md-8 col-md-offset-2"> <div class="blog-entry"> <h1>Introduction to pydruid</h1> by Igal Levy · April 15, 2014 We've already written about pairing <a href="/blog/2014/02/03/rdruid-and-twitterstream.html">R with RDruid</a>, but Python has powerful and free open-source analysis tools too. Collectively, these are often referred to as the <a href="http://www.scipy.org/stackspec.html">SciPy Stack</a>. To pair SciPy's analytic power with the advantages of querying time-series data in Druid, we created the pydruid connector. This allows Python users to query Druid—and export the results to useful formats—in a way that makes sense to them. <h2 id="getting-started">Getting Started</h2> pydruid should run with Python 2.x, and is known to run with Python 2.7.5. Install pydruid in the same way as you'd install any other Python module on your system. The simplest way is: <div class="highlight"><pre><code class="language-bash" data-lang="bash">pip install pydruid </code></pre></div> You should also install Pandas to execute the simple examples below: <div class="highlight"><pre><code class="language-bash" data-lang="bash">pip install pandas </code></pre></div> When you import pydruid into your example, it will try to load Pandas as well. <h2 id="run-the-druid-wikipedia-example">Run the Druid Wikipedia Example</h2> <a href="/downloads.html">Download Druid</a> and unpack Druid. If you are not familiar with Druid, see this <a href="/docs/latest/Tutorial:-A-First-Look-at-Druid.html">introductory tutorial</a>. From the Druid home directory, start the Druid Realtime node: <div class="highlight"><pre><code class="language-bash" data-lang="bash">$DRUID_HOME/run_example_server.sh </code></pre></div> When prompted, choose the "wikipedia" example. After the Druid realtime node is done starting up, messages should appear that start with the following: <div class="highlight"><pre><code class="language-text" data-lang="text">2014-04-03 18:01:32,852 INFO [wikipedia-incremental-persist] ... </code></pre></div> These messages confirm that the realtime node is ingesting data from the Wikipedia edit stream, and that data can be queried. <h2 id="write-execute-and-submit-a-pydruid-query">Write, Execute, and Submit a pydruid Query</h2> Let's say we want to see the top few languages for Wikipedia articles, in terms of number of edits. This is the query we could post directly to Druid: <div class="highlight"><pre><code class="language-json" data-lang="json">{ "queryType": "topN", "dataSource": "wikipedia", "dimension": "language", "threshold": 4, "metric": "edit_count", "granularity": "all", "filter": { "type": "selector", "dimension": "namespace", "value": "article" }, "aggregations": [ { "type": "longSum", "name": "edit_count", "fieldName": "count" } ], "intervals":["2013-06-01T00:00/2020-01-01T00"] } </code></pre></div> The results should appear similar to the following: <div class="highlight"><pre><code class="language-json" data-lang="json">[ { "timestamp" : "2014-04-03T17:59:00.000Z", "result" : [ { "language" : "en", "edit_count" : 4726 }, { "language" : "fr", "edit_count" : 1273 }, { "language" : "de", "edit_count" : 857 }, { "language" : "ja", "edit_count" : 176 } ] } ] </code></pre></div> NOTE: Due to limitations in the way the wikipedia example is set up, you may see a limited number of results appear. Here's that same query in Python: <div class="highlight"><pre><code class="language-python" data-lang="python">from pydruid.client import * query = PyDruid('http://localhost:8083', 'druid/v2/') top_langs = query.topn( datasource = "wikipedia", granularity = "all", intervals = "2013-06-01T00:00/2020-01-01T00", dimension = "language", filter = Dimension("namespace") == "article", aggregations = {"edit_count": longsum("count")}, metric = "edit_count", threshold = 4 ) print top_langs # Do this if you want to see the raw JSON </code></pre></div> Let's break this query down: <ul> <li>query – The <code>query</code> object is instantiated with the location of the Druid realtime node. <code>query</code> exposes various querying methods, including <code>topn</code>.</li> <li>datasource – This identifies the datasource. If Druid were ingesting from more than one datasource, this ID would identify the one we want.</li> <li>granularity – The rollup granularity, which could be set to a specific value such as <code>minute</code> or <code>hour</code>. We want to see the sum count across the entire interval, and so we choose <code>all</code>.</li> <li>intervals – The interval of time we're interested in. The value given is extended beyond our actual endpoints to make sure we cover all of the data.</li> <li>dimension – The dimension we're interested in, which happens to be language. Language is an attribute of the <a href="http://meta.wikimedia.org/wiki/IRC/Channels#Raw_feeds">Wikipedia recent-changes feed's metadata</a>.</li> <li>filter – Filters are used to specify a selector. In this case, we're selecting pages that have a namespace dimension with the value <code>article</code> (therefore excluding edits to Wikipedia pages that aren't articles).</li> <li>aggregations – We're interested in obtaining the total count of edited pages, per the language dimension, and we map it to a type of aggregation available in pydruid (longsum). We also rename this <code>count</code> metric to <code>edit_count</code>.</li> <li>metric – Names the metric to sort on.</li> <li>threshold – Sets the maximum number of aggregated results to return.</li> </ul> See the <a href="https://pythonhosted.org/pydruid/">pydruid documentation</a> for more information about queries. <h2 id="bringing-the-data-into-pandas">Bringing the Data Into Pandas</h2> Now that Druid is returning data, we'll pass that data to a Pandas dataframe, which allows us to analyze and visualize it: <div class="highlight"><pre><code class="language-python" data-lang="python">from pydruid.client import * from pylab import plt # Need to have matplotlib installed query = PyDruid('http://localhost:8083', 'druid/v2/') top_langs = query.topn( datasource = "wikipedia", granularity = "all", intervals = "2013-06-01T00:00/2020-01-01T00", dimension = "language", filter = Dimension("namespace") == "article", aggregations = {"edit_count": longsum("count")}, metric = "edit_count", threshold = 4 ) print top_langs # Do this if you want to see the raw JSON df = query.export_pandas() # Client will import Pandas, no need to do so separately. df = df.drop('timestamp', axis=1) # Don't need the timestamp column here df.index = range(1, len(df)+1) # Get a naturally numbered index print df df.plot(x='language', kind='bar') plt.show() </code></pre></div> Printing the results gives: <div class="highlight"><pre><code class="language-text" data-lang="text"> edit_count language 1 834 en 2 256 de 3 185 fr 4 38 ja </code></pre></div> The bar graph will look something like this: <img src="/img/wiki-edit-lang-plot.png" alt="Bar graph showing Wikipedia edits by language" title="Wikipedia Edits by Language"> If you were to repeat the query, you should see larger numbers under edit_count, since the Druid realtime node is continuing to ingest data from Wikipedia. <h2 id="conclusions">Conclusions</h2> In this blog, we showed how you can run ad-hoc queries against a data set that is being streamed into Druid. And while this is only a small example of pydruid and the power of Python, it serves as an effective introductory demonstration of the benefits of pairing Druid's ability to make data available in real-time with SciPi's powerful analytics tools. </div> </div> </div> </div>  <footer class="druid-footer"> <div class="container"> <div class="text-center"> <a href="/technology">Technology</a>&ensp;·&ensp; <a href="/use-cases">Use Cases</a>&ensp;·&ensp; <a href="/druid-powered">Powered by Druid</a>&ensp;·&ensp; <a href="/docs/latest/">Docs</a>&ensp;·&ensp; <a href="/community/">Community</a>&ensp;·&ensp; <a href="/downloads.html">Download</a>&ensp;·&ensp; <a href="/faq">FAQ</a> </div> <div class="text-center"> <a title="Join the user group" href="https://groups.google.com/forum/#!forum/druid-user" target="_blank"></a>&ensp;·&ensp; <a title="Follow Druid" href="https://twitter.com/druidio" target="_blank"></a>&ensp;·&ensp; <a title="GitHub" href="https://github.com/apache/druid" target="_blank"></a> </div> <div class="text-center license"> Copyright © 2020 <a href="https://www.apache.org/" target="_blank">Apache Software Foundation</a>. Except where otherwise noted, licensed under <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA 4.0</a>. Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. </div> </div> </footer> <script async src="https://www.googletagmanager.com/gtag/js?id=UA-131010415-1"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-131010415-1'); </script> <script> function trackDownload(type, url) { ga('send', 'event', 'download', type, url); } </script> <script src="//code.jquery.com/jquery.min.js"></script> <script src="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script> <script src="/assets/js/druid.js"></script>  </body> </html>

blog/2014/04/15/intro-to-pydruid.html (283 lines of code) (raw):