docs-archive/apache-airflow/1.10.2/plugins.html

<!DOCTYPE html>   <html class="no-js" lang="en" >  <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Plugins — Airflow Documentation</title> <link rel="stylesheet" href="_static/css/theme.css" type="text/css" /> <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> <link rel="index" title="Index" href="genindex.html" /> <link rel="search" title="Search" href="search.html" /> <link rel="next" title="Security" href="security.html" /> <link rel="prev" title="Scheduling & Triggers" href="scheduler.html" /> <script src="_static/js/modernizr.min.js"></script>  <script> var _paq = window._paq = window._paq || []; _paq.push(['disableCookies']); _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="https://analytics.apache.org/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '13']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); </script>  <link rel="canonical" href="https://airflow.apache.org/docs/apache-airflow/stable/plugins.html" /> </head> <body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> <nav data-toggle="wy-nav-shift" class="wy-nav-side"> <div class="wy-side-scroll"> <div class="wy-side-nav-search"> <a href="index.html" class="icon icon-home"> Airflow </a> <div class="version"> 1.10.2 </div> <div role="search"> <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> <input type="text" name="q" placeholder="Search docs" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> </div> <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li> <li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li> <li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li> <li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li> <li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li> <li class="toctree-l1"><a class="reference internal" href="howto/index.html">How-to Guides</a></li> <li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li> <li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li> <li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li> <li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li> <li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li> <li class="toctree-l1 current"><a class="current reference internal" href="#">Plugins</a><ul> <li class="toctree-l2"><a class="reference internal" href="#what-for">What for?</a></li> <li class="toctree-l2"><a class="reference internal" href="#why-build-on-top-of-airflow">Why build on top of Airflow?</a></li> <li class="toctree-l2"><a class="reference internal" href="#interface">Interface</a></li> <li class="toctree-l2"><a class="reference internal" href="#example">Example</a></li> <li class="toctree-l2"><a class="reference internal" href="#note-on-role-based-views">Note on role based views</a></li> <li class="toctree-l2"><a class="reference internal" href="#plugins-as-python-packages">Plugins as Python packages</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li> <li class="toctree-l1"><a class="reference internal" href="timezone.html">Time zones</a></li> <li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li> <li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li> <li class="toctree-l1"><a class="reference internal" href="metrics.html">Metrics</a></li> <li class="toctree-l1"><a class="reference internal" href="kubernetes.html">Kubernetes</a></li> <li class="toctree-l1"><a class="reference internal" href="lineage.html">Lineage</a></li> <li class="toctree-l1"><a class="reference internal" href="changelog.html">Changelog</a></li> <li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li> <li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li> </ul> </div> </div> </nav> <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> <nav class="wy-nav-top" aria-label="top navigation"> <a href="index.html">Airflow</a> </nav> <div class="wy-nav-content"> <div class="rst-content"> <div role="navigation" aria-label="breadcrumbs navigation"> <ul class="wy-breadcrumbs"> <li><a href="index.html">Docs</a> »</li> <li>Plugins</li> <li class="wy-breadcrumbs-aside"> <a href="_sources/plugins.rst.txt" rel="nofollow"> View page source</a> </li> </ul> <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> <div itemprop="articleBody"> <div class="section" id="plugins"> <h1>Plugins<a class="headerlink" href="#plugins" title="Permalink to this headline">¶</a></h1> Airflow has a simple plugin manager built-in that can integrate external features to its core by simply dropping files in your <code class="docutils literal notranslate">$AIRFLOW_HOME/plugins</code> folder. The python modules in the <code class="docutils literal notranslate">plugins</code> folder get imported, and hooks, operators, sensors, macros, executors and web views get integrated to Airflow’s main collections and become available for use. <div class="section" id="what-for"> <h2>What for?<a class="headerlink" href="#what-for" title="Permalink to this headline">¶</a></h2> Airflow offers a generic toolbox for working with data. Different organizations have different stacks and different needs. Using Airflow plugins can be a way for companies to customize their Airflow installation to reflect their ecosystem. Plugins can be used as an easy way to write, share and activate new sets of features. There’s also a need for a set of more complex applications to interact with different flavors of data and metadata. Examples: <ul class="simple"> <li>A set of tools to parse Hive logs and expose Hive metadata (CPU /IO / phases/ skew /…)</li> <li>An anomaly detection framework, allowing people to collect metrics, set thresholds and alerts</li> <li>An auditing tool, helping understand who accesses what</li> <li>A config-driven SLA monitoring tool, allowing you to set monitored tables and at what time they should land, alert people, and expose visualizations of outages</li> <li>…</li> </ul> </div> <div class="section" id="why-build-on-top-of-airflow"> <h2>Why build on top of Airflow?<a class="headerlink" href="#why-build-on-top-of-airflow" title="Permalink to this headline">¶</a></h2> Airflow has many components that can be reused when building an application: <ul class="simple"> <li>A web server you can use to render your views</li> <li>A metadata database to store your models</li> <li>Access to your databases, and knowledge of how to connect to them</li> <li>An array of workers that your application can push workload to</li> <li>Airflow is deployed, you can just piggy back on its deployment logistics</li> <li>Basic charting capabilities, underlying libraries and abstractions</li> </ul> </div> <div class="section" id="interface"> <h2>Interface<a class="headerlink" href="#interface" title="Permalink to this headline">¶</a></h2> To create a plugin you will need to derive the <code class="docutils literal notranslate">airflow.plugins_manager.AirflowPlugin</code> class and reference the objects you want to plug into Airflow. Here’s what the class you need to derive looks like: <div class="code python highlight-default notranslate"><div class="highlight"><pre>class AirflowPlugin(object): # The name of your plugin (str) name = None # A list of class(es) derived from BaseOperator operators = [] # A list of class(es) derived from BaseSensorOperator sensors = [] # A list of class(es) derived from BaseHook hooks = [] # A list of class(es) derived from BaseExecutor executors = [] # A list of references to inject into the macros namespace macros = [] # A list of objects created from a class derived # from flask_admin.BaseView admin_views = [] # A list of Blueprint object created from flask.Blueprint. For use with the flask_admin based GUI flask_blueprints = [] # A list of menu links (flask_admin.base.MenuLink). For use with the flask_admin based GUI menu_links = [] # A list of dictionaries containing FlaskAppBuilder BaseView object and some metadata. See example below appbuilder_views = [] # A list of dictionaries containing FlaskAppBuilder BaseView object and some metadata. See example below appbuilder_menu_items = [] </pre></div> </div> </div> <div class="section" id="example"> <h2>Example<a class="headerlink" href="#example" title="Permalink to this headline">¶</a></h2> The code below defines a plugin that injects a set of dummy object definitions in Airflow. <div class="code python highlight-default notranslate"><div class="highlight"><pre># This is the class you derive to create a plugin from airflow.plugins_manager import AirflowPlugin from flask import Blueprint from flask_admin import BaseView, expose from flask_admin.base import MenuLink # Importing base classes that we need to derive from airflow.hooks.base_hook import BaseHook from airflow.models import BaseOperator from airflow.sensors.base_sensor_operator import BaseSensorOperator from airflow.executors.base_executor import BaseExecutor # Will show up under airflow.hooks.test_plugin.PluginHook class PluginHook(BaseHook): pass # Will show up under airflow.operators.test_plugin.PluginOperator class PluginOperator(BaseOperator): pass # Will show up under airflow.sensors.test_plugin.PluginSensorOperator class PluginSensorOperator(BaseSensorOperator): pass # Will show up under airflow.executors.test_plugin.PluginExecutor class PluginExecutor(BaseExecutor): pass # Will show up under airflow.macros.test_plugin.plugin_macro def plugin_macro(): pass # Creating a flask admin BaseView class TestView(BaseView): @expose('/') def test(self): # in this example, put your test_plugin/test.html template at airflow/plugins/templates/test_plugin/test.html return self.render("test_plugin/test.html", content="Hello galaxy!") v = TestView(category="Test Plugin", name="Test View") # Creating a flask blueprint to integrate the templates and static folder bp = Blueprint( "test_plugin", __name__, template_folder='templates', # registers airflow/plugins/templates as a Jinja template folder static_folder='static', static_url_path='/static/test_plugin') ml = MenuLink( category='Test Plugin', name='Test Menu Link', url='https://airflow.apache.org/') # Creating a flask appbuilder BaseView class TestAppBuilderBaseView(AppBuilderBaseView): @expose("/") def test(self): return self.render("test_plugin/test.html", content="Hello galaxy!") v_appbuilder_view = TestAppBuilderBaseView() v_appbuilder_package = {"name": "Test View", "category": "Test Plugin", "view": v_appbuilder_view} # Creating a flask appbuilder Menu Item appbuilder_mitem = {"name": "Google", "category": "Search", "category_icon": "fa-th", "href": "https://www.google.com"} # Defining the plugin class class AirflowTestPlugin(AirflowPlugin): name = "test_plugin" operators = [PluginOperator] sensors = [PluginSensorOperator] hooks = [PluginHook] executors = [PluginExecutor] macros = [plugin_macro] admin_views = [v] flask_blueprints = [bp] menu_links = [ml] appbuilder_views = [v_appbuilder_package] appbuilder_menu_items = [appbuilder_mitem] </pre></div> </div> </div> <div class="section" id="note-on-role-based-views"> <h2>Note on role based views<a class="headerlink" href="#note-on-role-based-views" title="Permalink to this headline">¶</a></h2> Airflow 1.10 introduced role based views using FlaskAppBuilder. You can configure which UI is used by setting rbac = True. To support plugin views and links for both versions of the UI and maintain backwards compatibility, the fields appbuilder_views and appbuilder_menu_items were added to the AirflowTestPlugin class. </div> <div class="section" id="plugins-as-python-packages"> <h2>Plugins as Python packages<a class="headerlink" href="#plugins-as-python-packages" title="Permalink to this headline">¶</a></h2> It is possible to load plugins via <a href="#id1">`setuptools' entrypoint<https://packaging.python.org/guides/creating-and-discovering-plugins/#using-package-metadata>`_</a> mechanism. To do this link your plugin using an entrypoint in your package. If the package is installed, airflow will automatically load the registered plugins from the entrypoint list. _Note_: Neither the entrypoint name (eg, <cite>my_plugin</cite>) nor the name of the plugin class will contribute towards the module and class name of the plugin itself. The structure is determined by <cite>airflow.plugins_manager.AirflowPlugin.name</cite> and the class name of the plugin component with the pattern <cite>airflow.{component}.{name}.{component_class_name}</cite>. <div class="highlight-python notranslate"><div class="highlight"><pre># my_package/my_plugin.py from airflow.plugins_manager import AirflowPlugin from airflow.models import BaseOperator from airflow.hooks.base_hook import BaseHook class MyOperator(BaseOperator): pass class MyHook(BaseHook): pass class MyAirflowPlugin(AirflowPlugin): name = 'my_namespace' operators = [MyOperator] hooks = [MyHook] </pre></div> </div> <div class="highlight-python notranslate"><div class="highlight"><pre>from setuptools import setup setup( name="my-package", ... entry_points = { 'airflow.plugins': [ 'my_plugin = my_package.my_plugin:MyAirflowPlugin' ] } ) </pre></div> </div> <dl class="docutils"> <dt>This will create a hook, and an operator accessible at:</dt> <dd><ul class="first last simple"> <li><cite>airflow.hooks.my_namespace.MyHook</cite></li> <li><cite>airflow.operators.my_namespace.MyOperator</cite></li> </ul> </dd> </dl> </div> </div> </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> <a href="security.html" class="btn btn-neutral float-right" title="Security" accesskey="n" rel="next">Next </a> <a href="scheduler.html" class="btn btn-neutral" title="Scheduling & Triggers" accesskey="p" rel="prev"> Previous</a> </div> <hr/> <div role="contentinfo"> </div> Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. </footer> </div> </div> </section> </div> <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> <script type="text/javascript" src="_static/jquery.js"></script> <script type="text/javascript" src="_static/underscore.js"></script> <script type="text/javascript" src="_static/doctools.js"></script> <script type="text/javascript" src="_static/language_data.js"></script> <script type="text/javascript" src="_static/js/theme.js"></script> <script type="text/javascript"> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script> </body> </html>

docs-archive/apache-airflow/1.10.2/plugins.html (333 lines of code) (raw):