versions/1.5.0/api/python/io/io.html

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"/> <meta content="IE=edge" http-equiv="X-UA-Compatible"/> <meta content="width=device-width, initial-scale=1" name="viewport"/> <meta content="Data Loading API" property="og:title"> <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image"> <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image:secure_url"> <meta content="Data Loading API" property="og:description"/> <title>Data Loading API — mxnet documentation</title> <link crossorigin="anonymous" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" rel="stylesheet"/> <link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" rel="stylesheet"/> <link href="../../../_static/basic.css" rel="stylesheet" type="text/css"> <link href="../../../_static/pygments.css" rel="stylesheet" type="text/css"> <link href="../../../_static/mxnet.css" rel="stylesheet" type="text/css"/> <script type="text/javascript"> var DOCUMENTATION_OPTIONS = { URL_ROOT: '../../../', VERSION: '', COLLAPSE_INDEX: false, FILE_SUFFIX: '.html', HAS_SOURCE: true, SOURCELINK_SUFFIX: '.txt' }; </script> <script src="https://code.jquery.com/jquery-1.11.1.min.js" type="text/javascript"></script> <script src="../../../_static/underscore.js" type="text/javascript"></script> <script src="../../../_static/searchtools_custom.js" type="text/javascript"></script> <script src="../../../_static/doctools.js" type="text/javascript"></script> <script src="../../../_static/selectlang.js" type="text/javascript"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script> <script type="text/javascript"> jQuery(function() { Search.loadIndex("/versions/1.5.0/searchindex.js"); Search.init();}); </script>          <link href="../../../genindex.html" rel="index" title="Index"> <link href="../../../search.html" rel="search" title="Search"/> <link href="../index.html" rel="up" title="MXNet - Python API"/> <link href="../kvstore/kvstore.html" rel="next" title="KVStore API"/> <link href="../image/image.html" rel="prev" title="Image API"/> <link href="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-icon.png" rel="icon" type="image/png"/> </link></link></link></meta></meta></meta></head> <body background="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-background-compressed.jpeg" role="document"> <div class="content-block"><div class="navbar navbar-fixed-top"> <div class="container" id="navContainer"> <div class="innder" id="header-inner"> <h1 id="logo-wrap"> <a href="../../../" id="logo"><img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet_logo.png"/></a> </h1> <nav class="nav-bar" id="main-nav"> <a class="main-nav-link" href="/versions/1.5.0/install/index.html">Install</a> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Gluon </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> <li><a class="main-nav-link" href="/versions/1.5.0/tutorials/gluon/gluon.html">About</a></li> <li><a class="main-nav-link" href="https://www.d2l.ai/">Dive into Deep Learning</a></li> <li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li> <li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li> </ul> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">API </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> <li><a class="main-nav-link" href="/versions/1.5.0/api/python/index.html">Python</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/c++/index.html">C++</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/clojure/index.html">Clojure</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/java/index.html">Java</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/julia/index.html">Julia</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/perl/index.html">Perl</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/r/index.html">R</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/scala/index.html">Scala</a></li> </ul> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Docs </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-docs"> <li><a class="main-nav-link" href="/versions/1.5.0/faq/index.html">FAQ</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/tutorials/index.html">Tutorials</a> <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/1.5.0/example">Examples</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/architecture/index.html">Architecture</a></li> <li><a class="main-nav-link" href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home">Developer Wiki</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/model_zoo/index.html">Model Zoo</a></li> <li><a class="main-nav-link" href="https://github.com/onnx/onnx-mxnet">ONNX</a></li> </li></ul> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Community </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-community"> <li><a class="main-nav-link" href="http://discuss.mxnet.io">Forum</a></li> <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/1.5.0">Github</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/community/contribute.html">Contribute</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/community/ecosystem.html">Ecosystem</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/community/powered_by.html">Powered By</a></li> </ul> <a href="#" class="main-nav-link dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="true">1.5.0</a><ul id="package-dropdown-menu" class="dropdown-menu"><li><a href="/">master</a></li><li><a href="/versions/1.7.0/">1.7.0</a></li><li><a href=/versions/1.6.0/>1.6.0</a></li><li><a href=/versions/1.5.0/>1.5.0</a></li><li><a href=/versions/1.4.1/>1.4.1</a></li><li><a href=/versions/1.3.1/>1.3.1</a></li><li><a href=/versions/1.2.1/>1.2.1</a></li><li><a href=/versions/1.1.0/>1.1.0</a></li><li><a href=/versions/1.0.0/>1.0.0</a></li><li><a href=/versions/0.12.1/>0.12.1</a></li><li><a href=/versions/0.11.0/>0.11.0</a></li></ul></nav> <script> function getRootPath(){ return "../../../" } </script> <div class="burgerIcon dropdown"> <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">☰</a> <ul class="dropdown-menu" id="burgerMenu"> <li><a href="/versions/1.5.0/install/index.html">Install</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/tutorials/index.html">Tutorials</a></li> <li class="dropdown-submenu dropdown"> <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Gluon</a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> <li><a class="main-nav-link" href="/versions/1.5.0/tutorials/gluon/gluon.html">About</a></li> <li><a class="main-nav-link" href="http://gluon.mxnet.io">The Straight Dope (Tutorials)</a></li> <li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li> <li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li> </ul> </li> <li class="dropdown-submenu"> <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">API</a> <ul class="dropdown-menu"> <li><a class="main-nav-link" href="/versions/1.5.0/api/python/index.html">Python</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/c++/index.html">C++</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/clojure/index.html">Clojure</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/java/index.html">Java</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/julia/index.html">Julia</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/perl/index.html">Perl</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/r/index.html">R</a></li> <li><a class="main-nav-link" href="/versions/1.5.0/api/scala/index.html">Scala</a></li> </ul> </li> <li class="dropdown-submenu"> <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Docs</a> <ul class="dropdown-menu"> <li><a href="/versions/1.5.0/faq/index.html" tabindex="-1">FAQ</a></li> <li><a href="/versions/1.5.0/tutorials/index.html" tabindex="-1">Tutorials</a></li> <li><a href="https://github.com/apache/incubator-mxnet/tree/1.5.0/example" tabindex="-1">Examples</a></li> <li><a href="/versions/1.5.0/architecture/index.html" tabindex="-1">Architecture</a></li> <li><a href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home" tabindex="-1">Developer Wiki</a></li> <li><a href="/versions/1.5.0/model_zoo/index.html" tabindex="-1">Gluon Model Zoo</a></li> <li><a href="https://github.com/onnx/onnx-mxnet" tabindex="-1">ONNX</a></li> </ul> </li> <li class="dropdown-submenu dropdown"> <a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">Community</a> <ul class="dropdown-menu"> <li><a href="http://discuss.mxnet.io" tabindex="-1">Forum</a></li> <li><a href="https://github.com/apache/incubator-mxnet/tree/1.5.0" tabindex="-1">Github</a></li> <li><a href="/versions/1.5.0/community/contribute.html" tabindex="-1">Contribute</a></li> <li><a href="/versions/1.5.0/community/ecosystem.html" tabindex="-1">Ecosystem</a></li> <li><a href="/versions/1.5.0/community/powered_by.html" tabindex="-1">Powered By</a></li> </ul> </li> <li id="dropdown-menu-position-anchor-version-mobile" class="dropdown-submenu" style="position: relative"><a href="#" tabindex="-1">1.5.0</a><ul class="dropdown-menu"><li><a tabindex="-1" href=/>master</a></li><li><a tabindex="-1" href=/versions/1.6.0/>1.6.0</a></li><li><a tabindex="-1" href=/versions/1.5.0/>1.5.0</a></li><li><a tabindex="-1" href=/versions/1.4.1/>1.4.1</a></li><li><a tabindex="-1" href=/versions/1.3.1/>1.3.1</a></li><li><a tabindex="-1" href=/versions/1.2.1/>1.2.1</a></li><li><a tabindex="-1" href=/versions/1.1.0/>1.1.0</a></li><li><a tabindex="-1" href=/versions/1.0.0/>1.0.0</a></li><li><a tabindex="-1" href=/versions/0.12.1/>0.12.1</a></li><li><a tabindex="-1" href=/versions/0.11.0/>0.11.0</a></li></ul></li></ul> </div> <div class="plusIcon dropdown"> <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button"></a> <ul class="dropdown-menu dropdown-menu-right" id="plusMenu"></ul> </div> <div id="search-input-wrap"> <form action="../../../search.html" autocomplete="off" class="" method="get" role="search"> <div class="form-group inner-addon left-addon"> <input class="form-control" name="q" placeholder="Search" type="text"/> </div> <input name="check_keywords" type="hidden" value="yes"> <input name="area" type="hidden" value="default"/> </input></form> <div id="search-preview"></div> </div> <div id="searchIcon"> </div>            </div> </div> </div> <script type="text/javascript"> $('body').css('background', 'white'); </script> <div class="container"> <div class="row"> <div aria-label="main navigation" class="sphinxsidebar leftsidebar" role="navigation"> <div class="sphinxsidebarwrapper"> <ul> <li class="toctree-l1"><a class="reference internal" href="../../index.html">MXNet APIs</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../architecture/index.html">MXNet Architecture</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../community/index.html">MXNet Community</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../faq/index.html">MXNet FAQ</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../gluon/index.html">About Gluon</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../install/index.html">Installing MXNet</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../install/index.html#nvidia-jetson-tx-family">Nvidia Jetson TX family</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../install/index.html#source-download">Source Download</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../model_zoo/index.html">MXNet Model Zoo</a></li> <li class="toctree-l1"><a class="reference internal" href="../../../tutorials/index.html">Tutorials</a></li> </ul> </div> </div> <div class="content"> <div class="page-tracker"></div>            <div class="section" id="data-loading-api"> <h1>Data Loading API<a class="headerlink" href="#data-loading-api" title="Permalink to this headline">¶</a></h1> <div class="section" id="overview"> <h2>Overview<a class="headerlink" href="#overview" title="Permalink to this headline">¶</a></h2> This document summarizes supported data formats and iterator APIs to read the data including: <table border="1" class="longtable docutils"> <colgroup> <col width="10%"/> <col width="90%"/> </colgroup> <tbody valign="top"> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">mxnet.io</code></td> <td>Data iterators for common data formats and utility functions.</td> </tr> <tr class="row-even"><td><a class="reference internal" href="#module-mxnet.recordio" title="mxnet.recordio"><code class="xref py py-obj docutils literal">mxnet.recordio</code></a></td> <td>Read and write for the RecordIO data format.</td> </tr> <tr class="row-odd"><td><a class="reference internal" href="../image/image.html#module-mxnet.image" title="mxnet.image"><code class="xref py py-obj docutils literal">mxnet.image</code></a></td> <td>Image Iterators and image augmentation functions</td> </tr> </tbody> </table> First, let’s see how to write an iterator for a new data format. The following iterator can be used to train a symbol whose input data variable has name <code class="docutils literal">data</code> and input label variable has name <code class="docutils literal">softmax_label</code>. The iterator also provides information about the batch, including the shapes and name. <div class="highlight-python"><div class="highlight"><pre>>>> nd_iter = mx.io.NDArrayIter(data={'data':mx.nd.ones((100,10))}, ... label={'softmax_label':mx.nd.ones((100,))}, ... batch_size=25) >>> print(nd_iter.provide_data) [DataDesc[data,(25, 10L),<type 'numpy.float32'>,NCHW]] >>> print(nd_iter.provide_label) [DataDesc[softmax_label,(25,),<type 'numpy.float32'>,NCHW]] </pre></div> </div> Let’s see a complete example of how to use data iterator in model training. <div class="highlight-python"><div class="highlight"><pre>>>> data = mx.sym.Variable('data') >>> label = mx.sym.Variable('softmax_label') >>> fullc = mx.sym.FullyConnected(data=data, num_hidden=1) >>> loss = mx.sym.SoftmaxOutput(data=fullc, label=label) >>> mod = mx.mod.Module(loss, data_names=['data'], label_names=['softmax_label']) >>> mod.bind(data_shapes=nd_iter.provide_data, label_shapes=nd_iter.provide_label) >>> mod.fit(nd_iter, num_epoch=2) </pre></div> </div> A detailed tutorial is available at <a class="reference external" href="/versions/1.5.0/tutorials/basic/data.html">Iterators - Loading data</a>. </div> <div class="section" id="data-iterators"> <h2>Data iterators<a class="headerlink" href="#data-iterators" title="Permalink to this headline">¶</a></h2> <table border="1" class="longtable docutils"> <colgroup> <col width="10%"/> <col width="90%"/> </colgroup> <tbody valign="top"> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">io.NDArrayIter</code></td> <td>Returns an iterator for <code class="docutils literal">mx.nd.NDArray</code>, <code class="docutils literal">numpy.ndarray</code>, <code class="docutils literal">h5py.Dataset</code> <code class="docutils literal">mx.nd.sparse.CSRNDArray</code> or <code class="docutils literal">scipy.sparse.csr_matrix</code>.</td> </tr> <tr class="row-even"><td><code class="xref py py-obj docutils literal">io.CSVIter</code></td> <td>Returns the CSV file iterator.</td> </tr> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">io.LibSVMIter</code></td> <td>Returns the LibSVM iterator which returns data with <cite>csr</cite> storage type.</td> </tr> <tr class="row-even"><td><code class="xref py py-obj docutils literal">io.ImageRecordIter</code></td> <td>Iterates on image RecordIO files</td> </tr> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">io.ImageRecordInt8Iter</code></td> <td>Iterating on image RecordIO files</td> </tr> <tr class="row-even"><td><code class="xref py py-obj docutils literal">io.ImageRecordUInt8Iter</code></td> <td>Iterating on image RecordIO files</td> </tr> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">io.MNISTIter</code></td> <td>Iterating on the MNIST dataset.</td> </tr> <tr class="row-even"><td><a class="reference internal" href="#mxnet.recordio.MXRecordIO" title="mxnet.recordio.MXRecordIO"><code class="xref py py-obj docutils literal">recordio.MXRecordIO</code></a></td> <td>Reads/writes <cite>RecordIO</cite> data format, supporting sequential read and write.</td> </tr> <tr class="row-odd"><td><a class="reference internal" href="#mxnet.recordio.MXIndexedRecordIO" title="mxnet.recordio.MXIndexedRecordIO"><code class="xref py py-obj docutils literal">recordio.MXIndexedRecordIO</code></a></td> <td>Reads/writes <cite>RecordIO</cite> data format, supporting random access.</td> </tr> <tr class="row-even"><td><a class="reference internal" href="../image/image.html#mxnet.image.ImageIter" title="mxnet.image.ImageIter"><code class="xref py py-obj docutils literal">image.ImageIter</code></a></td> <td>Image data iterator with a large number of augmentation choices.</td> </tr> <tr class="row-odd"><td><a class="reference internal" href="../image/image.html#mxnet.image.ImageDetIter" title="mxnet.image.ImageDetIter"><code class="xref py py-obj docutils literal">image.ImageDetIter</code></a></td> <td>Image iterator with a large number of augmentation choices for detection.</td> </tr> </tbody> </table> </div> <div class="section" id="helper-classes-and-functions"> <h2>Helper classes and functions<a class="headerlink" href="#helper-classes-and-functions" title="Permalink to this headline">¶</a></h2> <div class="section" id="data-structures-and-other-iterators"> <h3>Data structures and other iterators<a class="headerlink" href="#data-structures-and-other-iterators" title="Permalink to this headline">¶</a></h3> <table border="1" class="longtable docutils"> <colgroup> <col width="10%"/> <col width="90%"/> </colgroup> <tbody valign="top"> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">io.DataDesc</code></td> <td>DataDesc is used to store name, shape, type and layout information of the data or the label.</td> </tr> <tr class="row-even"><td><code class="xref py py-obj docutils literal">io.DataBatch</code></td> <td>A data batch.</td> </tr> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">io.DataIter</code></td> <td>The base class for an MXNet data iterator.</td> </tr> <tr class="row-even"><td><code class="xref py py-obj docutils literal">io.ResizeIter</code></td> <td>Resize a data iterator to a given number of batches.</td> </tr> <tr class="row-odd"><td><code class="xref py py-obj docutils literal">io.PrefetchingIter</code></td> <td>Performs pre-fetch for other data iterators.</td> </tr> <tr class="row-even"><td><code class="xref py py-obj docutils literal">io.MXDataIter</code></td> <td>A python wrapper a C++ data iterator.</td> </tr> </tbody> </table> </div> <div class="section" id="functions-to-read-and-write-recordio-files"> <h3>Functions to read and write RecordIO files<a class="headerlink" href="#functions-to-read-and-write-recordio-files" title="Permalink to this headline">¶</a></h3> <table border="1" class="longtable docutils"> <colgroup> <col width="10%"/> <col width="90%"/> </colgroup> <tbody valign="top"> <tr class="row-odd"><td><a class="reference internal" href="#mxnet.recordio.pack" title="mxnet.recordio.pack"><code class="xref py py-obj docutils literal">recordio.pack</code></a></td> <td>Pack a string into MXImageRecord.</td> </tr> <tr class="row-even"><td><a class="reference internal" href="#mxnet.recordio.unpack" title="mxnet.recordio.unpack"><code class="xref py py-obj docutils literal">recordio.unpack</code></a></td> <td>Unpack a MXImageRecord to string.</td> </tr> <tr class="row-odd"><td><a class="reference internal" href="#mxnet.recordio.unpack_img" title="mxnet.recordio.unpack_img"><code class="xref py py-obj docutils literal">recordio.unpack_img</code></a></td> <td>Unpack a MXImageRecord to image.</td> </tr> <tr class="row-even"><td><a class="reference internal" href="#mxnet.recordio.pack_img" title="mxnet.recordio.pack_img"><code class="xref py py-obj docutils literal">recordio.pack_img</code></a></td> <td>Pack an image into <code class="docutils literal">MXImageRecord</code>.</td> </tr> </tbody> </table> </div> </div> <div class="section" id="how-to-develop-a-new-iterator"> <h2>How to develop a new iterator<a class="headerlink" href="#how-to-develop-a-new-iterator" title="Permalink to this headline">¶</a></h2> Writing a new data iterator in Python is straightforward. Most MXNet training/inference programs accept an iterable object with <code class="docutils literal">provide_data</code> and <code class="docutils literal">provide_label</code> properties. This <a class="reference external" href="/versions/1.5.0/tutorials/basic/data.html">tutorial</a> explains how to write an iterator from scratch. The following example demonstrates how to combine multiple data iterators into a single one. It can be used for multiple modality training such as image captioning, in which images are read by <code class="docutils literal">ImageRecordIter</code> while documents are read by <code class="docutils literal">CSVIter</code> <div class="highlight-python"><div class="highlight"><pre>class MultiIter: def __init__(self, iter_list): self.iters = iter_list def next(self): batches = [i.next() for i in self.iters] return DataBatch(data=[*b.data for b in batches], label=[*b.label for b in batches]) def reset(self): for i in self.iters: i.reset() @property def provide_data(self): return [*i.provide_data for i in self.iters] @property def provide_label(self): return [*i.provide_label for i in self.iters] iter = MultiIter([mx.io.ImageRecordIter('image.rec'), mx.io.CSVIter('txt.csv')]) </pre></div> </div> Parsing and performing another pre-processing such as augmentation may be expensive. If performance is critical, we can implement a data iterator in C++. Refer to <a class="reference external" href="https://github.com/dmlc/mxnet/tree/master/src/io">src/io</a> for examples. <div class="section" id="how-to-change-the-batch-layout"> <h3>How to change the batch layout<a class="headerlink" href="#how-to-change-the-batch-layout" title="Permalink to this headline">¶</a></h3> By default, the backend engine treats the first dimension of each data and label variable in data iterators as the batch size (i.e. <code class="docutils literal">NCHW</code> or <code class="docutils literal">NT</code> layout). In order to override the axis for batch size, the <code class="docutils literal">provide_data</code> (and <code class="docutils literal">provide_label</code> if there is label) properties should include the layouts. This is especially useful in RNN since <code class="docutils literal">TNC</code> layouts are often more efficient. For example: <div class="highlight-python"><div class="highlight"><pre>@property def provide_data(self): return [DataDesc(name='seq_var', shape=(seq_length, batch_size), layout='TN')] </pre></div> </div> The backend engine will recognize the index of <code class="docutils literal">N</code> in the <code class="docutils literal">layout</code> as the axis for batch size. </div> </div> <div class="section" id="api-reference"> <h2>API Reference<a class="headerlink" href="#api-reference" title="Permalink to this headline">¶</a></h2> <script src="../../../_static/js/auto_module_index.js" type="text/javascript"></script><div class="section" id="mxnet-io-data-iterators"> <h3>mxnet.io - Data Iterators<a class="headerlink" href="#mxnet-io-data-iterators" title="Permalink to this headline">¶</a></h3> Data iterators for common data formats and utility functions. <dl class="class"> <dt> class <code class="descclassname">mxnet.io.</code><code class="descname">NDArrayIter</code>(data, label=None, batch_size=1, shuffle=False, last_batch_handle='pad', data_name='data', label_name='softmax_label')<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter">[source]</a></dt> <dd>Returns an iterator for <code class="docutils literal">mx.nd.NDArray</code>, <code class="docutils literal">numpy.ndarray</code>, <code class="docutils literal">h5py.Dataset</code> <code class="docutils literal">mx.nd.sparse.CSRNDArray</code> or <code class="docutils literal">scipy.sparse.csr_matrix</code>. Examples <div class="highlight-default"><div class="highlight"><pre>>>> data = np.arange(40).reshape((10,2,2)) >>> labels = np.ones([10, 1]) >>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='discard') >>> for batch in dataiter: ... print batch.data[0].asnumpy() ... batch.data[0].shape ... [[[ 36. 37.] [ 38. 39.]] [[ 16. 17.] [ 18. 19.]] [[ 12. 13.] [ 14. 15.]]] (3L, 2L, 2L) [[[ 32. 33.] [ 34. 35.]] [[ 4. 5.] [ 6. 7.]] [[ 24. 25.] [ 26. 27.]]] (3L, 2L, 2L) [[[ 8. 9.] [ 10. 11.]] [[ 20. 21.] [ 22. 23.]] [[ 28. 29.] [ 30. 31.]]] (3L, 2L, 2L) >>> dataiter.provide_data # Returns a list of `DataDesc` [DataDesc[data,(3, 2L, 2L),<type 'numpy.float32'>,NCHW]] >>> dataiter.provide_label # Returns a list of `DataDesc` [DataDesc[softmax_label,(3, 1L),<type 'numpy.float32'>,NCHW]] </pre></div> </div> In the above example, data is shuffled as <cite>shuffle</cite> parameter is set to <cite>True</cite> and remaining examples are discarded as <cite>last_batch_handle</cite> parameter is set to <cite>discard</cite>. Usage of <cite>last_batch_handle</cite> parameter: <div class="highlight-default"><div class="highlight"><pre>>>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='pad') >>> batchidx = 0 >>> for batch in dataiter: ... batchidx += 1 ... >>> batchidx # Padding added after the examples read are over. So, 10/3+1 batches are created. 4 >>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='discard') >>> batchidx = 0 >>> for batch in dataiter: ... batchidx += 1 ... >>> batchidx # Remaining examples are discarded. So, 10/3 batches are created. 3 >>> dataiter = mx.io.NDArrayIter(data, labels, 3, False, last_batch_handle='roll_over') >>> batchidx = 0 >>> for batch in dataiter: ... batchidx += 1 ... >>> batchidx # Remaining examples are rolled over to the next iteration. 3 >>> dataiter.reset() >>> dataiter.next().data[0].asnumpy() [[[ 36. 37.] [ 38. 39.]] [[ 0. 1.] [ 2. 3.]] [[ 4. 5.] [ 6. 7.]]] (3L, 2L, 2L) </pre></div> </div> <cite>NDArrayIter</cite> also supports multiple input and labels. <div class="highlight-default"><div class="highlight"><pre>>>> data = {'data1':np.zeros(shape=(10,2,2)), 'data2':np.zeros(shape=(20,2,2))} >>> label = {'label1':np.zeros(shape=(10,1)), 'label2':np.zeros(shape=(20,1))} >>> dataiter = mx.io.NDArrayIter(data, label, 3, True, last_batch_handle='discard') </pre></div> </div> <cite>NDArrayIter</cite> also supports <code class="docutils literal">mx.nd.sparse.CSRNDArray</code> with <cite>last_batch_handle</cite> set to <cite>discard</cite>. <div class="highlight-default"><div class="highlight"><pre>>>> csr_data = mx.nd.array(np.arange(40).reshape((10,4))).tostype('csr') >>> labels = np.ones([10, 1]) >>> dataiter = mx.io.NDArrayIter(csr_data, labels, 3, last_batch_handle='discard') >>> [batch.data[0] for batch in dataiter] [ <CSRNDArray 3x4 @cpu(0)>, <CSRNDArray 3x4 @cpu(0)>, <CSRNDArray 3x4 @cpu(0)>] </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>data (<a class="reference internal" href="../ndarray/ndarray.html#mxnet.ndarray.array" title="mxnet.ndarray.array">array</a> or list of array or dict of string to array) – The input data.</li> <li>label (<a class="reference internal" href="../ndarray/ndarray.html#mxnet.ndarray.array" title="mxnet.ndarray.array">array</a> or list of array or dict of string to array, optional) – The input label.</li> <li>batch_size (int) – Batch size of data.</li> <li>shuffle (bool, optional) – Whether to shuffle the data. Only supported if no h5py.Dataset inputs are used.</li> <li>last_batch_handle (str, optional) – How to handle the last batch. This parameter can be ‘pad’, ‘discard’ or ‘roll_over’. If ‘pad’, the last batch will be padded with data starting from the begining If ‘discard’, the last batch will be discarded If ‘roll_over’, the remaining elements will be rolled over to the next iteration and note that it is intended for training and can cause problems if used for prediction.</li> <li>data_name (str, optional) – The data name.</li> <li>label_name (str, optional) – The label name.</li> </ul> </td> </tr> </tbody> </table> <dl class="method"> <dt> <code class="descname">getdata</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter.getdata">[source]</a></dt> <dd>Get data. </dd></dl> <dl class="method"> <dt> <code class="descname">getlabel</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter.getlabel">[source]</a></dt> <dd>Get label. </dd></dl> <dl class="method"> <dt> <code class="descname">getpad</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter.getpad">[source]</a></dt> <dd>Get pad value of DataBatch. </dd></dl> <dl class="method"> <dt> <code class="descname">hard_reset</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter.hard_reset">[source]</a></dt> <dd>Ignore roll over data and set to start. </dd></dl> <dl class="method"> <dt> <code class="descname">iter_next</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter.iter_next">[source]</a></dt> <dd>Increments the coursor by batch_size for next batch and check current cursor if it exceed the number of data points. </dd></dl> <dl class="method"> <dt> <code class="descname">next</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter.next">[source]</a></dt> <dd>Returns the next batch of data. </dd></dl> <dl class="attribute"> <dt> <code class="descname">provide_data</code></dt> <dd>The name and shape of data provided by this iterator. </dd></dl> <dl class="attribute"> <dt> <code class="descname">provide_label</code></dt> <dd>The name and shape of label provided by this iterator. </dd></dl> <dl class="method"> <dt> <code class="descname">reset</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#NDArrayIter.reset">[source]</a></dt> <dd>Resets the iterator to the beginning of the data. </dd></dl> </dd></dl> <dl class="function"> <dt> <code class="descclassname">mxnet.io.</code><code class="descname">CSVIter</code>(*args, **kwargs)</dt> <dd>Returns the CSV file iterator. In this function, the <cite>data_shape</cite> parameter is used to set the shape of each line of the input data. If a row in an input file is <cite>1,2,3,4,5,6`</cite> and <cite>data_shape</cite> is (3,2), that row will be reshaped, yielding the array [[1,2],[3,4],[5,6]] of shape (3,2). By default, the <cite>CSVIter</cite> has <cite>round_batch</cite> parameter set to <code class="docutils literal">True</code>. So, if <cite>batch_size</cite> is 3 and there are 4 total rows in CSV file, 2 more examples are consumed at the first round. If <cite>reset</cite> function is called after first round, the call is ignored and remaining examples are returned in the second round. If one wants all the instances in the second round after calling <cite>reset</cite>, make sure to set <cite>round_batch</cite> to False. If <code class="docutils literal">data_csv = 'data/'</code> is set, then all the files in this directory will be read. <code class="docutils literal">reset()</code> is expected to be called only after a complete pass of data. By default, the CSVIter parses all entries in the data file as float32 data type, if <cite>dtype</cite> argument is set to be ‘int32’ or ‘int64’ then CSVIter will parse all entries in the file as int32 or int64 data type accordingly. Examples: <div class="highlight-default"><div class="highlight"><pre>// Contents of CSV file ``data/data.csv``. 1,2,3 2,3,4 3,4,5 4,5,6 // Creates a `CSVIter` with `batch_size`=2 and default `round_batch`=True. CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), batch_size = 2) // Two batches read from the above iterator are as follows: [[ 1. 2. 3.] [ 2. 3. 4.]] [[ 3. 4. 5.] [ 4. 5. 6.]] // Creates a `CSVIter` with default `round_batch` set to True. CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), batch_size = 3) // Two batches read from the above iterator in the first pass are as follows: [[1. 2. 3.] [2. 3. 4.] [3. 4. 5.]] [[4. 5. 6.] [1. 2. 3.] [2. 3. 4.]] // Now, `reset` method is called. CSVIter.reset() // Batch read from the above iterator in the second pass is as follows: [[ 3. 4. 5.] [ 4. 5. 6.] [ 1. 2. 3.]] // Creates a `CSVIter` with `round_batch`=False. CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), batch_size = 3, round_batch=False) // Contents of two batches read from the above iterator in both passes, after calling // `reset` method before second pass, is as follows: [[1. 2. 3.] [2. 3. 4.] [3. 4. 5.]] [[4. 5. 6.] [2. 3. 4.] [3. 4. 5.]] // Creates a 'CSVIter' with `dtype`='int32' CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), batch_size = 3, round_batch=False, dtype='int32') // Contents of two batches read from the above iterator in both passes, after calling // `reset` method before second pass, is as follows: [[1 2 3] [2 3 4] [3 4 5]] [[4 5 6] [2 3 4] [3 4 5]] </pre></div> </div> Defined in src/io/iter_csv.cc:L308 <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>data_csv (string, required) – The input CSV file or a directory path.</li> <li>data_shape (Shape(tuple), required) – The shape of one example.</li> <li>label_csv (string, optional, default='NULL') – The input CSV file or a directory path. If NULL, all labels will be returned as 0.</li> <li>label_shape (Shape(tuple), optional, default=[1]) – The shape of one label.</li> <li>batch_size (int (non-negative), required) – Batch size.</li> <li>round_batch (boolean, optional, default=1) – Whether to use round robin to handle overflow batch or not.</li> <li>prefetch_buffer (long (non-negative), optional, default=4) – Maximum number of batches to prefetch.</li> <li>ctx ({'cpu', 'gpu'},optional, default='gpu') – Context data loader optimized for.</li> <li>dtype ({None, 'float16', 'float32', 'float64', 'int32', 'int64', 'int8', 'uint8'},optional, default='None') – Output data type. <code class="docutils literal">None</code> means no change.</li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">The result iterator. </td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">MXDataIter </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt> <code class="descclassname">mxnet.io.</code><code class="descname">LibSVMIter</code>(*args, **kwargs)</dt> <dd>Returns the LibSVM iterator which returns data with <cite>csr</cite> storage type. This iterator is experimental and should be used with care. The input data is stored in a format similar to LibSVM file format, except that the indices are expected to be zero-based instead of one-based, and the column indices for each row are expected to be sorted in ascending order. Details of the LibSVM format are available <a class="reference external" href="https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/">here.</a> The <cite>data_shape</cite> parameter is used to set the shape of each line of the data. The dimension of both <cite>data_shape</cite> and <cite>label_shape</cite> are expected to be 1. The <cite>data_libsvm</cite> parameter is used to set the path input LibSVM file. When it is set to a directory, all the files in the directory will be read. When <cite>label_libsvm</cite> is set to <code class="docutils literal">NULL</code>, both data and label are read from the file specified by <cite>data_libsvm</cite>. In this case, the data is stored in <cite>csr</cite> storage type, while the label is a 1D dense array. The <cite>LibSVMIter</cite> only support <cite>round_batch</cite> parameter set to <code class="docutils literal">True</code>. Therefore, if <cite>batch_size</cite> is 3 and there are 4 total rows in libsvm file, 2 more examples are consumed at the first round. When <cite>num_parts</cite> and <cite>part_index</cite> are provided, the data is split into <cite>num_parts</cite> partitions, and the iterator only reads the <cite>part_index</cite>-th partition. However, the partitions are not guaranteed to be even. <code class="docutils literal">reset()</code> is expected to be called only after a complete pass of data. Example: <div class="highlight-default"><div class="highlight"><pre># Contents of libsvm file ``data.t``. 1.0 0:0.5 2:1.2 -2.0 -3.0 0:0.6 1:2.4 2:1.2 4 2:-1.2 # Creates a `LibSVMIter` with `batch_size`=3. >>> data_iter = mx.io.LibSVMIter(data_libsvm = 'data.t', data_shape = (3,), batch_size = 3) # The data of the first batch is stored in csr storage type >>> batch = data_iter.next() >>> csr = batch.data[0] <CSRNDArray 3x3 @cpu(0)> >>> csr.asnumpy() [[ 0.5 0. 1.2 ] [ 0. 0. 0. ] [ 0.6 2.4 1.2]] # The label of first batch >>> label = batch.label[0] >>> label [ 1. -2. -3.] <NDArray 3 @cpu(0)> >>> second_batch = data_iter.next() # The data of the second batch >>> second_batch.data[0].asnumpy() [[ 0. 0. -1.2 ] [ 0.5 0. 1.2 ] [ 0. 0. 0. ]] # The label of the second batch >>> second_batch.label[0].asnumpy() [ 4. 1. -2.] >>> data_iter.reset() # To restart the iterator for the second pass of the data </pre></div> </div> When <cite>label_libsvm</cite> is set to the path to another LibSVM file, data is read from <cite>data_libsvm</cite> and label from <cite>label_libsvm</cite>. In this case, both data and label are stored in the csr format. If the label column in the <cite>data_libsvm</cite> file is ignored. Example: <div class="highlight-default"><div class="highlight"><pre># Contents of libsvm file ``label.t`` 1.0 -2.0 0:0.125 -3.0 2:1.2 4 1:1.0 2:-1.2 # Creates a `LibSVMIter` with specified label file >>> data_iter = mx.io.LibSVMIter(data_libsvm = 'data.t', data_shape = (3,), label_libsvm = 'label.t', label_shape = (3,), batch_size = 3) # Both data and label are in csr storage type >>> batch = data_iter.next() >>> csr_data = batch.data[0] <CSRNDArray 3x3 @cpu(0)> >>> csr_data.asnumpy() [[ 0.5 0. 1.2 ] [ 0. 0. 0. ] [ 0.6 2.4 1.2 ]] >>> csr_label = batch.label[0] <CSRNDArray 3x3 @cpu(0)> >>> csr_label.asnumpy() [[ 0. 0. 0. ] [ 0.125 0. 0. ] [ 0. 0. 1.2 ]] </pre></div> </div> Defined in src/io/iter_libsvm.cc:L298 <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>data_libsvm (string, required) – The input zero-base indexed LibSVM data file or a directory path.</li> <li>data_shape (Shape(tuple), required) – The shape of one example.</li> <li>label_libsvm (string, optional, default='NULL') – The input LibSVM label file or a directory path. If NULL, all labels will be read from <code class="docutils literal">data_libsvm</code>.</li> <li>label_shape (Shape(tuple), optional, default=[1]) – The shape of one label.</li> <li>num_parts (int, optional, default='1') – partition the data into multiple parts</li> <li>part_index (int, optional, default='0') – the index of the part will read</li> <li>batch_size (int (non-negative), required) – Batch size.</li> <li>round_batch (boolean, optional, default=1) – Whether to use round robin to handle overflow batch or not.</li> <li>prefetch_buffer (long (non-negative), optional, default=4) – Maximum number of batches to prefetch.</li> <li>ctx ({'cpu', 'gpu'},optional, default='gpu') – Context data loader optimized for.</li> <li>dtype ({None, 'float16', 'float32', 'float64', 'int32', 'int64', 'int8', 'uint8'},optional, default='None') – Output data type. <code class="docutils literal">None</code> means no change.</li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">The result iterator. </td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">MXDataIter </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt> <code class="descclassname">mxnet.io.</code><code class="descname">ImageRecordIter</code>(*args, **kwargs)</dt> <dd>Iterates on image RecordIO files Reads batches of images from .rec RecordIO files. One can use <code class="docutils literal">im2rec.py</code> tool (in tools/) to pack raw image files into RecordIO files. This iterator is less flexible to customization but is fast and has lot of language bindings. To iterate over raw images directly use <code class="docutils literal">ImageIter</code> instead (in Python). Example: <div class="highlight-default"><div class="highlight"><pre>data_iter = mx.io.ImageRecordIter( path_imgrec="./sample.rec", # The target record file. data_shape=(3, 227, 227), # Output data shape; 227x227 region will be cropped from the original image. batch_size=4, # Number of items per batch. resize=256 # Resize the shorter edge to 256 before cropping. # You can specify more augmentation options. Use help(mx.io.ImageRecordIter) to see all the options. ) # You can now use the data_iter to access batches of images. batch = data_iter.next() # first batch. images = batch.data[0] # This will contain 4 (=batch_size) images each of 3x227x227. # process the images ... data_iter.reset() # To restart the iterator from the beginning. </pre></div> </div> Defined in src/io/iter_image_recordio_2.cc:L897 <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>path_imglist (string, optional, default='') – Path to the image list (.lst) file. Generally created with tools/im2rec.py. Format (Tab separated): <index of record> <one or more labels> <relative path from root folder>.</li> <li>path_imgrec (string, optional, default='') – Path to the image RecordIO (.rec) file or a directory path. Created with tools/im2rec.py.</li> <li>path_imgidx (string, optional, default='') – Path to the image RecordIO index (.idx) file. Created with tools/im2rec.py.</li> <li>aug_seq (string, optional, default='aug_default') – The augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.</li> <li>label_width (int, optional, default='1') – The number of labels per image.</li> <li>data_shape (Shape(tuple), required) – The shape of one output image in (channels, height, width) format.</li> <li>preprocess_threads (int, optional, default='4') – The number of threads to do preprocessing.</li> <li>verbose (boolean, optional, default=1) – If or not output verbose information.</li> <li>num_parts (int, optional, default='1') – Virtually partition the data into these many parts.</li> <li>part_index (int, optional, default='0') – The i-th virtual partition to be read.</li> <li>device_id (int, optional, default='0') – The device id used to create context for internal NDArray. Setting device_id to -1 will create Context::CPU(0). Setting device_id to valid positive device id will create Context::CPUPinned(device_id). Default is 0.</li> <li>shuffle_chunk_size (long (non-negative), optional, default=0) – The data shuffle buffer size in MB. Only valid if shuffle is true.</li> <li>shuffle_chunk_seed (int, optional, default='0') – The random seed for shuffling</li> <li>seed_aug (int or None, optional, default='None') – Random seed for augmentations.</li> <li>shuffle (boolean, optional, default=0) – Whether to shuffle data randomly or not.</li> <li>seed (int, optional, default='0') – The random seed.</li> <li>batch_size (int (non-negative), required) – Batch size.</li> <li>round_batch (boolean, optional, default=1) – Whether to use round robin to handle overflow batch or not.</li> <li>prefetch_buffer (long (non-negative), optional, default=4) – Maximum number of batches to prefetch.</li> <li>ctx ({'cpu', 'gpu'},optional, default='gpu') – Context data loader optimized for.</li> <li>dtype ({None, 'float16', 'float32', 'float64', 'int32', 'int64', 'int8', 'uint8'},optional, default='None') – Output data type. <code class="docutils literal">None</code> means no change.</li> <li>resize (int, optional, default='-1') – Down scale the shorter edge to a new size before applying other augmentations.</li> <li>rand_crop (boolean, optional, default=0) – If or not randomly crop the image</li> <li>random_resized_crop (boolean, optional, default=0) – If or not perform random resized cropping on the image, as a standard preprocessing for resnet training on ImageNet data.</li> <li>max_rotate_angle (int, optional, default='0') – Rotate by a random degree in <code class="docutils literal">[-v, v]</code></li> <li>max_aspect_ratio (float, optional, default=0) – Change the aspect (namely width/height) to a random value. If min_aspect_ratio is None then the aspect ratio ins sampled from [1 - max_aspect_ratio, 1 + max_aspect_ratio], else it is in <code class="docutils literal">[min_aspect_ratio, max_aspect_ratio]</code></li> <li>min_aspect_ratio (float or None, optional, default=None) – Change the aspect (namely width/height) to a random value in <code class="docutils literal">[min_aspect_ratio, max_aspect_ratio]</code></li> <li>max_shear_ratio (float, optional, default=0) – Apply a shear transformation (namely <code class="docutils literal">(x,y)->(x+my,y)</code>) with <code class="docutils literal">m</code> randomly chose from <code class="docutils literal">[-max_shear_ratio, max_shear_ratio]</code></li> <li>max_crop_size (int, optional, default='-1') – Crop both width and height into a random size in <code class="docutils literal">[min_crop_size, max_crop_size].``Ignored if ``random_resized_crop</code> is True.</li> <li>min_crop_size (int, optional, default='-1') – Crop both width and height into a random size in <code class="docutils literal">[min_crop_size, max_crop_size].``Ignored if ``random_resized_crop</code> is True.</li> <li>max_random_scale (float, optional, default=1) – Resize into <code class="docutils literal">[width*s, height*s]</code> with <code class="docutils literal">s</code> randomly chosen from <code class="docutils literal">[min_random_scale, max_random_scale]</code>. Ignored if <code class="docutils literal">random_resized_crop</code> is True.</li> <li>min_random_scale (float, optional, default=1) – Resize into <code class="docutils literal">[width*s, height*s]</code> with <code class="docutils literal">s</code> randomly chosen from <code class="docutils literal">[min_random_scale, max_random_scale]``Ignored if ``random_resized_crop</code> is True.</li> <li>max_random_area (float, optional, default=1) – Change the area (namely width * height) to a random value in <code class="docutils literal">[min_random_area, max_random_area]</code>. Ignored if <code class="docutils literal">random_resized_crop</code> is False.</li> <li>min_random_area (float, optional, default=1) – Change the area (namely width * height) to a random value in <code class="docutils literal">[min_random_area, max_random_area]</code>. Ignored if <code class="docutils literal">random_resized_crop</code> is False.</li> <li>max_img_size (float, optional, default=1e+10) – Set the maximal width and height after all resize and rotate argumentation are applied</li> <li>min_img_size (float, optional, default=0) – Set the minimal width and height after all resize and rotate argumentation are applied</li> <li>brightness (float, optional, default=0) – Add a random value in <code class="docutils literal">[-brightness, brightness]</code> to the brightness of image.</li> <li>contrast (float, optional, default=0) – Add a random value in <code class="docutils literal">[-contrast, contrast]</code> to the contrast of image.</li> <li>saturation (float, optional, default=0) – Add a random value in <code class="docutils literal">[-saturation, saturation]</code> to the saturation of image.</li> <li>pca_noise (float, optional, default=0) – Add PCA based noise to the image.</li> <li>random_h (int, optional, default='0') – Add a random value in <code class="docutils literal">[-random_h, random_h]</code> to the H channel in HSL color space.</li> <li>random_s (int, optional, default='0') – Add a random value in <code class="docutils literal">[-random_s, random_s]</code> to the S channel in HSL color space.</li> <li>random_l (int, optional, default='0') – Add a random value in <code class="docutils literal">[-random_l, random_l]</code> to the L channel in HSL color space.</li> <li>rotate (int, optional, default='-1') – Rotate by an angle. If set, it overwrites the <code class="docutils literal">max_rotate_angle</code> option.</li> <li>fill_value (int, optional, default='255') – Set the padding pixels value to <code class="docutils literal">fill_value</code>.</li> <li>inter_method (int, optional, default='1') – The interpolation method: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.</li> <li>pad (int, optional, default='0') – Change size from <code class="docutils literal">[width, height]</code> into <code class="docutils literal">[pad + width + pad, pad + height + pad]</code> by padding pixes</li> <li>mirror (boolean, optional, default=0) – Whether to mirror the image or not. If true, images are flipped along the horizontal axis.</li> <li>rand_mirror (boolean, optional, default=0) – Whether to randomly mirror images or not. If true, 50% of the images will be randomly mirrored (flipped along the horizontal axis)</li> <li>mean_img (string, optional, default='') – Filename of the mean image.</li> <li>mean_r (float, optional, default=0) – The mean value to be subtracted on the R channel</li> <li>mean_g (float, optional, default=0) – The mean value to be subtracted on the G channel</li> <li>mean_b (float, optional, default=0) – The mean value to be subtracted on the B channel</li> <li>mean_a (float, optional, default=0) – The mean value to be subtracted on the alpha channel</li> <li>std_r (float, optional, default=1) – Augmentation Param: Standard deviation on R channel.</li> <li>std_g (float, optional, default=1) – Augmentation Param: Standard deviation on G channel.</li> <li>std_b (float, optional, default=1) – Augmentation Param: Standard deviation on B channel.</li> <li>std_a (float, optional, default=1) – Augmentation Param: Standard deviation on Alpha channel.</li> <li>scale (float, optional, default=1) – Multiply the image with a scale value.</li> <li>max_random_contrast (float, optional, default=0) – Change the contrast with a value randomly chosen from <code class="docutils literal">[-max_random_contrast, max_random_contrast]</code></li> <li>max_random_illumination (float, optional, default=0) – Change the illumination with a value randomly chosen from <code class="docutils literal">[-max_random_illumination, max_random_illumination]</code></li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">The result iterator. </td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">MXDataIter </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt> <code class="descclassname">mxnet.io.</code><code class="descname">ImageRecordUInt8Iter</code>(*args, **kwargs)</dt> <dd>Iterating on image RecordIO files <div class="admonition note"> Note ImageRecordUInt8Iter is deprecated. Use ImageRecordIter(dtype=’uint8’) instead. </div> This iterator is identical to <code class="docutils literal">ImageRecordIter</code> except for using <code class="docutils literal">uint8</code> as the data type instead of <code class="docutils literal">float</code>. Defined in src/io/iter_image_recordio_2.cc:L916 <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>path_imglist (string, optional, default='') – Path to the image list (.lst) file. Generally created with tools/im2rec.py. Format (Tab separated): <index of record> <one or more labels> <relative path from root folder>.</li> <li>path_imgrec (string, optional, default='') – Path to the image RecordIO (.rec) file or a directory path. Created with tools/im2rec.py.</li> <li>path_imgidx (string, optional, default='') – Path to the image RecordIO index (.idx) file. Created with tools/im2rec.py.</li> <li>aug_seq (string, optional, default='aug_default') – The augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.</li> <li>label_width (int, optional, default='1') – The number of labels per image.</li> <li>data_shape (Shape(tuple), required) – The shape of one output image in (channels, height, width) format.</li> <li>preprocess_threads (int, optional, default='4') – The number of threads to do preprocessing.</li> <li>verbose (boolean, optional, default=1) – If or not output verbose information.</li> <li>num_parts (int, optional, default='1') – Virtually partition the data into these many parts.</li> <li>part_index (int, optional, default='0') – The i-th virtual partition to be read.</li> <li>device_id (int, optional, default='0') – The device id used to create context for internal NDArray. Setting device_id to -1 will create Context::CPU(0). Setting device_id to valid positive device id will create Context::CPUPinned(device_id). Default is 0.</li> <li>shuffle_chunk_size (long (non-negative), optional, default=0) – The data shuffle buffer size in MB. Only valid if shuffle is true.</li> <li>shuffle_chunk_seed (int, optional, default='0') – The random seed for shuffling</li> <li>seed_aug (int or None, optional, default='None') – Random seed for augmentations.</li> <li>shuffle (boolean, optional, default=0) – Whether to shuffle data randomly or not.</li> <li>seed (int, optional, default='0') – The random seed.</li> <li>batch_size (int (non-negative), required) – Batch size.</li> <li>round_batch (boolean, optional, default=1) – Whether to use round robin to handle overflow batch or not.</li> <li>prefetch_buffer (long (non-negative), optional, default=4) – Maximum number of batches to prefetch.</li> <li>ctx ({'cpu', 'gpu'},optional, default='gpu') – Context data loader optimized for.</li> <li>dtype ({None, 'float16', 'float32', 'float64', 'int32', 'int64', 'int8', 'uint8'},optional, default='None') – Output data type. <code class="docutils literal">None</code> means no change.</li> <li>resize (int, optional, default='-1') – Down scale the shorter edge to a new size before applying other augmentations.</li> <li>rand_crop (boolean, optional, default=0) – If or not randomly crop the image</li> <li>random_resized_crop (boolean, optional, default=0) – If or not perform random resized cropping on the image, as a standard preprocessing for resnet training on ImageNet data.</li> <li>max_rotate_angle (int, optional, default='0') – Rotate by a random degree in <code class="docutils literal">[-v, v]</code></li> <li>max_aspect_ratio (float, optional, default=0) – Change the aspect (namely width/height) to a random value. If min_aspect_ratio is None then the aspect ratio ins sampled from [1 - max_aspect_ratio, 1 + max_aspect_ratio], else it is in <code class="docutils literal">[min_aspect_ratio, max_aspect_ratio]</code></li> <li>min_aspect_ratio (float or None, optional, default=None) – Change the aspect (namely width/height) to a random value in <code class="docutils literal">[min_aspect_ratio, max_aspect_ratio]</code></li> <li>max_shear_ratio (float, optional, default=0) – Apply a shear transformation (namely <code class="docutils literal">(x,y)->(x+my,y)</code>) with <code class="docutils literal">m</code> randomly chose from <code class="docutils literal">[-max_shear_ratio, max_shear_ratio]</code></li> <li>max_crop_size (int, optional, default='-1') – Crop both width and height into a random size in <code class="docutils literal">[min_crop_size, max_crop_size].``Ignored if ``random_resized_crop</code> is True.</li> <li>min_crop_size (int, optional, default='-1') – Crop both width and height into a random size in <code class="docutils literal">[min_crop_size, max_crop_size].``Ignored if ``random_resized_crop</code> is True.</li> <li>max_random_scale (float, optional, default=1) – Resize into <code class="docutils literal">[width*s, height*s]</code> with <code class="docutils literal">s</code> randomly chosen from <code class="docutils literal">[min_random_scale, max_random_scale]</code>. Ignored if <code class="docutils literal">random_resized_crop</code> is True.</li> <li>min_random_scale (float, optional, default=1) – Resize into <code class="docutils literal">[width*s, height*s]</code> with <code class="docutils literal">s</code> randomly chosen from <code class="docutils literal">[min_random_scale, max_random_scale]``Ignored if ``random_resized_crop</code> is True.</li> <li>max_random_area (float, optional, default=1) – Change the area (namely width * height) to a random value in <code class="docutils literal">[min_random_area, max_random_area]</code>. Ignored if <code class="docutils literal">random_resized_crop</code> is False.</li> <li>min_random_area (float, optional, default=1) – Change the area (namely width * height) to a random value in <code class="docutils literal">[min_random_area, max_random_area]</code>. Ignored if <code class="docutils literal">random_resized_crop</code> is False.</li> <li>max_img_size (float, optional, default=1e+10) – Set the maximal width and height after all resize and rotate argumentation are applied</li> <li>min_img_size (float, optional, default=0) – Set the minimal width and height after all resize and rotate argumentation are applied</li> <li>brightness (float, optional, default=0) – Add a random value in <code class="docutils literal">[-brightness, brightness]</code> to the brightness of image.</li> <li>contrast (float, optional, default=0) – Add a random value in <code class="docutils literal">[-contrast, contrast]</code> to the contrast of image.</li> <li>saturation (float, optional, default=0) – Add a random value in <code class="docutils literal">[-saturation, saturation]</code> to the saturation of image.</li> <li>pca_noise (float, optional, default=0) – Add PCA based noise to the image.</li> <li>random_h (int, optional, default='0') – Add a random value in <code class="docutils literal">[-random_h, random_h]</code> to the H channel in HSL color space.</li> <li>random_s (int, optional, default='0') – Add a random value in <code class="docutils literal">[-random_s, random_s]</code> to the S channel in HSL color space.</li> <li>random_l (int, optional, default='0') – Add a random value in <code class="docutils literal">[-random_l, random_l]</code> to the L channel in HSL color space.</li> <li>rotate (int, optional, default='-1') – Rotate by an angle. If set, it overwrites the <code class="docutils literal">max_rotate_angle</code> option.</li> <li>fill_value (int, optional, default='255') – Set the padding pixels value to <code class="docutils literal">fill_value</code>.</li> <li>inter_method (int, optional, default='1') – The interpolation method: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.</li> <li>pad (int, optional, default='0') – Change size from <code class="docutils literal">[width, height]</code> into <code class="docutils literal">[pad + width + pad, pad + height + pad]</code> by padding pixes</li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">The result iterator. </td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">MXDataIter </td> </tr> </tbody> </table> </dd></dl> <dl class="function"> <dt> <code class="descclassname">mxnet.io.</code><code class="descname">MNISTIter</code>(*args, **kwargs)</dt> <dd>Iterating on the MNIST dataset. One can download the dataset from <a class="reference external" href="http://yann.lecun.com/exdb/mnist/">http://yann.lecun.com/exdb/mnist/</a> Defined in src/io/iter_mnist.cc:L265 <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>image (string, optional, default='./train-images-idx3-ubyte') – Dataset Param: Mnist image path.</li> <li>label (string, optional, default='./train-labels-idx1-ubyte') – Dataset Param: Mnist label path.</li> <li>batch_size (int, optional, default='128') – Batch Param: Batch Size.</li> <li>shuffle (boolean, optional, default=1) – Augmentation Param: Whether to shuffle data.</li> <li>flat (boolean, optional, default=0) – Augmentation Param: Whether to flat the data into 1D.</li> <li>seed (int, optional, default='0') – Augmentation Param: Random Seed.</li> <li>silent (boolean, optional, default=0) – Auxiliary Param: Whether to print out data info.</li> <li>num_parts (int, optional, default='1') – partition the data into multiple parts</li> <li>part_index (int, optional, default='0') – the index of the part will read</li> <li>prefetch_buffer (long (non-negative), optional, default=4) – Maximum number of batches to prefetch.</li> <li>ctx ({'cpu', 'gpu'},optional, default='gpu') – Context data loader optimized for.</li> <li>dtype ({None, 'float16', 'float32', 'float64', 'int32', 'int64', 'int8', 'uint8'},optional, default='None') – Output data type. <code class="docutils literal">None</code> means no change.</li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">The result iterator. </td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">MXDataIter </td> </tr> </tbody> </table> </dd></dl> </div> <div class="section" id="mxnet-io-helper-classes-functions"> <h3>mxnet.io - Helper Classes & Functions<a class="headerlink" href="#mxnet-io-helper-classes-functions" title="Permalink to this headline">¶</a></h3> Data iterators for common data formats and utility functions. <dl class="class"> <dt> class <code class="descclassname">mxnet.io.</code><code class="descname">DataBatch</code>(data, label=None, pad=None, index=None, bucket_key=None, provide_data=None, provide_label=None)<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataBatch">[source]</a></dt> <dd>A data batch. MXNet’s data iterator returns a batch of data for each <cite>next</cite> call. This data contains <cite>batch_size</cite> number of examples. If the input data consists of images, then shape of these images depend on the <cite>layout</cite> attribute of <cite>DataDesc</cite> object in <cite>provide_data</cite> parameter. If <cite>layout</cite> is set to ‘NCHW’ then, images should be stored in a 4-D matrix of shape <code class="docutils literal">(batch_size, num_channel, height, width)</code>. If <cite>layout</cite> is set to ‘NHWC’ then, images should be stored in a 4-D matrix of shape <code class="docutils literal">(batch_size, height, width, num_channel)</code>. The channels are often in RGB order. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>data (list of <cite>NDArray</cite>, each array containing <cite>batch_size</cite> examples.) – A list of input data.</li> <li>label (list of <cite>NDArray</cite>, each array often containing a 1-dimensional array. optional) – A list of input labels.</li> <li>pad (int, optional) – The number of examples padded at the end of a batch. It is used when the total number of examples read is not divisible by the <cite>batch_size</cite>. These extra padded examples are ignored in prediction.</li> <li>index (numpy.array, optional) – The example indices in this batch.</li> <li>bucket_key (int, optional) – The bucket key, used for bucketing module.</li> <li>provide_data (list of <cite>DataDesc</cite>, optional) – A list of <cite>DataDesc</cite> objects. <cite>DataDesc</cite> is used to store name, shape, type and layout information of the data. The i-th element describes the name and shape of <code class="docutils literal">data[i]</code>.</li> <li>provide_label (list of <cite>DataDesc</cite>, optional) – A list of <cite>DataDesc</cite> objects. <cite>DataDesc</cite> is used to store name, shape, type and layout information of the label. The i-th element describes the name and shape of <code class="docutils literal">label[i]</code>.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> <dl class="class"> <dt> class <code class="descclassname">mxnet.io.</code><code class="descname">DataDesc</code><a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataDesc">[source]</a></dt> <dd>DataDesc is used to store name, shape, type and layout information of the data or the label. The <cite>layout</cite> describes how the axes in <cite>shape</cite> should be interpreted, for example for image data setting <cite>layout=NCHW</cite> indicates that the first axis is number of examples in the batch(N), C is number of channels, H is the height and W is the width of the image. For sequential data, by default <cite>layout</cite> is set to <code class="docutils literal">NTC</code>, where N is number of examples in the batch, T the temporal axis representing time and C is the number of channels. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>cls (DataDesc) – The class.</li> <li>name (str) – Data name.</li> <li>shape (tuple of int) – Data shape.</li> <li>dtype (np.dtype, optional) – Data type.</li> <li>layout (str, optional) – Data layout.</li> </ul> </td> </tr> </tbody> </table> <dl class="staticmethod"> <dt> static <code class="descname">get_batch_axis</code>(layout)<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataDesc.get_batch_axis">[source]</a></dt> <dd>Get the dimension that corresponds to the batch size. When data parallelism is used, the data will be automatically split and concatenated along the batch-size dimension. Axis can be -1, which means the whole array will be copied for each data-parallelism device. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body">layout (str) – layout string. For example, “NCHW”.</td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">An axis indicating the batch_size dimension.</td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">int</td> </tr> </tbody> </table> </dd></dl> <dl class="staticmethod"> <dt> static <code class="descname">get_list</code>(shapes, types)<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataDesc.get_list">[source]</a></dt> <dd>Get DataDesc list from attribute lists. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>shapes (a tuple of (<a class="reference external" href="mxnet.symbol.Symbol.name">name</a>, <a class="reference external" href="mxnet.ndarray.NDArray.shape">shape</a>)) – </li> <li>types (a tuple of (<a class="reference external" href="mxnet.symbol.Symbol.name">name</a>, np.dtype)) – </li> </ul> </td> </tr> </tbody> </table> </dd></dl> </dd></dl> <dl class="class"> <dt> class <code class="descclassname">mxnet.io.</code><code class="descname">DataIter</code>(batch_size=0)<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter">[source]</a></dt> <dd>The base class for an MXNet data iterator. All I/O in MXNet is handled by specializations of this class. Data iterators in MXNet are similar to standard-iterators in Python. On each call to <cite>next</cite> they return a <cite>DataBatch</cite> which represents the next batch of data. When there is no more data to return, it raises a <cite>StopIteration</cite> exception. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body">batch_size (int, optional) – The batch size, namely the number of items in the batch.</td> </tr> </tbody> </table> <div class="admonition seealso"> See also <dl class="last docutils"> <dt><code class="xref py py-class docutils literal">NDArrayIter</code></dt> <dd>Data-iterator for MXNet NDArray or numpy-ndarray objects.</dd> <dt><code class="xref py py-class docutils literal">CSVIter</code></dt> <dd>Data-iterator for csv data.</dd> <dt><code class="xref py py-class docutils literal">LibSVMIter</code></dt> <dd>Data-iterator for libsvm data.</dd> <dt><code class="xref py py-class docutils literal">ImageIter</code></dt> <dd>Data-iterator for images.</dd> </dl> </div> <dl class="method"> <dt> <code class="descname">getdata</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter.getdata">[source]</a></dt> <dd>Get data of current batch. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">The data of the current batch.</td> </tr> <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">list of NDArray</td> </tr> </tbody> </table> </dd></dl> <dl class="method"> <dt> <code class="descname">getindex</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter.getindex">[source]</a></dt> <dd>Get index of the current batch. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">index – The indices of examples in the current batch.</td> </tr> <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">numpy.array</td> </tr> </tbody> </table> </dd></dl> <dl class="method"> <dt> <code class="descname">getlabel</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter.getlabel">[source]</a></dt> <dd>Get label of the current batch. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">The label of the current batch.</td> </tr> <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">list of NDArray</td> </tr> </tbody> </table> </dd></dl> <dl class="method"> <dt> <code class="descname">getpad</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter.getpad">[source]</a></dt> <dd>Get the number of padding examples in the current batch. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">Number of padding examples in the current batch.</td> </tr> <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">int</td> </tr> </tbody> </table> </dd></dl> <dl class="method"> <dt> <code class="descname">iter_next</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter.iter_next">[source]</a></dt> <dd>Move to the next batch. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">Whether the move is successful.</td> </tr> <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">boolean</td> </tr> </tbody> </table> </dd></dl> <dl class="method"> <dt> <code class="descname">next</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter.next">[source]</a></dt> <dd>Get next data batch from iterator. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">The data of next batch.</td> </tr> <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">DataBatch</td> </tr> <tr class="field-odd field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal">StopIteration</code> – If the end of the data is reached.</td> </tr> </tbody> </table> </dd></dl> <dl class="method"> <dt> <code class="descname">reset</code>()<a class="reference internal" href="../../../_modules/mxnet/io/io.html#DataIter.reset">[source]</a></dt> <dd>Reset the iterator to the begin of the data. </dd></dl> </dd></dl> <dl class="class"> <dt> class <code class="descclassname">mxnet.io.</code><code class="descname">MXDataIter</code>(handle, data_name='data', label_name='softmax_label', **_)<a class="reference internal" href="../../../_modules/mxnet/io/io.html#MXDataIter">[source]</a></dt> <dd>A python wrapper a C++ data iterator. This iterator is the Python wrapper to all native C++ data iterators, such as <cite>CSVIter</cite>, <cite>ImageRecordIter</cite>, <cite>MNISTIter</cite>, etc. When initializing <cite>CSVIter</cite> for example, you will get an <cite>MXDataIter</cite> instance to use in your Python code. Calls to <cite>next</cite>, <cite>reset</cite>, etc will be delegated to the underlying C++ data iterators. Usually you don’t need to interact with <cite>MXDataIter</cite> directly unless you are implementing your own data iterators in C++. To do that, please refer to examples under the <cite>src/io</cite> folder. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>handle (DataIterHandle, required) – The handle to the underlying C++ Data Iterator.</li> <li>data_name (str, optional) – Data name. Default to “data”.</li> <li>label_name (str, optional) – Label name. Default to “softmax_label”.</li> </ul> </td> </tr> </tbody> </table> <div class="admonition seealso"> See also src/io : The underlying C++ data iterator implementation, e.g., <cite>CSVIter</cite>. </div> </dd></dl> <dl class="class"> <dt> class <code class="descclassname">mxnet.io.</code><code class="descname">PrefetchingIter</code>(iters, rename_data=None, rename_label=None)<a class="reference internal" href="../../../_modules/mxnet/io/io.html#PrefetchingIter">[source]</a></dt> <dd>Performs pre-fetch for other data iterators. This iterator will create another thread to perform <code class="docutils literal">iter_next</code> and then store the data in memory. It potentially accelerates the data read, at the cost of more memory usage. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>iters (DataIter or list of DataIter) – The data iterators to be pre-fetched.</li> <li>rename_data (None or list of dict) – The i-th element is a renaming map for the i-th iter, in the form of {‘original_name’ : ‘new_name’}. Should have one entry for each entry in iter[i].provide_data.</li> <li>rename_label (None or list of dict) – Similar to <code class="docutils literal">rename_data</code>.</li> </ul> </td> </tr> </tbody> </table> Examples <div class="highlight-default"><div class="highlight"><pre>>>> iter1 = mx.io.NDArrayIter({'data':mx.nd.ones((100,10))}, batch_size=25) >>> iter2 = mx.io.NDArrayIter({'data':mx.nd.ones((100,10))}, batch_size=25) >>> piter = mx.io.PrefetchingIter([iter1, iter2], ... rename_data=[{'data': 'data_1'}, {'data': 'data_2'}]) >>> print(piter.provide_data) [DataDesc[data_1,(25, 10L),<type 'numpy.float32'>,NCHW], DataDesc[data_2,(25, 10L),<type 'numpy.float32'>,NCHW]] </pre></div> </div> </dd></dl> <dl class="class"> <dt> class <code class="descclassname">mxnet.io.</code><code class="descname">ResizeIter</code>(data_iter, size, reset_internal=True)<a class="reference internal" href="../../../_modules/mxnet/io/io.html#ResizeIter">[source]</a></dt> <dd>Resize a data iterator to a given number of batches. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>data_iter (DataIter) – The data iterator to be resized.</li> <li>size (int) – The number of batches per epoch to resize to.</li> <li>reset_internal (bool) – Whether to reset internal iterator on ResizeIter.reset.</li> </ul> </td> </tr> </tbody> </table> Examples <div class="highlight-default"><div class="highlight"><pre>>>> nd_iter = mx.io.NDArrayIter(mx.nd.ones((100,10)), batch_size=25) >>> resize_iter = mx.io.ResizeIter(nd_iter, 2) >>> for batch in resize_iter: ... print(batch.data) [<NDArray 25x10 @cpu(0)>] [<NDArray 25x10 @cpu(0)>] </pre></div> </div> </dd></dl> </div> <div class="section" id="module-mxnet.recordio"> <h3>mxnet.recordio<a class="headerlink" href="#module-mxnet.recordio" title="Permalink to this headline">¶</a></h3> Read and write for the RecordIO data format. <dl class="class"> <dt id="mxnet.recordio.MXRecordIO"> class <code class="descclassname">mxnet.recordio.</code><code class="descname">MXRecordIO</code>(uri, flag)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO">[source]</a><a class="headerlink" href="#mxnet.recordio.MXRecordIO" title="Permalink to this definition">¶</a></dt> <dd>Reads/writes <cite>RecordIO</cite> data format, supporting sequential read and write. Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXRecordIO('tmp.rec', 'w') <mxnet.recordio.MXRecordIO object at 0x10ef40ed0> >>> for i in range(5): ... record.write('record_%d'%i) >>> record.close() >>> record = mx.recordio.MXRecordIO('tmp.rec', 'r') >>> for i in range(5): ... item = record.read() ... print(item) record_0 record_1 record_2 record_3 record_4 >>> record.close() </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>uri (string) – Path to the record file.</li> <li>flag (string) – ‘w’ for write or ‘r’ for read.</li> </ul> </td> </tr> </tbody> </table> <dl class="method"> <dt id="mxnet.recordio.MXRecordIO.open"> <code class="descname">open</code>()<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.open">[source]</a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.open" title="Permalink to this definition">¶</a></dt> <dd>Opens the record file. </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXRecordIO.close"> <code class="descname">close</code>()<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.close">[source]</a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.close" title="Permalink to this definition">¶</a></dt> <dd>Closes the record file. </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXRecordIO.reset"> <code class="descname">reset</code>()<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.reset">[source]</a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.reset" title="Permalink to this definition">¶</a></dt> <dd>Resets the pointer to first item. If the record is opened with ‘w’, this function will truncate the file to empty. Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXRecordIO('tmp.rec', 'r') >>> for i in range(2): ... item = record.read() ... print(item) record_0 record_1 >>> record.reset() # Pointer is reset. >>> print(record.read()) # Started reading from start again. record_0 >>> record.close() </pre></div> </div> </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXRecordIO.write"> <code class="descname">write</code>(buf)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.write">[source]</a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.write" title="Permalink to this definition">¶</a></dt> <dd>Inserts a string buffer as a record. Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXRecordIO('tmp.rec', 'w') >>> for i in range(5): ... record.write('record_%d'%i) >>> record.close() </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body">buf (string (python2), bytes (python3)) – Buffer to write.</td> </tr> </tbody> </table> </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXRecordIO.read"> <code class="descname">read</code>()<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.read">[source]</a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.read" title="Permalink to this definition">¶</a></dt> <dd>Returns record as a string. Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXRecordIO('tmp.rec', 'r') >>> for i in range(5): ... item = record.read() ... print(item) record_0 record_1 record_2 record_3 record_4 >>> record.close() </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">buf – Buffer read.</td> </tr> <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">string</td> </tr> </tbody> </table> </dd></dl> </dd></dl> <dl class="class"> <dt id="mxnet.recordio.MXIndexedRecordIO"> class <code class="descclassname">mxnet.recordio.</code><code class="descname">MXIndexedRecordIO</code>(idx_path, uri, flag, key_type=<type 'int'>)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO">[source]</a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO" title="Permalink to this definition">¶</a></dt> <dd>Reads/writes <cite>RecordIO</cite> data format, supporting random access. Examples <div class="highlight-default"><div class="highlight"><pre>>>> for i in range(5): ... record.write_idx(i, 'record_%d'%i) >>> record.close() >>> record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'r') >>> record.read_idx(3) record_3 </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>idx_path (str) – Path to the index file.</li> <li>uri (str) – Path to the record file. Only supports seekable file types.</li> <li>flag (str) – ‘w’ for write or ‘r’ for read.</li> <li>key_type (<a class="reference internal" href="../kvstore/kvstore.html#mxnet.kvstore.KVStore.type" title="mxnet.kvstore.KVStore.type">type</a>) – Data type for keys.</li> </ul> </td> </tr> </tbody> </table> <dl class="method"> <dt id="mxnet.recordio.MXIndexedRecordIO.close"> <code class="descname">close</code>()<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.close">[source]</a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.close" title="Permalink to this definition">¶</a></dt> <dd>Closes the record file. </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXIndexedRecordIO.seek"> <code class="descname">seek</code>(idx)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.seek">[source]</a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.seek" title="Permalink to this definition">¶</a></dt> <dd>Sets the current read pointer position. This function is internally called by <cite>read_idx(idx)</cite> to find the current reader pointer position. It doesn’t return anything. </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXIndexedRecordIO.tell"> <code class="descname">tell</code>()<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.tell">[source]</a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.tell" title="Permalink to this definition">¶</a></dt> <dd>Returns the current position of write head. Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'w') >>> print(record.tell()) 0 >>> for i in range(5): ... record.write_idx(i, 'record_%d'%i) ... print(record.tell()) 16 32 48 64 80 </pre></div> </div> </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXIndexedRecordIO.read_idx"> <code class="descname">read_idx</code>(idx)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.read_idx">[source]</a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.read_idx" title="Permalink to this definition">¶</a></dt> <dd>Returns the record at given index. Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'w') >>> for i in range(5): ... record.write_idx(i, 'record_%d'%i) >>> record.close() >>> record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'r') >>> record.read_idx(3) record_3 </pre></div> </div> </dd></dl> <dl class="method"> <dt id="mxnet.recordio.MXIndexedRecordIO.write_idx"> <code class="descname">write_idx</code>(idx, buf)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.write_idx">[source]</a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.write_idx" title="Permalink to this definition">¶</a></dt> <dd>Inserts input record at given index. Examples <div class="highlight-default"><div class="highlight"><pre>>>> for i in range(5): ... record.write_idx(i, 'record_%d'%i) >>> record.close() </pre></div> </div> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>idx (int) – Index of a file.</li> <li>buf – Record to write.</li> </ul> </td> </tr> </tbody> </table> </dd></dl> </dd></dl> <dl class="attribute"> <dt id="mxnet.recordio.IRHeader"> <code class="descclassname">mxnet.recordio.</code><code class="descname">IRHeader</code><a class="headerlink" href="#mxnet.recordio.IRHeader" title="Permalink to this definition">¶</a></dt> <dd>An alias for HEADER. Used to store metadata (e.g. labels) accompanying a record. See mxnet.recordio.pack and mxnet.recordio.pack_img for example uses. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <li>flag (int) – Available for convenience, can be set arbitrarily.</li> <li>label (float or an array of float) – Typically used to store label(s) for a record.</li> <li>id (int) – Usually a unique id representing record.</li> <li>id2 (int) – Higher order bits of the unique id, should be set to 0 (in most cases).</li> </ul> </td> </tr> </tbody> </table> alias of <code class="xref py py-class docutils literal">HEADER</code> </dd></dl> <dl class="function"> <dt id="mxnet.recordio.pack"> <code class="descclassname">mxnet.recordio.</code><code class="descname">pack</code>(header, s)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#pack">[source]</a><a class="headerlink" href="#mxnet.recordio.pack" title="Permalink to this definition">¶</a></dt> <dd>Pack a string into MXImageRecord. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>header (<a class="reference internal" href="#mxnet.recordio.IRHeader" title="mxnet.recordio.IRHeader">IRHeader</a>) – Header of the image record. <code class="docutils literal">header.label</code> can be a number or an array. See more detail in <code class="docutils literal">IRHeader</code>.</li> <li>s (str) – Raw image string to be packed.</li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">s – The packed string. </td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">str </td> </tr> </tbody> </table> Examples <div class="highlight-default"><div class="highlight"><pre>>>> label = 4 # label can also be a 1-D array, for example: label = [1,2,3] >>> id = 2574 >>> header = mx.recordio.IRHeader(0, label, id, 0) >>> with open(path, 'r') as file: ... s = file.read() >>> packed_s = mx.recordio.pack(header, s) </pre></div> </div> </dd></dl> <dl class="function"> <dt id="mxnet.recordio.unpack"> <code class="descclassname">mxnet.recordio.</code><code class="descname">unpack</code>(s)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#unpack">[source]</a><a class="headerlink" href="#mxnet.recordio.unpack" title="Permalink to this definition">¶</a></dt> <dd>Unpack a MXImageRecord to string. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body">s (str) – String buffer from <code class="docutils literal">MXRecordIO.read</code>.</td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><ul class="simple"> <li>header (IRHeader) – Header of the image record.</li> <li>s (str) – Unpacked string.</li> </ul> </td> </tr> </tbody> </table> Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXRecordIO('test.rec', 'r') >>> item = record.read() >>> header, s = mx.recordio.unpack(item) >>> header HEADER(flag=0, label=14.0, id=20129312, id2=0) </pre></div> </div> </dd></dl> <dl class="function"> <dt id="mxnet.recordio.unpack_img"> <code class="descclassname">mxnet.recordio.</code><code class="descname">unpack_img</code>(s, iscolor=-1)<a class="reference internal" href="../../../_modules/mxnet/recordio.html#unpack_img">[source]</a><a class="headerlink" href="#mxnet.recordio.unpack_img" title="Permalink to this definition">¶</a></dt> <dd>Unpack a MXImageRecord to image. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>s (str) – String buffer from <code class="docutils literal">MXRecordIO.read</code>.</li> <li>iscolor (int) – Image format option for <code class="docutils literal">cv2.imdecode</code>.</li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><ul class="simple"> <li>header (IRHeader) – Header of the image record.</li> <li>img (numpy.ndarray) – Unpacked image.</li> </ul> </td> </tr> </tbody> </table> Examples <div class="highlight-default"><div class="highlight"><pre>>>> record = mx.recordio.MXRecordIO('test.rec', 'r') >>> item = record.read() >>> header, img = mx.recordio.unpack_img(item) >>> header HEADER(flag=0, label=14.0, id=20129312, id2=0) >>> img array([[[ 23, 27, 45], [ 28, 32, 50], ..., [ 36, 40, 59], [ 35, 39, 58]], ..., [[ 91, 92, 113], [ 97, 98, 119], ..., [168, 169, 167], [166, 167, 165]]], dtype=uint8) </pre></div> </div> </dd></dl> <dl class="function"> <dt id="mxnet.recordio.pack_img"> <code class="descclassname">mxnet.recordio.</code><code class="descname">pack_img</code>(header, img, quality=95, img_fmt='.jpg')<a class="reference internal" href="../../../_modules/mxnet/recordio.html#pack_img">[source]</a><a class="headerlink" href="#mxnet.recordio.pack_img" title="Permalink to this definition">¶</a></dt> <dd>Pack an image into <code class="docutils literal">MXImageRecord</code>. <table class="docutils field-list" frame="void" rules="none"> <col class="field-name"/> <col class="field-body"/> <tbody valign="top"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> <li>header (<a class="reference internal" href="#mxnet.recordio.IRHeader" title="mxnet.recordio.IRHeader">IRHeader</a>) – Header of the image record. <code class="docutils literal">header.label</code> can be a number or an array. See more detail in <code class="docutils literal">IRHeader</code>.</li> <li>img (numpy.ndarray) – Image to be packed.</li> <li>quality (int) – Quality for JPEG encoding in range 1-100, or compression for PNG encoding in range 1-9.</li> <li>img_fmt (str) – Encoding of the image (.jpg for JPEG, .png for PNG).</li> </ul> </td> </tr> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">s – The packed string. </td> </tr> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">str </td> </tr> </tbody> </table> Examples <div class="highlight-default"><div class="highlight"><pre>>>> label = 4 # label can also be a 1-D array, for example: label = [1,2,3] >>> id = 2574 >>> header = mx.recordio.IRHeader(0, label, id, 0) >>> img = cv2.imread('test.jpg') >>> packed_s = mx.recordio.pack_img(header, img) </pre></div> </div> </dd></dl> <script>auto_index("api-reference");</script></div> </div> </div> </div> </div> <div aria-label="main navigation" class="sphinxsidebar rightsidebar" role="navigation"> <div class="sphinxsidebarwrapper"> <h3><a href="../../../index.html">Table Of Contents</a></h3> <ul> <li><a class="reference internal" href="#">Data Loading API</a><ul> <li><a class="reference internal" href="#overview">Overview</a></li> <li><a class="reference internal" href="#data-iterators">Data iterators</a></li> <li><a class="reference internal" href="#helper-classes-and-functions">Helper classes and functions</a><ul> <li><a class="reference internal" href="#data-structures-and-other-iterators">Data structures and other iterators</a></li> <li><a class="reference internal" href="#functions-to-read-and-write-recordio-files">Functions to read and write RecordIO files</a></li> </ul> </li> <li><a class="reference internal" href="#how-to-develop-a-new-iterator">How to develop a new iterator</a><ul> <li><a class="reference internal" href="#how-to-change-the-batch-layout">How to change the batch layout</a></li> </ul> </li> <li><a class="reference internal" href="#api-reference">API Reference</a><ul> <li><a class="reference internal" href="#mxnet-io-data-iterators">mxnet.io - Data Iterators</a></li> <li><a class="reference internal" href="#mxnet-io-helper-classes-functions">mxnet.io - Helper Classes & Functions</a></li> <li><a class="reference internal" href="#module-mxnet.recordio">mxnet.recordio</a></li> </ul> </li> </ul> </li> </ul> </div> </div> </div><div class="footer"> <div class="section-disclaimer"> <div class="container"> <div> <img height="60" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/apache_incubator_logo.png"/> Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. "Copyright © 2017-2018, The Apache Software Foundation Apache MXNet, MXNet, Apache, the Apache feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the Apache Software Foundation." </div> </div> </div> </div>  </div> <script crossorigin="anonymous" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script> <script src="../../../_static/js/sidebar.js" type="text/javascript"></script> <script src="../../../_static/js/search.js" type="text/javascript"></script> <script src="../../../_static/js/navbar.js" type="text/javascript"></script> <script src="../../../_static/js/clipboard.min.js" type="text/javascript"></script> <script src="../../../_static/js/copycode.js" type="text/javascript"></script> <script src="../../../_static/js/page.js" type="text/javascript"></script> <script src="../../../_static/js/docversion.js" type="text/javascript"></script> <script type="text/javascript"> $('body').ready(function () { $('body').css('visibility', 'visible'); }); </script> </body> </html>

versions/1.5.0/api/python/io/io.html (1,739 lines of code) (raw):