versions/0.11.0/tutorials/python/matrix

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"/> <meta content="IE=edge" http-equiv="X-UA-Compatible"/> <meta content="width=device-width, initial-scale=1" name="viewport"/> <meta content="Matrix Factorization" property="og:title"> <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image"> <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image:secure_url"> <meta content="Matrix Factorization" property="og:description"/> <title>Matrix Factorization — mxnet documentation</title> <link crossorigin="anonymous" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" rel="stylesheet"/> <link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" rel="stylesheet"/> <link href="../../_static/basic.css" rel="stylesheet" type="text/css"> <link href="../../_static/pygments.css" rel="stylesheet" type="text/css"> <link href="../../_static/mxnet.css" rel="stylesheet" type="text/css"/> <script type="text/javascript"> var DOCUMENTATION_OPTIONS = { URL_ROOT: '../../', VERSION: '', COLLAPSE_INDEX: false, FILE_SUFFIX: '.html', HAS_SOURCE: true, SOURCELINK_SUFFIX: '.txt' }; </script> <script src="https://code.jquery.com/jquery-1.11.1.min.js" type="text/javascript"></script> <script src="../../_static/underscore.js" type="text/javascript"></script> <script src="../../_static/searchtools_custom.js" type="text/javascript"></script> <script src="../../_static/doctools.js" type="text/javascript"></script> <script src="../../_static/selectlang.js" type="text/javascript"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script> <script type="text/javascript"> jQuery(function() { Search.loadIndex("/versions/0.11.0/searchindex.js"); Search.init();}); </script>          <link href="../../genindex.html" rel="index" title="Index"> <link href="../../search.html" rel="search" title="Search"/> <link href="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-icon.png" rel="icon" type="image/png"/> </link></link></link></meta></meta></meta></head> <body background="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-background-compressed.jpeg" role="document"> <div class="content-block"><div class="navbar navbar-fixed-top"> <div class="container" id="navContainer"> <div class="innder" id="header-inner"> <h1 id="logo-wrap"> <a href="../../" id="logo"><img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet_logo.png"/></a> </h1> <nav class="nav-bar" id="main-nav"> <a class="main-nav-link" href="/versions/0.11.0/get_started/install.html">Install</a> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Gluon </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> <li><a class="main-nav-link" href="/versions/0.11.0/tutorials/gluon/gluon.html">About</a></li> <li><a class="main-nav-link" href="https://www.d2l.ai/">Dive into Deep Learning</a></li> <li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li> <li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li> </ul> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">API </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> <li><a class="main-nav-link" href="/versions/0.11.0/api/python/index.html">Python</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/c++/index.html">C++</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/julia/index.html">Julia</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/perl/index.html">Perl</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/r/index.html">R</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/scala/index.html">Scala</a></li> </ul> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Docs </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-docs"> <li><a class="main-nav-link" href="/versions/0.11.0/how_to/faq.html">FAQ</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/tutorials/index.html">Tutorials</a> <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/v0.11.0/example">Examples</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/architecture/index.html">Architecture</a></li> <li><a class="main-nav-link" href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home">Developer Wiki</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/model_zoo/index.html">Model Zoo</a></li> <li><a class="main-nav-link" href="https://github.com/onnx/onnx-mxnet">ONNX</a></li> </li></ul> <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Community </a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-community"> <li><a class="main-nav-link" href="http://discuss.mxnet.io">Forum</a></li> <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/v0.11.0">Github</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/community/contribute.html">Contribute</a></li> </ul> <a href="#" class="main-nav-link dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="true">0.11.0</a><ul id="package-dropdown-menu" class="dropdown-menu"><li><a href="/">master</a></li><li><a href="/versions/1.7.0/">1.7.0</a></li><li><a href=/versions/1.6.0/>1.6.0</a></li><li><a href=/versions/1.5.0/>1.5.0</a></li><li><a href=/versions/1.4.1/>1.4.1</a></li><li><a href=/versions/1.3.1/>1.3.1</a></li><li><a href=/versions/1.2.1/>1.2.1</a></li><li><a href=/versions/1.1.0/>1.1.0</a></li><li><a href=/versions/1.0.0/>1.0.0</a></li><li><a href=/versions/0.12.1/>0.12.1</a></li><li><a href=/versions/0.11.0/>0.11.0</a></li></ul></nav> <script> function getRootPath(){ return "../../" } </script> <div class="burgerIcon dropdown"> <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">☰</a> <ul class="dropdown-menu" id="burgerMenu"> <li><a href="/versions/0.11.0/get_started/install.html">Install</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/tutorials/index.html">Tutorials</a></li> <li class="dropdown-submenu dropdown"> <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Gluon</a> <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> <li><a class="main-nav-link" href="/versions/0.11.0/tutorials/gluon/gluon.html">About</a></li> <li><a class="main-nav-link" href="http://gluon.mxnet.io">The Straight Dope (Tutorials)</a></li> <li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li> <li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li> </ul> </li> <li class="dropdown-submenu"> <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">API</a> <ul class="dropdown-menu"> <li><a class="main-nav-link" href="/versions/0.11.0/api/python/index.html">Python</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/c++/index.html">C++</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/julia/index.html">Julia</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/perl/index.html">Perl</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/r/index.html">R</a></li> <li><a class="main-nav-link" href="/versions/0.11.0/api/scala/index.html">Scala</a></li> </ul> </li> <li class="dropdown-submenu"> <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Docs</a> <ul class="dropdown-menu"> <li><a href="/versions/0.11.0/how_to/faq.html" tabindex="-1">FAQ</a></li> <li><a href="/versions/0.11.0/tutorials/index.html" tabindex="-1">Tutorials</a></li> <li><a href="https://github.com/apache/incubator-mxnet/tree/v0.11.0/example" tabindex="-1">Examples</a></li> <li><a href="/versions/0.11.0/architecture/index.html" tabindex="-1">Architecture</a></li> <li><a href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home" tabindex="-1">Developer Wiki</a></li> <li><a href="/versions/0.11.0/model_zoo/index.html" tabindex="-1">Gluon Model Zoo</a></li> <li><a href="https://github.com/onnx/onnx-mxnet" tabindex="-1">ONNX</a></li> </ul> </li> <li class="dropdown-submenu dropdown"> <a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">Community</a> <ul class="dropdown-menu"> <li><a href="http://discuss.mxnet.io" tabindex="-1">Forum</a></li> <li><a href="https://github.com/apache/incubator-mxnet/tree/v0.11.0" tabindex="-1">Github</a></li> <li><a href="/versions/0.11.0/community/contribute.html" tabindex="-1">Contribute</a></li> </ul> </li> <li id="dropdown-menu-position-anchor-version-mobile" class="dropdown-submenu" style="position: relative"><a href="#" tabindex="-1">0.11.0</a><ul class="dropdown-menu"><li><a tabindex="-1" href=/>master</a></li><li><a tabindex="-1" href=/versions/1.6.0/>1.6.0</a></li><li><a tabindex="-1" href=/versions/1.5.0/>1.5.0</a></li><li><a tabindex="-1" href=/versions/1.4.1/>1.4.1</a></li><li><a tabindex="-1" href=/versions/1.3.1/>1.3.1</a></li><li><a tabindex="-1" href=/versions/1.2.1/>1.2.1</a></li><li><a tabindex="-1" href=/versions/1.1.0/>1.1.0</a></li><li><a tabindex="-1" href=/versions/1.0.0/>1.0.0</a></li><li><a tabindex="-1" href=/versions/0.12.1/>0.12.1</a></li><li><a tabindex="-1" href=/versions/0.11.0/>0.11.0</a></li></ul></li></ul> </div> <div class="plusIcon dropdown"> <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button"></a> <ul class="dropdown-menu dropdown-menu-right" id="plusMenu"></ul> </div> <div id="search-input-wrap"> <form action="../../search.html" autocomplete="off" class="" method="get" role="search"> <div class="form-group inner-addon left-addon"> <input class="form-control" name="q" placeholder="Search" type="text"/> </div> <input name="check_keywords" type="hidden" value="yes"> <input name="area" type="hidden" value="default"/> </input></form> <div id="search-preview"></div> </div> <div id="searchIcon"> </div>            </div> </div> </div> <script type="text/javascript"> $('body').css('background', 'white'); </script> <div class="container"> <div class="row"> <div aria-label="main navigation" class="sphinxsidebar leftsidebar" role="navigation"> <div class="sphinxsidebarwrapper"> <ul> <li class="toctree-l1"><a class="reference internal" href="../../api/python/index.html">Python Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../../api/r/index.html">R Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../../api/julia/index.html">Julia Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../../api/c++/index.html">C++ Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../../api/scala/index.html">Scala Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../../api/perl/index.html">Perl Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../../how_to/index.html">HowTo Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../../architecture/index.html">System Documents</a></li> <li class="toctree-l1"><a class="reference internal" href="../index.html">Tutorials</a></li> </ul> </div> </div> <div class="content"> <div class="page-tracker"></div> <div class="section" id="matrix-factorization"> <h1>Matrix Factorization<a class="headerlink" href="#matrix-factorization" title="Permalink to this headline">¶</a></h1> In a recommendation system, there is a group of users and a set of items. Given that each users have rated some items in the system, we would like to predict how the users would rate the items that they have not yet rated, such that we can make recommendations to the users. Matrix factorization is one of the mainly used algorithm in recommendation systems. It can be used to discover latent features underlying the interactions between two different kinds of entities. Assume we assign a k-dimensional vector to each user and a k-dimensional vector to each item such that the dot product of these two vectors gives the user’s rating of that item. We can learn the user and item vectors directly, which is essentially performing SVD on the user-item matrix. We can also try to learn the latent features using multi-layer neural networks. In this tutorial, we will work though the steps to implement these ideas in MXNet. <div class="section" id="prepare-data"> <h2>Prepare Data<a class="headerlink" href="#prepare-data" title="Permalink to this headline">¶</a></h2> We use the <a class="reference external" href="http://grouplens.org/datasets/movielens/">MovieLens</a> data here, but it can apply to other datasets as well. Each row of this dataset contains a tuple of user id, movie id, rating, and time stamp, we will only use the first three items. We first define the a batch which contains n tuples. It also provides name and shape information to MXNet about the data and label. <div class="highlight-python"><div class="highlight"><pre>class Batch(object): def __init__(self, data_names, data, label_names, label): self.data = data self.label = label self.data_names = data_names self.label_names = label_names @property def provide_data(self): return [(n, x.shape) for n, x in zip(self.data_names, self.data)] @property def provide_label(self): return [(n, x.shape) for n, x in zip(self.label_names, self.label)] </pre></div> </div> Then we define a data iterator, which returns a batch of tuples each time. <div class="highlight-python"><div class="highlight"><pre>import mxnet as mx import random class Batch(object): def __init__(self, data_names, data, label_names, label): self.data = data self.label = label self.data_names = data_names self.label_names = label_names @property def provide_data(self): return [(n, x.shape) for n, x in zip(self.data_names, self.data)] @property def provide_label(self): return [(n, x.shape) for n, x in zip(self.label_names, self.label)] class DataIter(mx.io.DataIter): def __init__(self, fname, batch_size): super(DataIter, self).__init__() self.batch_size = batch_size self.data = [] for line in file(fname): tks = line.strip().split('\t') if len(tks) != 4: continue self.data.append((int(tks[0]), int(tks[1]), float(tks[2]))) self.provide_data = [('user', (batch_size, )), ('item', (batch_size, ))] self.provide_label = [('score', (self.batch_size, ))] def __iter__(self): for k in range(len(self.data) / self.batch_size): users = [] items = [] scores = [] for i in range(self.batch_size): j = k * self.batch_size + i user, item, score = self.data[j] users.append(user) items.append(item) scores.append(score) data_all = [mx.nd.array(users), mx.nd.array(items)] label_all = [mx.nd.array(scores)] data_names = ['user', 'item'] label_names = ['score'] data_batch = Batch(data_names, data_all, label_names, label_all) yield data_batch def reset(self): random.shuffle(self.data) </pre></div> </div> Now we download the data and provide a function to obtain the data iterator: <div class="highlight-python"><div class="highlight"><pre>import os import urllib import zipfile if not os.path.exists('ml-100k.zip'): urllib.urlretrieve('http://files.grouplens.org/datasets/movielens/ml-100k.zip', 'ml-100k.zip') with zipfile.ZipFile("ml-100k.zip","r") as f: f.extractall("./") def get_data(batch_size): return (DataIter('./ml-100k/u1.base', batch_size), DataIter('./ml-100k/u1.test', batch_size)) </pre></div> </div> Finally we calculate the numbers of users and items for later use. <div class="highlight-python"><div class="highlight"><pre>def max_id(fname): mu = 0 mi = 0 for line in file(fname): tks = line.strip().split('\t') if len(tks) != 4: continue mu = max(mu, int(tks[0])) mi = max(mi, int(tks[1])) return mu + 1, mi + 1 max_user, max_item = max_id('./ml-100k/u.data') (max_user, max_item) </pre></div> </div> </div> <div class="section" id="optimization"> <h2>Optimization<a class="headerlink" href="#optimization" title="Permalink to this headline">¶</a></h2> We first implement the RMSE (root-mean-square error) measurement, which is commonly used by matrix factorization. <div class="highlight-python"><div class="highlight"><pre>import math def RMSE(label, pred): ret = 0.0 n = 0.0 pred = pred.flatten() for i in range(len(label)): ret += (label[i] - pred[i]) * (label[i] - pred[i]) n += 1.0 return math.sqrt(ret / n) </pre></div> </div> Then we define a general training module, which is borrowed from the image classification application. <div class="highlight-python"><div class="highlight"><pre>def train(network, batch_size, num_epoch, learning_rate): model = mx.model.FeedForward( ctx = mx.gpu(0), symbol = network, num_epoch = num_epoch, learning_rate = learning_rate, wd = 0.0001, momentum = 0.9) batch_size = 64 train, test = get_data(batch_size) import logging head = '%(asctime)-15s %(message)s' logging.basicConfig(level=logging.DEBUG) model.fit(X = train, eval_data = test, eval_metric = RMSE, batch_end_callback=mx.callback.Speedometer(batch_size, 20000/batch_size),) </pre></div> </div> </div> <div class="section" id="networks"> <h2>Networks<a class="headerlink" href="#networks" title="Permalink to this headline">¶</a></h2> Now we try various networks. We first learn the latent vectors directly. <div class="highlight-python"><div class="highlight"><pre>def plain_net(k): # input user = mx.symbol.Variable('user') item = mx.symbol.Variable('item') score = mx.symbol.Variable('score') # user feature lookup user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k) # item feature lookup item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k) # predict by the inner product, which is elementwise product and then sum pred = user * item pred = mx.symbol.sum_axis(data = pred, axis = 1) pred = mx.symbol.Flatten(data = pred) # loss layer pred = mx.symbol.LinearRegressionOutput(data = pred, label = score) return pred train(plain_net(64), batch_size=64, num_epoch=10, learning_rate=.05) </pre></div> </div> Next we try to use 2 layers neural network to learn the latent variables, which stack a fully connected layer above the embedding layers: <div class="highlight-python"><div class="highlight"><pre>def get_one_layer_mlp(hidden, k): # input user = mx.symbol.Variable('user') item = mx.symbol.Variable('item') score = mx.symbol.Variable('score') # user latent features user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k) user = mx.symbol.Activation(data = user, act_type="relu") user = mx.symbol.FullyConnected(data = user, num_hidden = hidden) # item latent features item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k) item = mx.symbol.Activation(data = item, act_type="relu") item = mx.symbol.FullyConnected(data = item, num_hidden = hidden) # predict by the inner product pred = user * item pred = mx.symbol.sum_axis(data = pred, axis = 1) pred = mx.symbol.Flatten(data = pred) # loss layer pred = mx.symbol.LinearRegressionOutput(data = pred, label = score) return pred train(get_one_layer_mlp(64, 64), batch_size=64, num_epoch=10, learning_rate=.05) </pre></div> </div> Adding dropout layers to relief the over-fitting. <div class="highlight-python"><div class="highlight"><pre>def get_one_layer_dropout_mlp(hidden, k): # input user = mx.symbol.Variable('user') item = mx.symbol.Variable('item') score = mx.symbol.Variable('score') # user latent features user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k) user = mx.symbol.Activation(data = user, act_type="relu") user = mx.symbol.FullyConnected(data = user, num_hidden = hidden) user = mx.symbol.Dropout(data=user, p=0.5) # item latent features item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k) item = mx.symbol.Activation(data = item, act_type="relu") item = mx.symbol.FullyConnected(data = item, num_hidden = hidden) item = mx.symbol.Dropout(data=item, p=0.5) # predict by the inner product pred = user * item pred = mx.symbol.sum_axis(data = pred, axis = 1) pred = mx.symbol.Flatten(data = pred) # loss layer pred = mx.symbol.LinearRegressionOutput(data = pred, label = score) return pred train(get_one_layer_mlp(256, 512), batch_size=64, num_epoch=10, learning_rate=.05) </pre></div> </div> <div class="btn-group" role="group"> <div class="download-btn"><a download="matrix_factorization.ipynb" href="matrix_factorization.ipynb"> matrix_factorization.ipynb</a></div></div></div> </div> </div> </div> <div aria-label="main navigation" class="sphinxsidebar rightsidebar" role="navigation"> <div class="sphinxsidebarwrapper"> <h3><a href="../../index.html">Table Of Contents</a></h3> <ul> <li><a class="reference internal" href="#">Matrix Factorization</a><ul> <li><a class="reference internal" href="#prepare-data">Prepare Data</a></li> <li><a class="reference internal" href="#optimization">Optimization</a></li> <li><a class="reference internal" href="#networks">Networks</a></li> </ul> </li> </ul> </div> </div> </div><div class="footer"> <div class="section-disclaimer"> <div class="container"> <div> <img height="60" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/apache_incubator_logo.png"/> Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. "Copyright © 2017-2018, The Apache Software Foundation Apache MXNet, MXNet, Apache, the Apache feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the Apache Software Foundation." </div> </div> </div> </div>  </div> <script crossorigin="anonymous" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script> <script src="../../_static/js/sidebar.js" type="text/javascript"></script> <script src="../../_static/js/search.js" type="text/javascript"></script> <script src="../../_static/js/navbar.js" type="text/javascript"></script> <script src="../../_static/js/clipboard.min.js" type="text/javascript"></script> <script src="../../_static/js/copycode.js" type="text/javascript"></script> <script src="../../_static/js/page.js" type="text/javascript"></script> <script src="../../_static/js/docversion.js" type="text/javascript"></script> <script type="text/javascript"> $('body').ready(function () { $('body').css('visibility', 'visible'); }); </script> </body> </html>

versions/0.11.0/tutorials/python/matrix_factorization.html (450 lines of code) (raw):