notebooks/managing_ipu_resources.ipynb (445 lines of code) (raw):
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Managing IPU Resources from Notebooks\n",
"\n",
"The execution model of IPUs and notebooks means that as you experiment with different models\n",
"you might keep hardware locked in an idle state, preventing other users from using it. It may also happen that \n",
"your experiments fail because you have insufficient hardware.\n",
"Releasing hardware is particularly important in notebooks as the long life time of the\n",
"underlying `ipython` kernel can keep a lock on IPUs long after you are done interacting\n",
"with the hardware.\n",
"\n",
"The Graphcore frameworks operate a computational architecture of 1 model = 1 IPU device;\n",
"this means that each model will attach to specific IPUs and will only release them when\n",
"that model goes out of scope or when resources are explicitly released.\n",
"\n",
"In this notebook you will learn how to:\n",
"\n",
"- Monitor how many IPUs your notebook is currently using\n",
"- Release IPUs by detaching a model\n",
"- Reattach a model to IPUs, to continue using a model after a period of inactivity.\n",
"\n",
"For more information on the basics of IPU computational architecture refer to \n",
"the [IPU Programmer's Guide](https://docs.graphcore.ai/projects/ipu-programmers-guide/en/latest/ipu_introduction.html)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Environment Setup\n",
"\n",
"The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n",
"\n",
"To run the demo using other IPU hardware, you need to have the Poplar SDK enabled with PopTorch installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK and install PopTorch. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to improve usability and support for future users, Graphcore would like to collect information about the\n",
"applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n",
"\n",
"- User progression through the notebook\n",
"- Notebook details: number of cells, code being run and the output of the cells\n",
"- Environment details\n",
"\n",
"You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dependencies and configuration\n",
"\n",
"Install the dependencies for this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install \"optimum-graphcore==0.7\"\n",
"%pip install graphcore-cloud-tools[logger]@git+https://github.com/graphcore/graphcore-cloud-tools\n",
"%load_ext graphcore_cloud_tools.notebook_logging.gc_logger"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Monitoring resources\n",
"\n",
"Grapchore provides the `gc-monitor` utility for inspecting the number of available IPUs and their usage:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"In a notebook, we can run this Bash command using `!` in a regular code cell. It provides detailed information on the IPUs that exist in the current partition.\n",
"The first section of the output is the `card-info`, which is generic information about the IP addresses and serial numbers of all the cards visible to the process.\n",
"The second section of the output indicates usage information of the IPU and shows the user, host and PIDs which are attached to the different IPUs.\n",
"\n",
"When monitoring IPUs, it can be useful to run `gc-monitor` without displaying the static IPU information:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we can write a command that will monitor only the IPUs which are attached from this specific notebook. We do that by only displaying the IPUs attached to a specific PID:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info | grep ${os.getpid()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Since we've not attached to any IPUs yet, there is no output.\n",
"\n",
"Beyond `gc-monitor`, Graphcore also provides a library for monitoring usage called `gcipuinfo` which can be used in Python. This library is not covered in this tutorial but [examples are available in the `gcipuinfo` documentation](https://docs.graphcore.ai/projects/gcipuinfo/en/latest/examples.html)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Creating models\n",
"\n",
"Now let's create some models and attach them to IPUs. The simplest way to create a small model is using the inference `pipeline` provided by the `optimum-graphcore` library."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from optimum.graphcore import pipelines\n",
"sentiment_pipeline = pipelines.pipeline(\"sentiment-analysis\")\n",
"sentiment_pipeline([\"IPUs are great!\", \"Notebooks are easy to program in\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's check how many IPUs are in use:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info | grep ${os.getpid()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"These IPUs will be associated with the model in the pipeline until:\n",
"\n",
"- The `sentiment_pipeline` object goes out of scope or\n",
"- The model is explicitly detached from the IPU.\n",
"\n",
"By remaining attached to the IPUs, the model can be very fast, providing fast responses to new prompts:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%timeit\n",
"sentiment_pipeline([\"IPUs are fast once the pipeline is attached\", \"and Notebooks are easy to program in\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"If you are testing different models you might have multiple pipelines using IPUs:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sentiment_pipeline_2 = pipelines.pipeline(\"text-classification\")\n",
"sentiment_pipeline_2([\"IPUs are great!\", \"Notebooks are easy to program in\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Checking the IPU usage we can see that we are now using four IPUs:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info | grep ${os.getpid()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Managing resources\n",
"\n",
"From this we see that we are using four IPUs, two per active pipeline. While it may make sense for us to keep both pipelines active if we are testing both at the same time, we may need to free up resources to continue experimenting with more models.\n",
"\n",
"To do that we can call the `detachFromDevice` method on the model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sentiment_pipeline.model.detachFromDevice()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info | grep ${os.getpid()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This method has freed up the IPU resources while keeping the pipeline object available, meaning that we can quickly reattach the same pipeline to an IPU simply by calling it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"simple_test_data=[\"I love you.\", \"I hate you!\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"sentiment_pipeline(simple_test_data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info | grep ${os.getpid()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The first call is slow as the model is loaded onto the accelerator, but subsequent calls will be fast:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"sentiment_pipeline(simple_test_data)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The other way to release resources is to let the `sentiment_pipeline` Python variable go out of scope.\n",
"There are two main ways to do that:\n",
"\n",
"1. if you want to use the resources for another pipeline you can assign another variable to the same name:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sentiment_pipeline = sentiment_pipeline_2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info | grep ${os.getpid()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"2. Explicitly use `del` to delete the variables:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Note that after the assignment sentiment_pipeline and sentiment_pipeline_2\n",
"# refer to the same object so both symbols must be deleted to release the resources\n",
"del sentiment_pipeline\n",
"del sentiment_pipeline_2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!gc-monitor --no-card-info | grep ${os.getpid()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"As expected, no IPUs are used by the process anymore.\n",
"\n",
"Alternatively, all IPUs will be released when the notebook kernel is restarted. This can be done from the Notebook graphical user interface by clicking on `Kernel > Restart`:\n",
"\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"In this simple tutorial we saw how to manage IPU resources from a notebook to make sure that we do not try to use more IPUs than are available on a single system.\n",
"\n",
"Refer to the guide [Using IPUs from Jupyter Notebooks](https://github.com/graphcore/examples/tree/master/tutorials//tutorials/standard_tools/using_jupyter) for more information on using IPUs and the Poplar SDK through Jupyter notebooks."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.10 ('3.0.0+1145_poptorch')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "46bde714a99d715eba7e507975e678b0968e7177d805932276a51e552e29fed0"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}