course/videos/pipeline_function.ipynb (379 lines of code) (raw):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook regroups the code sample of the video below, which is a part of the [Hugging Face course](https://huggingface.co/course)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form"
},
"outputs": [
{
"data": {
"text/html": [
"<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/tiZFewofSLM?rel=0&controls=0&showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title\n",
"from IPython.display import HTML\n",
"\n",
"HTML('<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/tiZFewofSLM?rel=0&controls=0&showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Install the Transformers and Datasets libraries to run this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install datasets transformers[sentencepiece]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'label': 'POSITIVE', 'score': 0.9598047137260437}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"classifier = pipeline(\"sentiment-analysis\")\n",
"classifier(\"I've been waiting for a HuggingFace course my whole life.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'label': 'POSITIVE', 'score': 0.9598047137260437},\n",
" {'label': 'NEGATIVE', 'score': 0.9994558095932007}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"classifier = pipeline(\"sentiment-analysis\")\n",
"classifier([\n",
" \"I've been waiting for a HuggingFace course my whole life.\", \n",
" \"I hate this so much!\"\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'sequence': 'This is a course about the Transformers library',\n",
" 'labels': ['education', 'business', 'politics'],\n",
" 'scores': [0.8445963859558105, 0.1119762435555458, 0.043427448719739914]}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"classifier = pipeline(\"zero-shot-classification\")\n",
"classifier(\n",
" \"This is a course about the Transformers library\",\n",
" candidate_labels=[\"education\", \"politics\", \"business\"],\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'In this course, we will teach you how to apply a real-time flow strategy of a distributed system called \"n-to-5th dimension\" in real-world situations. The techniques we will demonstrate are as follows:\\n\\nCreate your'}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"generator = pipeline(\"text-generation\")\n",
"generator(\"In this course, we will teach you how to\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'In this course, we will teach you how to design your software and help you get the most out of it.\\n\\nStep 10: Start coding'},\n",
" {'generated_text': \"In this course, we will teach you how to handle errors in the program. From scratch, you'll learn the basic operations that can be performed as\"}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"generator = pipeline(\"text-generation\")\n",
"generator(\n",
" \"In this course, we will teach you how to\",\n",
" max_length=30,\n",
" num_return_sequences=2,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'sequence': 'This course will teach you all about mathematical models.',\n",
" 'score': 0.19619831442832947,\n",
" 'token': 30412,\n",
" 'token_str': ' mathematical'},\n",
" {'sequence': 'This course will teach you all about computational models.',\n",
" 'score': 0.04052725434303284,\n",
" 'token': 38163,\n",
" 'token_str': ' computational'}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"unmasker = pipeline(\"fill-mask\")\n",
"unmasker(\"This course will teach you all about <mask> models.\", top_k=2)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/sgugger/git/transformers/src/transformers/pipelines/token_classification.py:155: UserWarning: `grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy=\"AggregationStrategy.SIMPLE\"` instead.\n",
" f'`grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy=\"{aggregation_strategy}\"` instead.'\n"
]
},
{
"data": {
"text/plain": [
"[{'entity_group': 'PER',\n",
" 'score': 0.9981694,\n",
" 'word': 'Sylvain',\n",
" 'start': 11,\n",
" 'end': 18},\n",
" {'entity_group': 'ORG',\n",
" 'score': 0.97960204,\n",
" 'word': 'Hugging Face',\n",
" 'start': 33,\n",
" 'end': 45},\n",
" {'entity_group': 'LOC',\n",
" 'score': 0.99321055,\n",
" 'word': 'Brooklyn',\n",
" 'start': 49,\n",
" 'end': 57}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"ner = pipeline(\"ner\", grouped_entities=True)\n",
"ner(\"My name is Sylvain and I work at Hugging Face in Brooklyn.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'score': 0.6385916471481323, 'start': 33, 'end': 45, 'answer': 'Hugging Face'}"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"question_answerer = pipeline(\"question-answering\")\n",
"question_answerer(\n",
" question=\"Where do I work?\",\n",
" context=\"My name is Sylvain and I work at Hugging Face in Brooklyn.\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'summary_text': ' America has changed dramatically during recent years . The number of engineering graduates in the U.S. has declined in traditional engineering disciplines such as mechanical, civil, electrical, chemical, and aeronautical engineering . Rapidly developing economies such as China and India, as well as other industrial countries in Europe and Asia, continue to encourage and advance engineering .'}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"summarizer = pipeline(\"summarization\")\n",
"summarizer(\"\"\"\n",
" America has changed dramatically during recent years. Not only has the number of \n",
" graduates in traditional engineering disciplines such as mechanical, civil, \n",
" electrical, chemical, and aeronautical engineering declined, but in most of \n",
" the premier American universities engineering curricula now concentrate on \n",
" and encourage largely the study of engineering science. As a result, there \n",
" are declining offerings in engineering subjects dealing with infrastructure, \n",
" the environment, and related issues, and greater concentration on high \n",
" technology subjects, largely supporting increasingly complex scientific \n",
" developments. While the latter is important, it should not be at the expense \n",
" of more traditional engineering.\n",
"\n",
" Rapidly developing economies such as China and India, as well as other \n",
" industrial countries in Europe and Asia, continue to encourage and advance \n",
" the teaching of engineering. Both China and India, respectively, graduate \n",
" six and eight times as many traditional engineers as does the United States. \n",
" Other industrial countries at minimum maintain their output, while America \n",
" suffers an increasingly serious decline in the number of engineering graduates \n",
" and a lack of well-educated engineers.\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'translation_text': 'This course is produced by Hugging Face.'}]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"translator = pipeline(\"translation\", model=\"Helsinki-NLP/opus-mt-fr-en\")\n",
"translator(\"Ce cours est produit par Hugging Face.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"name": "The pipeline function",
"provenance": []
}
},
"nbformat": 4,
"nbformat_minor": 4
}