courses/machine_learning/deepdive2/text_classification/labs/word2vec.ipynb (5,195 lines of code) (raw):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"# Word2Vec\n",
"\n",
"**Learning Objectives**\n",
"\n",
"1. Compile all steps into one function\n",
"2. Prepare training data for Word2Vec\n",
"3. Model and Training\n",
"4. Embedding lookup and analysis\n",
"\n",
"\n",
"\n",
"\n",
"## Introduction \n",
"Word2Vec is not a singular algorithm, rather, it is a family of model architectures and optimizations that can be used to learn word embeddings from large datasets. Embeddings learned through Word2Vec have proven to be successful on a variety of downstream natural language processing tasks.\n",
"\n",
"Note: This notebook is based on [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf) and\n",
"[Distributed\n",
"Representations of Words and Phrases and their Compositionality](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf). It is not an exact implementation of the papers. Rather, it is intended to illustrate the key ideas.\n",
"\n",
"These papers proposed two methods for learning representations of words: \n",
"\n",
"* **Continuous Bag-of-Words Model** which predicts the middle word based on surrounding context words. The context consists of a few words before and after the current (middle) word. This architecture is called a bag-of-words model as the order of words in the context is not important.\n",
"* **Continuous Skip-gram Model** which predict words within a certain range before and after the current word in the same sentence. A worked example of this is given below.\n",
"\n",
"\n",
"You'll use the skip-gram approach in this notebook. First, you'll explore skip-grams and other concepts using a single sentence for illustration. Next, you'll train your own Word2Vec model on a small dataset. This notebook also contains code to export the trained embeddings and visualize them in the [TensorFlow Embedding Projector](http://projector.tensorflow.org/).\n",
"\n",
"\n",
" Each learning objective will correspond to a _#TODO_ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/word2vec.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xP00WlaMWBZC"
},
"source": [
"## Skip-gram and Negative Sampling "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Zr2wjv0bW236"
},
"source": [
"While a bag-of-words model predicts a word given the neighboring context, a skip-gram model predicts the context (or neighbors) of a word, given the word itself. The model is trained on skip-grams, which are n-grams that allow tokens to be skipped (see the diagram below for an example). The context of a word can be represented through a set of skip-gram pairs of `(target_word, context_word)` where `context_word` appears in the neighboring context of `target_word`. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ICjc-McbaVTd"
},
"source": [
"Consider the following sentence of 8 words.\n",
"> The wide road shimmered in the hot sun. \n",
"\n",
"The context words for each of the 8 words of this sentence are defined by a window size. The window size determines the span of words on either side of a `target_word` that can be considered `context word`. Take a look at this table of skip-grams for target words based on different window sizes."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YKE87IKT_YT8"
},
"source": [
"Note: For this tutorial, a window size of *n* implies n words on each side with a total window span of 2*n+1 words across a word."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RsCwQ07E8mqU"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gK1gN1jwkMpU"
},
"source": [
"The training objective of the skip-gram model is to maximize the probability of predicting context words given the target word. For a sequence of words *w<sub>1</sub>, w<sub>2</sub>, ... w<sub>T</sub>*, the objective can be written as the average log probability"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pILO_iAc84e-"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Gsy6TUbtnz_K"
},
"source": [
"where `c` is the size of the training context. The basic skip-gram formulation defines this probability using the softmax function."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "P81Qavbb9APd"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "axZvd-hhotVB"
},
"source": [
"where *v* and *v<sup>'<sup>* are target and context vector representations of words and *W* is vocabulary size. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SoLzxbqSpT6_"
},
"source": [
"Computing the denominator of this formulation involves performing a full softmax over the entire vocabulary words which is often large (10<sup>5</sup>-10<sup>7</sup>) terms. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Y5VWYtmFzHkU"
},
"source": [
"The [Noise Contrastive Estimation](https://www.tensorflow.org/api_docs/python/tf/nn/nce_loss) loss function is an efficient approximation for a full softmax. With an objective to learn word embeddings instead of modelling the word distribution, NCE loss can be [simplified](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) to use negative sampling. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WTZBPf1RsOsg"
},
"source": [
"The simplified negative sampling objective for a target word is to distinguish the context word from *num_ns* negative samples drawn from noise distribution *P<sub>n</sub>(w)* of words. More precisely, an efficient approximation of full softmax over the vocabulary is, for a skip-gram pair, to pose the loss for a target word as a classification problem between the context word and *num_ns* negative samples. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Cl0rSfHjt6Mf"
},
"source": [
"A negative sample is defined as a (target_word, context_word) pair such that the context_word does not appear in the `window_size` neighborhood of the target_word. For the example sentence, these are few potential negative samples (when `window_size` is 2).\n",
"\n",
"```\n",
"(hot, shimmered)\n",
"(wide, hot)\n",
"(wide, sun)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kq0q2uqbucFg"
},
"source": [
"In the next section, you'll generate skip-grams and negative samples for a single sentence. You'll also learn about subsampling techniques and train a classification model for positive and negative training examples later in the tutorial."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mk4-Hpe1CH16"
},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use the chown command to change the ownership of repository to user.\n",
"!sudo chown -R jupyter:jupyter /home/jupyter/training-data-analyst"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:17.637454Z",
"iopub.status.busy": "2021-01-21T03:00:17.633831Z",
"iopub.status.idle": "2021-01-21T03:00:18.983028Z",
"shell.execute_reply": "2021-01-21T03:00:18.982359Z"
},
"id": "OSUalgiCchft"
},
"outputs": [],
"source": [
"!pip install -q tqdm"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:18.988694Z",
"iopub.status.busy": "2021-01-21T03:00:18.987959Z",
"iopub.status.idle": "2021-01-21T03:00:25.234405Z",
"shell.execute_reply": "2021-01-21T03:00:25.234859Z"
},
"id": "RutaI-Tpev3T"
},
"outputs": [],
"source": [
"# You can use any Python source file as a module by executing an import statement in some other Python source file.\n",
"# The import statement combines two operations; it searches for the named module, then it binds the\n",
"# results of that search to a name in the local scope.\n",
"import io\n",
"import itertools\n",
"import numpy as np\n",
"import os\n",
"import re\n",
"import string\n",
"import tensorflow as tf\n",
"import tqdm\n",
"\n",
"from tensorflow.keras import Model, Sequential\n",
"from tensorflow.keras.layers import Activation, Dense, Dot, Embedding, Flatten, GlobalAveragePooling1D, Reshape\n",
"from tensorflow.keras.layers.experimental.preprocessing import TextVectorization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Please check your tensorflow version using the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TensorFlow version: 2.6.0\n"
]
}
],
"source": [
"# Show the currently installed version of TensorFlow\n",
"print(\"TensorFlow version: \",tf.version.VERSION)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:25.239217Z",
"iopub.status.busy": "2021-01-21T03:00:25.238545Z",
"iopub.status.idle": "2021-01-21T03:00:25.240981Z",
"shell.execute_reply": "2021-01-21T03:00:25.240501Z"
},
"id": "XkJ5299Tek6B"
},
"outputs": [],
"source": [
"SEED = 42 \n",
"AUTOTUNE = tf.data.experimental.AUTOTUNE"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RW-g5buCHwh3"
},
"source": [
"### Vectorize an example sentence"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y8TfZIgoQrcP"
},
"source": [
"Consider the following sentence: \n",
"`The wide road shimmered in the hot sun.`\n",
"\n",
"Tokenize the sentence:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:25.245707Z",
"iopub.status.busy": "2021-01-21T03:00:25.245019Z",
"iopub.status.idle": "2021-01-21T03:00:25.249082Z",
"shell.execute_reply": "2021-01-21T03:00:25.249520Z"
},
"id": "bsl7jBzV6_KK"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"8\n"
]
}
],
"source": [
"sentence = \"The wide road shimmered in the hot sun\"\n",
"tokens = list(sentence.lower().split())\n",
"print(len(tokens))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PU-bs1XtThEw"
},
"source": [
"Create a vocabulary to save mappings from tokens to integer indices."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:25.254232Z",
"iopub.status.busy": "2021-01-21T03:00:25.253611Z",
"iopub.status.idle": "2021-01-21T03:00:25.255627Z",
"shell.execute_reply": "2021-01-21T03:00:25.256085Z"
},
"id": "UdYv1HJUQ8XA"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'<pad>': 0, 'the': 1, 'wide': 2, 'road': 3, 'shimmered': 4, 'in': 5, 'hot': 6, 'sun': 7}\n"
]
}
],
"source": [
"vocab, index = {}, 1 # start indexing from 1\n",
"vocab['<pad>'] = 0 # add a padding token \n",
"for token in tokens:\n",
" if token not in vocab: \n",
" vocab[token] = index\n",
" index += 1\n",
"vocab_size = len(vocab)\n",
"print(vocab)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZpuP43Dddasr"
},
"source": [
"Create an inverse vocabulary to save mappings from integer indices to tokens."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:25.260004Z",
"iopub.status.busy": "2021-01-21T03:00:25.259391Z",
"iopub.status.idle": "2021-01-21T03:00:25.261324Z",
"shell.execute_reply": "2021-01-21T03:00:25.261687Z"
},
"id": "o9ULAJYtEvKl"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{0: '<pad>', 1: 'the', 2: 'wide', 3: 'road', 4: 'shimmered', 5: 'in', 6: 'hot', 7: 'sun'}\n"
]
}
],
"source": [
"inverse_vocab = {index: token for token, index in vocab.items()}\n",
"print(inverse_vocab)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n3qtuyxIRyii"
},
"source": [
"Vectorize your sentence.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:25.265064Z",
"iopub.status.busy": "2021-01-21T03:00:25.264505Z",
"iopub.status.idle": "2021-01-21T03:00:25.266446Z",
"shell.execute_reply": "2021-01-21T03:00:25.266783Z"
},
"id": "CsB3-9uQQYyl"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1, 2, 3, 4, 5, 1, 6, 7]\n"
]
}
],
"source": [
"example_sequence = [vocab[word] for word in tokens]\n",
"print(example_sequence)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ox1I28JRIOdM"
},
"source": [
"### Generate skip-grams from one sentence"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "t7NNKAmSiHvy"
},
"source": [
"The `tf.keras.preprocessing.sequence` module provides useful functions that simplify data preparation for Word2Vec. You can use the `tf.keras.preprocessing.sequence.skipgrams` to generate skip-gram pairs from the `example_sequence` with a given `window_size` from tokens in the range `[0, vocab_size)`.\n",
"\n",
"Note: `negative_samples` is set to `0` here as batching negative samples generated by this function requires a bit of code. You will use another function to perform negative sampling in the next section.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:25.271094Z",
"iopub.status.busy": "2021-01-21T03:00:25.270506Z",
"iopub.status.idle": "2021-01-21T03:00:25.272960Z",
"shell.execute_reply": "2021-01-21T03:00:25.272521Z"
},
"id": "USAJxW4RD7pn"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"26\n"
]
}
],
"source": [
"window_size = 2\n",
"positive_skip_grams, _ = tf.keras.preprocessing.sequence.skipgrams(\n",
" example_sequence, \n",
" vocabulary_size=vocab_size,\n",
" window_size=window_size,\n",
" negative_samples=0)\n",
"print(len(positive_skip_grams))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uc9uhiMwY-AQ"
},
"source": [
"Take a look at few positive skip-grams."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:25.277382Z",
"iopub.status.busy": "2021-01-21T03:00:25.276766Z",
"iopub.status.idle": "2021-01-21T03:00:25.279506Z",
"shell.execute_reply": "2021-01-21T03:00:25.279019Z"
},
"id": "SCnqEukIE9pt"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(1, 3): (the, road)\n",
"(4, 1): (shimmered, the)\n",
"(5, 6): (in, hot)\n",
"(4, 2): (shimmered, wide)\n",
"(3, 2): (road, wide)\n"
]
}
],
"source": [
"for target, context in positive_skip_grams[:5]:\n",
" print(f\"({target}, {context}): ({inverse_vocab[target]}, {inverse_vocab[context]})\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_ua9PkMTISF0"
},
"source": [
"### Negative sampling for one skip-gram "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Esqn8WBfZnEK"
},
"source": [
"The `skipgrams` function returns all positive skip-gram pairs by sliding over a given window span. To produce additional skip-gram pairs that would serve as negative samples for training, you need to sample random words from the vocabulary. Use the `tf.random.log_uniform_candidate_sampler` function to sample `num_ns` number of negative samples for a given target word in a window. You can call the funtion on one skip-grams's target word and pass the context word as true class to exclude it from being sampled.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AgH3aSvw3xTD"
},
"source": [
"Key point: *num_ns* (number of negative samples per positive context word) between [5, 20] is [shown to work](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) best for smaller datasets, while *num_ns* between [2,5] suffices for larger datasets. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:26.683966Z",
"iopub.status.busy": "2021-01-21T03:00:26.683244Z",
"iopub.status.idle": "2021-01-21T03:00:26.964417Z",
"shell.execute_reply": "2021-01-21T03:00:26.963972Z"
},
"id": "m_LmdzqIGr5L"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tf.Tensor([2 1 4 3], shape=(4,), dtype=int64)\n",
"['wide', 'the', 'shimmered', 'road']\n"
]
}
],
"source": [
"# Get target and context words for one positive skip-gram.\n",
"target_word, context_word = positive_skip_grams[0]\n",
"\n",
"# Set the number of negative samples per positive context. \n",
"num_ns = 4\n",
"\n",
"context_class = tf.reshape(tf.constant(context_word, dtype=\"int64\"), (1, 1))\n",
"negative_sampling_candidates, _, _ = tf.random.log_uniform_candidate_sampler(\n",
" true_classes=context_class, # class that should be sampled as 'positive'\n",
" num_true=1, # each positive skip-gram has 1 positive context class\n",
" num_sampled=num_ns, # number of negative context words to sample\n",
" unique=True, # all the negative samples should be unique\n",
" range_max=vocab_size, # pick index of the samples from [0, vocab_size]\n",
" seed=SEED, # seed for reproducibility\n",
" name=\"negative_sampling\" # name of this operation\n",
")\n",
"print(negative_sampling_candidates)\n",
"print([inverse_vocab[index.numpy()] for index in negative_sampling_candidates])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8MSxWCrLIalp"
},
"source": [
"### Construct one training example"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q6uEWdj8vKKv"
},
"source": [
"For a given positive `(target_word, context_word)` skip-gram, you now also have `num_ns` negative sampled context words that do not appear in the window size neighborhood of `target_word`. Batch the `1` positive `context_word` and `num_ns` negative context words into one tensor. This produces a set of positive skip-grams (labelled as `1`) and negative samples (labelled as `0`) for each target word."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:26.969368Z",
"iopub.status.busy": "2021-01-21T03:00:26.968744Z",
"iopub.status.idle": "2021-01-21T03:00:26.971488Z",
"shell.execute_reply": "2021-01-21T03:00:26.971917Z"
},
"id": "zSiZwifuLvHf"
},
"outputs": [],
"source": [
"# Add a dimension so you can use concatenation (on the next step).\n",
"negative_sampling_candidates = tf.expand_dims(negative_sampling_candidates, 1)\n",
"\n",
"# Concat positive context word with negative sampled words.\n",
"context = tf.concat([context_class, negative_sampling_candidates], 0)\n",
"\n",
"# Label first context word as 1 (positive) followed by num_ns 0s (negative).\n",
"label = tf.constant([1] + [0]*num_ns, dtype=\"int64\") \n",
"\n",
"# Reshape target to shape (1,) and context and label to (num_ns+1,).\n",
"target = tf.squeeze(target_word)\n",
"context = tf.squeeze(context)\n",
"label = tf.squeeze(label)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OIJeoFCAwtXJ"
},
"source": [
"Take a look at the context and the corresponding labels for the target word from the skip-gram example above. "
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:26.975824Z",
"iopub.status.busy": "2021-01-21T03:00:26.975217Z",
"iopub.status.idle": "2021-01-21T03:00:26.979930Z",
"shell.execute_reply": "2021-01-21T03:00:26.980290Z"
},
"id": "tzyCPCuZwmdL"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"target_index : 1\n",
"target_word : the\n",
"context_indices : [3 2 1 4 3]\n",
"context_words : ['road', 'wide', 'the', 'shimmered', 'road']\n",
"label : [1 0 0 0 0]\n"
]
}
],
"source": [
"print(f\"target_index : {target}\")\n",
"print(f\"target_word : {inverse_vocab[target_word]}\")\n",
"print(f\"context_indices : {context}\")\n",
"print(f\"context_words : {[inverse_vocab[c.numpy()] for c in context]}\")\n",
"print(f\"label : {label}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gBtTcUVQr8EO"
},
"source": [
"A tuple of `(target, context, label)` tensors constitutes one training example for training your skip-gram negative sampling Word2Vec model. Notice that the target is of shape `(1,)` while the context and label are of shape `(1+num_ns,)`"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:26.984327Z",
"iopub.status.busy": "2021-01-21T03:00:26.983717Z",
"iopub.status.idle": "2021-01-21T03:00:26.986322Z",
"shell.execute_reply": "2021-01-21T03:00:26.986728Z"
},
"id": "x-FwkR8jx9-Z"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"target : tf.Tensor(1, shape=(), dtype=int32)\n",
"context : tf.Tensor([3 2 1 4 3], shape=(5,), dtype=int64)\n",
"label : tf.Tensor([1 0 0 0 0], shape=(5,), dtype=int64)\n"
]
}
],
"source": [
"print(f\"target :\", target)\n",
"print(f\"context :\", context )\n",
"print(f\"label :\", label )"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4bRJIlow4Dlv"
},
"source": [
"### Summary"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pWkuha0oykG5"
},
"source": [
"This picture summarizes the procedure of generating training example from a sentence. \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_KlwdiAa9crJ"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9wmdO_MEIpaM"
},
"source": [
"## Lab Task 1: Compile all steps into one function\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iLKwNAczHsKg"
},
"source": [
"### Skip-gram Sampling table "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TUUK3uDtFNFE"
},
"source": [
"A large dataset means larger vocabulary with higher number of more frequent words such as stopwords. Training examples obtained from sampling commonly occuring words (such as `the`, `is`, `on`) don't add much useful information for the model to learn from. [Mikolov et al.](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) suggest subsampling of frequent words as a helpful practice to improve embedding quality. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bPtbv7zNP7Dx"
},
"source": [
"The `tf.keras.preprocessing.sequence.skipgrams` function accepts a sampling table argument to encode probabilities of sampling any token. You can use the `tf.keras.preprocessing.sequence.make_sampling_table` to generate a word-frequency rank based probabilistic sampling table and pass it to `skipgrams` function. Take a look at the sampling probabilities for a `vocab_size` of 10."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:26.991491Z",
"iopub.status.busy": "2021-01-21T03:00:26.990898Z",
"iopub.status.idle": "2021-01-21T03:00:26.993349Z",
"shell.execute_reply": "2021-01-21T03:00:26.992949Z"
},
"id": "Rn9zAnDccyRg"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.00315225 0.00315225 0.00547597 0.00741556 0.00912817 0.01068435\n",
" 0.01212381 0.01347162 0.01474487 0.0159558 ]\n"
]
}
],
"source": [
"sampling_table = tf.keras.preprocessing.sequence.make_sampling_table(size=10)\n",
"print(sampling_table)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EHvSptcPk5fp"
},
"source": [
"`sampling_table[i]` denotes the probability of sampling the i-th most common word in a dataset. The function assumes a [Zipf's distribution](https://en.wikipedia.org/wiki/Zipf%27s_law) of the word frequencies for sampling."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mRHMssMmHgH-"
},
"source": [
"Key point: The `tf.random.log_uniform_candidate_sampler` already assumes that the vocabulary frequency follows a log-uniform (Zipf's) distribution. Using these distribution weighted sampling also helps approximate the Noise Contrastive Estimation (NCE) loss with simpler loss functions for training a negative sampling objective."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aj--8RFK6fgW"
},
"source": [
"### Generate training data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dy5hl4lQ0B2M"
},
"source": [
"Compile all the steps described above into a function that can be called on a list of vectorized sentences obtained from any text dataset. Notice that the sampling table is built before sampling skip-gram word pairs. You will use this function in the later sections."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:27.001289Z",
"iopub.status.busy": "2021-01-21T03:00:27.000700Z",
"iopub.status.idle": "2021-01-21T03:00:27.002547Z",
"shell.execute_reply": "2021-01-21T03:00:27.002130Z"
},
"id": "63INISDEX1Hu"
},
"outputs": [],
"source": [
"# Generates skip-gram pairs with negative sampling for a list of sequences\n",
"# (int-encoded sentences) based on window size, number of negative samples\n",
"# and vocabulary size.\n",
"def generate_training_data(sequences, window_size, num_ns, vocab_size, seed):\n",
" # Elements of each training example are appended to these lists.\n",
" targets, contexts, labels = [], [], []\n",
"\n",
" # Build the sampling table for vocab_size tokens.\n",
"\n",
" # TODO 1a -- your code goes here\n",
"\n",
"\n",
" # Iterate over all sequences (sentences) in dataset.\n",
" for sequence in tqdm.tqdm(sequences):\n",
"\n",
" # Generate positive skip-gram pairs for a sequence (sentence).\n",
" positive_skip_grams, _ = tf.keras.preprocessing.sequence.skipgrams(\n",
" sequence, \n",
" vocabulary_size=vocab_size,\n",
" sampling_table=sampling_table,\n",
" window_size=window_size,\n",
" negative_samples=0)\n",
" \n",
" # Iterate over each positive skip-gram pair to produce training examples \n",
" # with positive context word and negative samples.\n",
"\n",
" # TODO 1b -- your code goes here\n",
" \n",
" \n",
" # Build context and label vectors (for one target word)\n",
" negative_sampling_candidates = tf.expand_dims(\n",
" negative_sampling_candidates, 1)\n",
"\n",
" context = tf.concat([context_class, negative_sampling_candidates], 0)\n",
" label = tf.constant([1] + [0]*num_ns, dtype=\"int64\")\n",
"\n",
" # Append each element from the training example to global lists.\n",
" targets.append(target_word)\n",
" contexts.append(context)\n",
" labels.append(label)\n",
"\n",
" return targets, contexts, labels"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "shvPC8Ji2cMK"
},
"source": [
"## Lab Task 2: Prepare training data for Word2Vec"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j5mbZsZu6uKg"
},
"source": [
"With an understanding of how to work with one sentence for a skip-gram negative sampling based Word2Vec model, you can proceed to generate training examples from a larger list of sentences!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OFlikI6L26nh"
},
"source": [
"### Download text corpus\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rEFavOgN98al"
},
"source": [
"You will use a text file of Shakespeare's writing for this tutorial. Change the following line to run this code on your own data."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:27.007710Z",
"iopub.status.busy": "2021-01-21T03:00:27.007122Z",
"iopub.status.idle": "2021-01-21T03:00:27.209026Z",
"shell.execute_reply": "2021-01-21T03:00:27.208344Z"
},
"id": "QFkitxzVVaAi"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" 8192/1115394 [..............................] - ETA: 0s"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"1122304/1115394 [==============================] - 0s 0us/step\n"
]
}
],
"source": [
"path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sOsbLq8a37dr"
},
"source": [
"Read text from the file and take a look at the first few lines. "
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:27.213900Z",
"iopub.status.busy": "2021-01-21T03:00:27.213203Z",
"iopub.status.idle": "2021-01-21T03:00:27.223892Z",
"shell.execute_reply": "2021-01-21T03:00:27.223373Z"
},
"id": "lfgnsUw3ofMD"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"First Citizen:\n",
"Before we proceed any further, hear me speak.\n",
"\n",
"All:\n",
"Speak, speak.\n",
"\n",
"First Citizen:\n",
"You are all resolved rather to die than to famish?\n",
"\n",
"All:\n",
"Resolved. resolved.\n",
"\n",
"First Citizen:\n",
"First, you know Caius Marcius is chief enemy to the people.\n",
"\n",
"All:\n",
"We know't, we know't.\n",
"\n",
"First Citizen:\n",
"Let us kill him, and we'll have corn at our own price.\n"
]
}
],
"source": [
"with open(path_to_file) as f: \n",
" lines = f.read().splitlines()\n",
"for line in lines[:20]:\n",
" print(line)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gTNZYqUs5C2V"
},
"source": [
"Use the non empty lines to construct a `tf.data.TextLineDataset` object for next steps."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:27.229363Z",
"iopub.status.busy": "2021-01-21T03:00:27.228703Z",
"iopub.status.idle": "2021-01-21T03:00:27.275948Z",
"shell.execute_reply": "2021-01-21T03:00:27.275227Z"
},
"id": "ViDrwy-HjAs9"
},
"outputs": [],
"source": [
"# TODO 2a -- your code goes here"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vfsc88zE9upk"
},
"source": [
"### Vectorize sentences from the corpus"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XfgZo8zR94KK"
},
"source": [
"You can use the `TextVectorization` layer to vectorize sentences from the corpus. Learn more about using this layer in this [Text Classification](https://www.tensorflow.org/tutorials/keras/text_classification) tutorial. Notice from the first few sentences above that the text needs to be in one case and punctuation needs to be removed. To do this, define a `custom_standardization function` that can be used in the TextVectorization layer."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:27.286198Z",
"iopub.status.busy": "2021-01-21T03:00:27.280936Z",
"iopub.status.idle": "2021-01-21T03:00:27.308776Z",
"shell.execute_reply": "2021-01-21T03:00:27.309189Z"
},
"id": "2MlsXzo-ZlfK"
},
"outputs": [],
"source": [
"# We create a custom standardization function to lowercase the text and \n",
"# remove punctuation.\n",
"def custom_standardization(input_data):\n",
" lowercase = tf.strings.lower(input_data)\n",
" return tf.strings.regex_replace(lowercase,\n",
" '[%s]' % re.escape(string.punctuation), '')\n",
"\n",
"# Define the vocabulary size and number of words in a sequence.\n",
"vocab_size = 4096\n",
"sequence_length = 10\n",
"\n",
"# Use the text vectorization layer to normalize, split, and map strings to\n",
"# integers. Set output_sequence_length length to pad all samples to same length.\n",
"vectorize_layer = TextVectorization(\n",
" standardize=custom_standardization,\n",
" max_tokens=vocab_size,\n",
" output_mode='int',\n",
" output_sequence_length=sequence_length)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "g92LuvnyBmz1"
},
"source": [
"Call `adapt` on the text dataset to create vocabulary.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:27.313372Z",
"iopub.status.busy": "2021-01-21T03:00:27.312717Z",
"iopub.status.idle": "2021-01-21T03:00:28.558323Z",
"shell.execute_reply": "2021-01-21T03:00:28.557711Z"
},
"id": "seZau_iYMPFT"
},
"outputs": [],
"source": [
"vectorize_layer.adapt(text_ds.batch(1024))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jg2z7eeHMnH-"
},
"source": [
"Once the state of the layer has been adapted to represent the text corpus, the vocabulary can be accessed with `get_vocabulary()`. This function returns a list of all vocabulary tokens sorted (descending) by their frequency. "
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:28.562873Z",
"iopub.status.busy": "2021-01-21T03:00:28.562273Z",
"iopub.status.idle": "2021-01-21T03:00:28.569259Z",
"shell.execute_reply": "2021-01-21T03:00:28.569653Z"
},
"id": "jgw9pTA7MRaU"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['', '[UNK]', 'the', 'and', 'to', 'i', 'of', 'you', 'my', 'a', 'that', 'in', 'is', 'not', 'for', 'with', 'me', 'it', 'be', 'your']\n"
]
}
],
"source": [
"# Save the created vocabulary for reference.\n",
"inverse_vocab = vectorize_layer.get_vocabulary()\n",
"print(inverse_vocab[:20])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DOQ30Tx6KA2G"
},
"source": [
"The vectorize_layer can now be used to generate vectors for each element in the `text_ds`."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:28.574518Z",
"iopub.status.busy": "2021-01-21T03:00:28.573933Z",
"iopub.status.idle": "2021-01-21T03:00:28.621475Z",
"shell.execute_reply": "2021-01-21T03:00:28.620922Z"
},
"id": "yUVYrDp0araQ"
},
"outputs": [],
"source": [
"def vectorize_text(text):\n",
" text = tf.expand_dims(text, -1)\n",
" return tf.squeeze(vectorize_layer(text))\n",
"\n",
"# Vectorize the data in text_ds.\n",
"text_vector_ds = text_ds.batch(1024).prefetch(AUTOTUNE).map(vectorize_layer).unbatch()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7YyH_SYzB72p"
},
"source": [
"### Obtain sequences from the dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NFUQLX0_KaRC"
},
"source": [
"You now have a `tf.data.Dataset` of integer encoded sentences. To prepare the dataset for training a Word2Vec model, flatten the dataset into a list of sentence vector sequences. This step is required as you would iterate over each sentence in the dataset to produce positive and negative examples. \n",
"\n",
"Note: Since the `generate_training_data()` defined earlier uses non-TF python/numpy functions, you could also use a `tf.py_function` or `tf.numpy_function` with `tf.data.Dataset.map()`."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:28.627815Z",
"iopub.status.busy": "2021-01-21T03:00:28.627206Z",
"iopub.status.idle": "2021-01-21T03:00:30.982078Z",
"shell.execute_reply": "2021-01-21T03:00:30.981446Z"
},
"id": "sGXoOh9y11pM"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"32777\n"
]
}
],
"source": [
"sequences = list(text_vector_ds.as_numpy_iterator())\n",
"print(len(sequences))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tDc4riukLTqg"
},
"source": [
"Take a look at few examples from `sequences`.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:30.987147Z",
"iopub.status.busy": "2021-01-21T03:00:30.986485Z",
"iopub.status.idle": "2021-01-21T03:00:30.989065Z",
"shell.execute_reply": "2021-01-21T03:00:30.988596Z"
},
"id": "WZf1RIbB2Dfb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ 89 270 0 0 0 0 0 0 0 0] => ['first', 'citizen', '', '', '', '', '', '', '', '']\n",
"[138 36 982 144 673 125 16 106 0 0] => ['before', 'we', 'proceed', 'any', 'further', 'hear', 'me', 'speak', '', '']\n",
"[34 0 0 0 0 0 0 0 0 0] => ['all', '', '', '', '', '', '', '', '', '']\n",
"[106 106 0 0 0 0 0 0 0 0] => ['speak', 'speak', '', '', '', '', '', '', '', '']\n",
"[ 89 270 0 0 0 0 0 0 0 0] => ['first', 'citizen', '', '', '', '', '', '', '', '']\n"
]
}
],
"source": [
"for seq in sequences[:5]:\n",
" print(f\"{seq} => {[inverse_vocab[i] for i in seq]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yDzSOjNwCWNh"
},
"source": [
"### Generate training examples from sequences"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BehvYr-nEKyY"
},
"source": [
"`sequences` is now a list of int encoded sentences. Just call the `generate_training_data()` function defined earlier to generate training examples for the Word2Vec model. To recap, the function iterates over each word from each sequence to collect positive and negative context words. Length of target, contexts and labels should be same, representing the total number of training examples."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:30.993704Z",
"iopub.status.busy": "2021-01-21T03:00:30.993116Z",
"iopub.status.idle": "2021-01-21T03:00:58.391465Z",
"shell.execute_reply": "2021-01-21T03:00:58.391946Z"
},
"id": "44DJ22M6nX5o"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 0%| | 0/32777 [00:00<?, ?it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 0%| | 130/32777 [00:00<00:25, 1290.67it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 1%| | 260/32777 [00:00<00:26, 1204.74it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 1%| | 403/32777 [00:00<00:24, 1302.79it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 2%|▏ | 534/32777 [00:00<00:26, 1220.29it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 2%|▏ | 704/32777 [00:00<00:23, 1374.92it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 3%|▎ | 843/32777 [00:00<00:24, 1314.59it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 3%|▎ | 985/32777 [00:00<00:23, 1343.36it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 3%|▎ | 1141/32777 [00:00<00:22, 1400.11it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 4%|▍ | 1282/32777 [00:00<00:22, 1395.62it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 4%|▍ | 1425/32777 [00:01<00:22, 1398.25it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 5%|▍ | 1566/32777 [00:01<00:25, 1227.86it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 5%|▌ | 1693/32777 [00:01<00:26, 1161.30it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 6%|▌ | 1822/32777 [00:01<00:25, 1193.64it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 6%|▌ | 1944/32777 [00:01<00:25, 1189.19it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 7%|▋ | 2139/32777 [00:01<00:21, 1396.93it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 7%|▋ | 2313/32777 [00:01<00:20, 1489.45it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 8%|▊ | 2537/32777 [00:01<00:17, 1705.36it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 8%|▊ | 2710/32777 [00:01<00:17, 1687.36it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 9%|▉ | 2881/32777 [00:02<00:18, 1657.44it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 9%|▉ | 3048/32777 [00:02<00:18, 1642.90it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 10%|▉ | 3214/32777 [00:02<00:18, 1618.60it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 10%|█ | 3377/32777 [00:02<00:19, 1539.00it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 11%|█ | 3532/32777 [00:02<00:19, 1501.18it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 11%|█ | 3683/32777 [00:02<00:20, 1419.09it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 12%|█▏ | 3880/32777 [00:02<00:18, 1563.15it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 12%|█▏ | 4039/32777 [00:02<00:18, 1554.95it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 13%|█▎ | 4196/32777 [00:02<00:18, 1529.13it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 13%|█▎ | 4350/32777 [00:03<00:20, 1359.67it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 14%|█▎ | 4490/32777 [00:03<00:22, 1245.60it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 14%|█▍ | 4631/32777 [00:03<00:21, 1286.33it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 15%|█▍ | 4794/32777 [00:03<00:20, 1373.31it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 15%|█▌ | 4935/32777 [00:03<00:21, 1293.66it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 15%|█▌ | 5068/32777 [00:03<00:21, 1261.17it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 16%|█▌ | 5209/32777 [00:03<00:21, 1300.66it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 16%|█▋ | 5372/32777 [00:03<00:19, 1391.87it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 17%|█▋ | 5514/32777 [00:03<00:19, 1395.77it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 17%|█▋ | 5655/32777 [00:04<00:21, 1289.41it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 18%|█▊ | 5787/32777 [00:04<00:22, 1187.64it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 18%|█▊ | 5913/32777 [00:04<00:22, 1206.02it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 18%|█▊ | 6042/32777 [00:04<00:21, 1224.97it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 19%|█▉ | 6173/32777 [00:04<00:21, 1247.39it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 19%|█▉ | 6303/32777 [00:04<00:21, 1258.49it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 20%|█▉ | 6444/32777 [00:04<00:20, 1296.15it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 20%|██ | 6575/32777 [00:04<00:20, 1252.69it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 20%|██ | 6703/32777 [00:04<00:20, 1259.99it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 21%|██ | 6830/32777 [00:05<00:21, 1231.77it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 21%|██ | 6954/32777 [00:05<00:21, 1175.20it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 22%|██▏ | 7073/32777 [00:05<00:22, 1131.32it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 22%|██▏ | 7206/32777 [00:05<00:21, 1183.55it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 22%|██▏ | 7339/32777 [00:05<00:20, 1223.83it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 23%|██▎ | 7463/32777 [00:05<00:21, 1177.79it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 23%|██▎ | 7582/32777 [00:05<00:21, 1176.36it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 24%|██▎ | 7742/32777 [00:05<00:19, 1290.61it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 24%|██▍ | 7872/32777 [00:05<00:19, 1274.76it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 24%|██▍ | 8000/32777 [00:06<00:20, 1196.45it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 25%|██▍ | 8121/32777 [00:06<00:26, 919.60it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 25%|██▌ | 8223/32777 [00:06<00:26, 936.05it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 25%|██▌ | 8327/32777 [00:06<00:25, 960.95it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 26%|██▌ | 8429/32777 [00:06<00:26, 914.45it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 26%|██▌ | 8573/32777 [00:06<00:23, 1050.69it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 26%|██▋ | 8684/32777 [00:06<00:22, 1048.02it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 27%|██▋ | 8818/32777 [00:06<00:21, 1125.35it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 27%|██▋ | 8988/32777 [00:06<00:18, 1285.96it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 28%|██▊ | 9120/32777 [00:07<00:18, 1250.11it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 28%|██▊ | 9248/32777 [00:07<00:20, 1160.04it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 29%|██▊ | 9376/32777 [00:07<00:19, 1183.71it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 29%|██▉ | 9498/32777 [00:07<00:19, 1190.33it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 29%|██▉ | 9621/32777 [00:07<00:19, 1193.34it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 30%|██▉ | 9742/32777 [00:07<00:19, 1177.71it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 30%|███ | 9861/32777 [00:07<00:19, 1166.33it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 30%|███ | 9979/32777 [00:07<00:21, 1075.96it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 31%|███ | 10089/32777 [00:07<00:21, 1037.96it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 31%|███ | 10198/32777 [00:08<00:21, 1048.57it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 31%|███▏ | 10304/32777 [00:08<00:23, 943.35it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 32%|███▏ | 10401/32777 [00:08<00:23, 947.13it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 32%|███▏ | 10509/32777 [00:08<00:22, 981.23it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 32%|███▏ | 10611/32777 [00:08<00:22, 987.06it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 33%|███▎ | 10739/32777 [00:08<00:20, 1065.95it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 33%|███▎ | 10860/32777 [00:08<00:19, 1103.49it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 34%|███▎ | 11000/32777 [00:08<00:18, 1183.49it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 34%|███▍ | 11123/32777 [00:08<00:18, 1189.01it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 34%|███▍ | 11243/32777 [00:09<00:19, 1112.40it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 35%|███▍ | 11356/32777 [00:09<00:20, 1065.67it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 35%|███▍ | 11464/32777 [00:09<00:20, 1023.70it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 35%|███▌ | 11576/32777 [00:09<00:20, 1050.07it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 36%|███▌ | 11685/32777 [00:09<00:19, 1060.09it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 36%|███▌ | 11792/32777 [00:09<00:20, 1040.53it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 36%|███▋ | 11897/32777 [00:09<00:22, 927.83it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 37%|███▋ | 11993/32777 [00:09<00:23, 899.47it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 37%|███▋ | 12087/32777 [00:09<00:22, 908.80it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 37%|███▋ | 12180/32777 [00:10<00:22, 907.81it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 37%|███▋ | 12285/32777 [00:10<00:21, 942.30it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 38%|███▊ | 12415/32777 [00:10<00:19, 1040.59it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 38%|███▊ | 12534/32777 [00:10<00:18, 1082.86it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 39%|███▊ | 12658/32777 [00:10<00:17, 1126.25it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 39%|███▉ | 12772/32777 [00:10<00:17, 1114.55it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 39%|███▉ | 12884/32777 [00:10<00:18, 1102.98it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 40%|███▉ | 13030/32777 [00:10<00:16, 1207.19it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 40%|████ | 13172/32777 [00:10<00:15, 1260.32it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 41%|████ | 13299/32777 [00:11<00:17, 1126.86it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 41%|████ | 13415/32777 [00:11<00:17, 1129.64it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 41%|████▏ | 13532/32777 [00:11<00:16, 1137.15it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 42%|████▏ | 13660/32777 [00:11<00:16, 1168.83it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 42%|████▏ | 13806/32777 [00:11<00:15, 1251.14it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 43%|████▎ | 13933/32777 [00:11<00:15, 1205.02it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 43%|████▎ | 14055/32777 [00:11<00:16, 1151.19it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 43%|████▎ | 14189/32777 [00:11<00:15, 1198.47it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 44%|████▎ | 14310/32777 [00:11<00:16, 1148.91it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 44%|████▍ | 14463/32777 [00:11<00:14, 1250.78it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 45%|████▍ | 14590/32777 [00:12<00:15, 1170.78it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 45%|████▍ | 14709/32777 [00:12<00:15, 1157.86it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 45%|████▌ | 14848/32777 [00:12<00:14, 1214.00it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 46%|████▌ | 14971/32777 [00:12<00:15, 1172.55it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 46%|████▌ | 15090/32777 [00:12<00:16, 1104.12it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 46%|████▋ | 15211/32777 [00:12<00:15, 1130.11it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 47%|████▋ | 15329/32777 [00:12<00:15, 1140.80it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 47%|████▋ | 15460/32777 [00:12<00:14, 1184.63it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 48%|████▊ | 15600/32777 [00:12<00:13, 1241.88it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 48%|████▊ | 15762/32777 [00:13<00:12, 1345.56it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 49%|████▊ | 15898/32777 [00:13<00:13, 1253.70it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 49%|████▉ | 16035/32777 [00:13<00:13, 1284.49it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 49%|████▉ | 16165/32777 [00:13<00:13, 1227.88it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 50%|████▉ | 16290/32777 [00:13<00:13, 1225.49it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 50%|█████ | 16414/32777 [00:13<00:14, 1144.95it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 50%|█████ | 16547/32777 [00:13<00:13, 1192.57it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 51%|█████ | 16673/32777 [00:13<00:13, 1206.39it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 51%|█████ | 16795/32777 [00:13<00:13, 1147.37it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 52%|█████▏ | 16940/32777 [00:14<00:12, 1231.27it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 52%|█████▏ | 17097/32777 [00:14<00:11, 1326.58it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 53%|█████▎ | 17232/32777 [00:14<00:11, 1319.29it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 53%|█████▎ | 17365/32777 [00:14<00:12, 1247.83it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 53%|█████▎ | 17492/32777 [00:14<00:12, 1183.33it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 54%|█████▎ | 17612/32777 [00:14<00:12, 1167.53it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 54%|█████▍ | 17730/32777 [00:14<00:14, 1068.22it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 55%|█████▍ | 17864/32777 [00:14<00:13, 1133.02it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 55%|█████▍ | 17980/32777 [00:14<00:13, 1113.10it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 55%|█████▌ | 18093/32777 [00:15<00:14, 1016.62it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 56%|█████▌ | 18197/32777 [00:15<00:15, 927.30it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 56%|█████▌ | 18293/32777 [00:15<00:15, 916.76it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 56%|█████▌ | 18396/32777 [00:15<00:15, 944.96it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 56%|█████▋ | 18502/32777 [00:15<00:14, 976.30it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 57%|█████▋ | 18643/32777 [00:15<00:12, 1097.76it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 57%|█████▋ | 18755/32777 [00:15<00:13, 1066.93it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 58%|█████▊ | 18864/32777 [00:15<00:13, 1047.34it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 58%|█████▊ | 18998/32777 [00:15<00:12, 1123.54it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 58%|█████▊ | 19120/32777 [00:16<00:11, 1146.14it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 59%|█████▊ | 19239/32777 [00:16<00:11, 1155.74it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 59%|█████▉ | 19388/32777 [00:16<00:10, 1239.93it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 60%|█████▉ | 19513/32777 [00:16<00:12, 1074.10it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 60%|█████▉ | 19625/32777 [00:16<00:13, 967.20it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 60%|██████ | 19735/32777 [00:16<00:13, 999.28it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 61%|██████ | 19839/32777 [00:16<00:13, 960.15it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 61%|██████ | 19977/32777 [00:16<00:12, 1063.78it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 61%|██████▏ | 20087/32777 [00:16<00:12, 998.98it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 62%|██████▏ | 20190/32777 [00:17<00:14, 879.51it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 62%|██████▏ | 20310/32777 [00:17<00:13, 957.27it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 62%|██████▏ | 20410/32777 [00:17<00:12, 959.97it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 63%|██████▎ | 20509/32777 [00:17<00:12, 952.17it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 63%|██████▎ | 20607/32777 [00:17<00:12, 951.56it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 63%|██████▎ | 20712/32777 [00:17<00:12, 976.29it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 64%|██████▎ | 20834/32777 [00:17<00:11, 1043.17it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 64%|██████▍ | 20961/32777 [00:17<00:10, 1102.90it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 64%|██████▍ | 21103/32777 [00:17<00:09, 1187.95it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 65%|██████▍ | 21223/32777 [00:18<00:09, 1158.79it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 65%|██████▌ | 21359/32777 [00:18<00:09, 1212.72it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 66%|██████▌ | 21492/32777 [00:18<00:09, 1246.48it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 66%|██████▌ | 21645/32777 [00:18<00:08, 1328.81it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 66%|██████▋ | 21779/32777 [00:18<00:08, 1244.05it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 67%|██████▋ | 21905/32777 [00:18<00:09, 1172.14it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 67%|██████▋ | 22032/32777 [00:18<00:08, 1194.44it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 68%|██████▊ | 22164/32777 [00:18<00:08, 1220.60it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 68%|██████▊ | 22288/32777 [00:18<00:09, 1090.22it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 68%|██████▊ | 22401/32777 [00:19<00:11, 907.78it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 69%|██████▊ | 22499/32777 [00:19<00:11, 890.76it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 69%|██████▉ | 22629/32777 [00:19<00:10, 990.92it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 69%|██████▉ | 22743/32777 [00:19<00:09, 1029.57it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 70%|██████▉ | 22851/32777 [00:19<00:09, 1022.50it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 70%|███████ | 22957/32777 [00:19<00:10, 954.09it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 70%|███████ | 23056/32777 [00:19<00:10, 915.06it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 71%|███████ | 23162/32777 [00:19<00:10, 951.84it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 71%|███████ | 23260/32777 [00:20<00:09, 952.61it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 71%|███████▏ | 23412/32777 [00:20<00:08, 1109.97it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 72%|███████▏ | 23539/32777 [00:20<00:07, 1155.53it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 72%|███████▏ | 23677/32777 [00:20<00:07, 1213.40it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 73%|███████▎ | 23818/32777 [00:20<00:07, 1268.89it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 73%|███████▎ | 23971/32777 [00:20<00:06, 1342.22it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 74%|███████▎ | 24114/32777 [00:20<00:06, 1363.78it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 74%|███████▍ | 24251/32777 [00:20<00:07, 1211.59it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 74%|███████▍ | 24376/32777 [00:20<00:07, 1179.99it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 75%|███████▍ | 24513/32777 [00:20<00:06, 1230.80it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 75%|███████▌ | 24653/32777 [00:21<00:06, 1277.97it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 76%|███████▌ | 24815/32777 [00:21<00:05, 1370.16it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 76%|███████▌ | 24954/32777 [00:21<00:05, 1329.11it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 77%|███████▋ | 25094/32777 [00:21<00:05, 1335.46it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 77%|███████▋ | 25304/32777 [00:21<00:04, 1550.06it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 78%|███████▊ | 25461/32777 [00:21<00:04, 1484.39it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 78%|███████▊ | 25636/32777 [00:21<00:04, 1554.38it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 79%|███████▊ | 25793/32777 [00:21<00:04, 1555.36it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 79%|███████▉ | 25960/32777 [00:21<00:04, 1587.37it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 80%|███████▉ | 26120/32777 [00:22<00:04, 1473.36it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 80%|████████ | 26270/32777 [00:22<00:04, 1431.14it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 81%|████████ | 26415/32777 [00:22<00:05, 1134.15it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 81%|████████ | 26539/32777 [00:22<00:05, 1139.16it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 81%|████████▏ | 26691/32777 [00:22<00:04, 1229.78it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 82%|████████▏ | 26821/32777 [00:22<00:04, 1230.82it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 82%|████████▏ | 26962/32777 [00:22<00:04, 1275.90it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 83%|████████▎ | 27094/32777 [00:22<00:04, 1214.54it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 83%|████████▎ | 27232/32777 [00:23<00:04, 1257.86it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 84%|████████▎ | 27372/32777 [00:23<00:04, 1291.94it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 84%|████████▍ | 27512/32777 [00:23<00:03, 1320.56it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 84%|████████▍ | 27647/32777 [00:23<00:03, 1326.09it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 85%|████████▍ | 27781/32777 [00:23<00:03, 1285.54it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 85%|████████▌ | 27922/32777 [00:23<00:03, 1312.17it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 86%|████████▌ | 28054/32777 [00:23<00:03, 1222.29it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 86%|████████▌ | 28178/32777 [00:23<00:03, 1195.10it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 86%|████████▋ | 28299/32777 [00:23<00:04, 1076.82it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 87%|████████▋ | 28410/32777 [00:24<00:04, 1045.85it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 87%|████████▋ | 28541/32777 [00:24<00:03, 1114.29it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 87%|████████▋ | 28668/32777 [00:24<00:03, 1153.79it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 88%|████████▊ | 28786/32777 [00:24<00:03, 1074.73it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 88%|████████▊ | 28896/32777 [00:24<00:03, 1054.76it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 89%|████████▊ | 29044/32777 [00:24<00:03, 1169.72it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 89%|████████▉ | 29225/32777 [00:24<00:02, 1349.33it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 90%|████████▉ | 29363/32777 [00:24<00:02, 1333.65it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 90%|█████████ | 29522/32777 [00:24<00:02, 1401.68it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 91%|█████████ | 29664/32777 [00:24<00:02, 1285.23it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 91%|█████████ | 29796/32777 [00:25<00:02, 1261.55it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 91%|█████████▏| 29925/32777 [00:25<00:02, 1185.38it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 92%|█████████▏| 30071/32777 [00:25<00:02, 1254.46it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 92%|█████████▏| 30199/32777 [00:25<00:02, 1184.73it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 93%|█████████▎| 30353/32777 [00:25<00:01, 1280.64it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 93%|█████████▎| 30492/32777 [00:25<00:01, 1307.37it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 93%|█████████▎| 30635/32777 [00:25<00:01, 1340.02it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 94%|█████████▍| 30771/32777 [00:25<00:01, 1312.51it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 94%|█████████▍| 30904/32777 [00:26<00:01, 973.49it/s] "
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 95%|█████████▍| 31043/32777 [00:26<00:01, 1068.95it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 95%|█████████▌| 31208/32777 [00:26<00:01, 1207.38it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 96%|█████████▌| 31374/32777 [00:26<00:01, 1325.21it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 96%|█████████▌| 31547/32777 [00:26<00:00, 1430.27it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 97%|█████████▋| 31698/32777 [00:26<00:00, 1358.48it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 97%|█████████▋| 31840/32777 [00:26<00:00, 1343.18it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 98%|█████████▊| 31979/32777 [00:26<00:00, 1283.90it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 98%|█████████▊| 32111/32777 [00:26<00:00, 1272.53it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 98%|█████████▊| 32241/32777 [00:27<00:00, 1242.12it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 99%|█████████▉| 32382/32777 [00:27<00:00, 1286.56it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
" 99%|█████████▉| 32571/32777 [00:27<00:00, 1457.17it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"100%|█████████▉| 32727/32777 [00:27<00:00, 1485.36it/s]"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"100%|██████████| 32777/32777 [00:27<00:00, 1196.59it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"64362 64362 64362\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"targets, contexts, labels = generate_training_data(\n",
" sequences=sequences, \n",
" window_size=2, \n",
" num_ns=4, \n",
" vocab_size=vocab_size, \n",
" seed=SEED)\n",
"print(len(targets), len(contexts), len(labels))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "97PqsusOFEpc"
},
"source": [
"### Configure the dataset for performance"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7jnFVySViQTj"
},
"source": [
"To perform efficient batching for the potentially large number of training examples, use the `tf.data.Dataset` API. After this step, you would have a `tf.data.Dataset` object of `(target_word, context_word), (label)` elements to train your Word2Vec model!"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:00:58.446610Z",
"iopub.status.busy": "2021-01-21T03:00:58.430902Z",
"iopub.status.idle": "2021-01-21T03:01:03.282771Z",
"shell.execute_reply": "2021-01-21T03:01:03.283225Z"
},
"id": "nbu8PxPSnVY2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<BatchDataset shapes: (((1024,), (1024, 5, 1)), (1024, 5)), types: ((tf.int32, tf.int64), tf.int64)>\n"
]
}
],
"source": [
"BATCH_SIZE = 1024\n",
"BUFFER_SIZE = 10000\n",
"dataset = tf.data.Dataset.from_tensor_slices(((targets, contexts), labels))\n",
"dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)\n",
"print(dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tyrNX6Fs6K3F"
},
"source": [
"Add `cache()` and `prefetch()` to improve performance."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:03.288682Z",
"iopub.status.busy": "2021-01-21T03:01:03.287892Z",
"iopub.status.idle": "2021-01-21T03:01:03.290788Z",
"shell.execute_reply": "2021-01-21T03:01:03.291164Z"
},
"id": "Y5Ueg6bcFPVL"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<PrefetchDataset shapes: (((1024,), (1024, 5, 1)), (1024, 5)), types: ((tf.int32, tf.int64), tf.int64)>\n"
]
}
],
"source": [
"dataset = dataset.cache().prefetch(buffer_size=AUTOTUNE)\n",
"print(dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1S-CmUMszyEf"
},
"source": [
"## Lab Task 3: Model and Training"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sQFqaBMPwBqC"
},
"source": [
"The Word2Vec model can be implemented as a classifier to distinguish between true context words from skip-grams and false context words obtained through negative sampling. You can perform a dot product between the embeddings of target and context words to obtain predictions for labels and compute loss against true labels in the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oc7kTbiwD9sy"
},
"source": [
"### Subclassed Word2Vec Model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Jvr9pM1G1sQN"
},
"source": [
"Use the [Keras Subclassing API](https://www.tensorflow.org/guide/keras/custom_layers_and_models) to define your Word2Vec model with the following layers:\n",
"\n",
"\n",
"* `target_embedding`: A `tf.keras.layers.Embedding` layer which looks up the embedding of a word when it appears as a target word. The number of parameters in this layer are `(vocab_size * embedding_dim)`.\n",
"* `context_embedding`: Another `tf.keras.layers.Embedding` layer which looks up the embedding of a word when it appears as a context word. The number of parameters in this layer are the same as those in `target_embedding`, i.e. `(vocab_size * embedding_dim)`.\n",
"* `dots`: A `tf.keras.layers.Dot` layer that computes the dot product of target and context embeddings from a training pair.\n",
"* `flatten`: A `tf.keras.layers.Flatten` layer to flatten the results of `dots` layer into logits.\n",
"\n",
"With the sublassed model, you can define the `call()` function that accepts `(target, context)` pairs which can then be passed into their corresponding embedding layer. Reshape the `context_embedding` to perform a dot product with `target_embedding` and return the flattened result."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KiAwuIqqw7-7"
},
"source": [
"Key point: The `target_embedding` and `context_embedding` layers can be shared as well. You could also use a concatenation of both embeddings as the final Word2Vec embedding."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:03.298311Z",
"iopub.status.busy": "2021-01-21T03:01:03.297643Z",
"iopub.status.idle": "2021-01-21T03:01:03.299432Z",
"shell.execute_reply": "2021-01-21T03:01:03.299791Z"
},
"id": "i9ec-sS6xd8Z"
},
"outputs": [],
"source": [
"class Word2Vec(Model):\n",
" def __init__(self, vocab_size, embedding_dim):\n",
" super(Word2Vec, self).__init__()\n",
" self.target_embedding = Embedding(vocab_size, \n",
" embedding_dim,\n",
" input_length=1,\n",
" name=\"w2v_embedding\", )\n",
" self.context_embedding = Embedding(vocab_size, \n",
" embedding_dim, \n",
" input_length=num_ns+1)\n",
" self.dots = Dot(axes=(3,2))\n",
" self.flatten = Flatten()\n",
"\n",
" def call(self, pair):\n",
" target, context = pair\n",
" we = self.target_embedding(target)\n",
" ce = self.context_embedding(context)\n",
" dots = self.dots([ce, we])\n",
" return self.flatten(dots)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-RLKz9LFECXu"
},
"source": [
"### Define loss function and compile model\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I3Md-9QanqBM"
},
"source": [
"For simplicity, you can use `tf.keras.losses.CategoricalCrossEntropy` as an alternative to the negative sampling loss. If you would like to write your own custom loss function, you can also do so as follows:\n",
"\n",
"``` python\n",
"def custom_loss(x_logit, y_true):\n",
" return tf.nn.sigmoid_cross_entropy_with_logits(logits=x_logit, labels=y_true)\n",
"```\n",
"\n",
"It's time to build your model! Instantiate your Word2Vec class with an embedding dimension of 128 (you could experiment with different values). Compile the model with the `tf.keras.optimizers.Adam` optimizer. "
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:03.306680Z",
"iopub.status.busy": "2021-01-21T03:01:03.306015Z",
"iopub.status.idle": "2021-01-21T03:01:03.338993Z",
"shell.execute_reply": "2021-01-21T03:01:03.339416Z"
},
"id": "ekQg_KbWnnmQ"
},
"outputs": [],
"source": [
"# TODO 3a -- your code goes here"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "P3MUMrluqNX2"
},
"source": [
"Also define a callback to log training statistics for tensorboard."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:03.344614Z",
"iopub.status.busy": "2021-01-21T03:01:03.343968Z",
"iopub.status.idle": "2021-01-21T03:01:03.346083Z",
"shell.execute_reply": "2021-01-21T03:01:03.345649Z"
},
"id": "9d-ftBCeEZIR"
},
"outputs": [],
"source": [
"tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=\"logs\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "h5wEBotlGZ7B"
},
"source": [
"Train the model with `dataset` prepared above for some number of epochs."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:03.350012Z",
"iopub.status.busy": "2021-01-21T03:01:03.349464Z",
"iopub.status.idle": "2021-01-21T03:01:11.787650Z",
"shell.execute_reply": "2021-01-21T03:01:11.788032Z"
},
"id": "gmC1BJalEZIY"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/20\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" 1/62 [..............................] - ETA: 57s - loss: 1.6086 - accuracy: 0.2236"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
" 3/62 [>.............................] - ETA: 2s - loss: 1.6090 - accuracy: 0.2168 "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 1.6093 - accuracy: 0.2127"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"20/62 [========>.....................] - ETA: 0s - loss: 1.6093 - accuracy: 0.2135"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 1.6092 - accuracy: 0.2141"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"37/62 [================>.............] - ETA: 0s - loss: 1.6092 - accuracy: 0.2154"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"47/62 [=====================>........] - ETA: 0s - loss: 1.6091 - accuracy: 0.2170"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"55/62 [=========================>....] - ETA: 0s - loss: 1.6090 - accuracy: 0.2183"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 1s 7ms/step - loss: 1.6089 - accuracy: 0.2197\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 2/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.5899 - accuracy: 0.7832"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"10/62 [===>..........................] - ETA: 0s - loss: 1.5930 - accuracy: 0.6796"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"18/62 [=======>......................] - ETA: 0s - loss: 1.5937 - accuracy: 0.6478"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 1.5939 - accuracy: 0.6270"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"38/62 [=================>............] - ETA: 0s - loss: 1.5937 - accuracy: 0.6143"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"46/62 [=====================>........] - ETA: 0s - loss: 1.5935 - accuracy: 0.6064"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"56/62 [==========================>...] - ETA: 0s - loss: 1.5931 - accuracy: 0.5984"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.5928 - accuracy: 0.5937\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 3/20\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.5601 - accuracy: 0.7656"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
" 9/62 [===>..........................] - ETA: 0s - loss: 1.5630 - accuracy: 0.7238"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"18/62 [=======>......................] - ETA: 0s - loss: 1.5627 - accuracy: 0.7008"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 1.5616 - accuracy: 0.6862"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"36/62 [================>.............] - ETA: 0s - loss: 1.5604 - accuracy: 0.6776"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"46/62 [=====================>........] - ETA: 0s - loss: 1.5588 - accuracy: 0.6681"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"56/62 [==========================>...] - ETA: 0s - loss: 1.5571 - accuracy: 0.6596"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.5558 - accuracy: 0.6543\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 4/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.4925 - accuracy: 0.6670"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 1.4947 - accuracy: 0.6363"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"21/62 [=========>....................] - ETA: 0s - loss: 1.4927 - accuracy: 0.6251"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"30/62 [=============>................] - ETA: 0s - loss: 1.4903 - accuracy: 0.6187"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"39/62 [=================>............] - ETA: 0s - loss: 1.4878 - accuracy: 0.6135"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"49/62 [======================>.......] - ETA: 0s - loss: 1.4850 - accuracy: 0.6085"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"58/62 [===========================>..] - ETA: 0s - loss: 1.4824 - accuracy: 0.6045"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.4810 - accuracy: 0.6025\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 5/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.3973 - accuracy: 0.6113"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 1.3988 - accuracy: 0.6039"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"20/62 [========>.....................] - ETA: 0s - loss: 1.3964 - accuracy: 0.6010"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 1.3936 - accuracy: 0.5986"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"39/62 [=================>............] - ETA: 0s - loss: 1.3905 - accuracy: 0.5963"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"47/62 [=====================>........] - ETA: 0s - loss: 1.3881 - accuracy: 0.5946"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"56/62 [==========================>...] - ETA: 0s - loss: 1.3855 - accuracy: 0.5928"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.3835 - accuracy: 0.5918\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 6/20\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.2968 - accuracy: 0.6182"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"10/62 [===>..........................] - ETA: 0s - loss: 1.2982 - accuracy: 0.6144"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 1.2959 - accuracy: 0.6141"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 1.2929 - accuracy: 0.6134"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"38/62 [=================>............] - ETA: 0s - loss: 1.2903 - accuracy: 0.6129"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"47/62 [=====================>........] - ETA: 0s - loss: 1.2880 - accuracy: 0.6121"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"57/62 [==========================>...] - ETA: 0s - loss: 1.2855 - accuracy: 0.6113"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.2840 - accuracy: 0.6110\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 7/20\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.2016 - accuracy: 0.6445"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"10/62 [===>..........................] - ETA: 0s - loss: 1.2031 - accuracy: 0.6404"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 1.2010 - accuracy: 0.6409"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 1.1985 - accuracy: 0.6415"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"36/62 [================>.............] - ETA: 0s - loss: 1.1965 - accuracy: 0.6417"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"45/62 [====================>.........] - ETA: 0s - loss: 1.1945 - accuracy: 0.6416"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"54/62 [=========================>....] - ETA: 0s - loss: 1.1927 - accuracy: 0.6413"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - ETA: 0s - loss: 1.1911 - accuracy: 0.6412"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.1909 - accuracy: 0.6412\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 8/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.1132 - accuracy: 0.6758"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"10/62 [===>..........................] - ETA: 0s - loss: 1.1151 - accuracy: 0.6757"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 1.1134 - accuracy: 0.6760"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"27/62 [============>.................] - ETA: 0s - loss: 1.1114 - accuracy: 0.6762"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"36/62 [================>.............] - ETA: 0s - loss: 1.1094 - accuracy: 0.6766"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"45/62 [====================>.........] - ETA: 0s - loss: 1.1077 - accuracy: 0.6766"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"53/62 [========================>.....] - ETA: 0s - loss: 1.1064 - accuracy: 0.6764"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - ETA: 0s - loss: 1.1049 - accuracy: 0.6763"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.1047 - accuracy: 0.6763\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 9/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 1.0313 - accuracy: 0.7100"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"10/62 [===>..........................] - ETA: 0s - loss: 1.0339 - accuracy: 0.7087"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 1.0325 - accuracy: 0.7093"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 1.0301 - accuracy: 0.7093"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"37/62 [================>.............] - ETA: 0s - loss: 1.0286 - accuracy: 0.7095"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"46/62 [=====================>........] - ETA: 0s - loss: 1.0272 - accuracy: 0.7093"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"56/62 [==========================>...] - ETA: 0s - loss: 1.0259 - accuracy: 0.7090"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 1.0249 - accuracy: 0.7090\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 10/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.9556 - accuracy: 0.7344"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"10/62 [===>..........................] - ETA: 0s - loss: 0.9586 - accuracy: 0.7377"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 0.9576 - accuracy: 0.7383"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 0.9555 - accuracy: 0.7384"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"36/62 [================>.............] - ETA: 0s - loss: 0.9541 - accuracy: 0.7387"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"45/62 [====================>.........] - ETA: 0s - loss: 0.9529 - accuracy: 0.7386"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"54/62 [=========================>....] - ETA: 0s - loss: 0.9519 - accuracy: 0.7382"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.9508 - accuracy: 0.7382\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 11/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.8857 - accuracy: 0.7627"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 0.8891 - accuracy: 0.7651"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 0.8883 - accuracy: 0.7651"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 0.8863 - accuracy: 0.7650"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"37/62 [================>.............] - ETA: 0s - loss: 0.8848 - accuracy: 0.7652"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"46/62 [=====================>........] - ETA: 0s - loss: 0.8838 - accuracy: 0.7651"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"55/62 [=========================>....] - ETA: 0s - loss: 0.8830 - accuracy: 0.7647"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.8822 - accuracy: 0.7645\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 12/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.8213 - accuracy: 0.7881"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"10/62 [===>..........................] - ETA: 0s - loss: 0.8248 - accuracy: 0.7873"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 0.8243 - accuracy: 0.7871"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 0.8223 - accuracy: 0.7872"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"37/62 [================>.............] - ETA: 0s - loss: 0.8210 - accuracy: 0.7876"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"46/62 [=====================>........] - ETA: 0s - loss: 0.8202 - accuracy: 0.7875"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"56/62 [==========================>...] - ETA: 0s - loss: 0.8195 - accuracy: 0.7872"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.8188 - accuracy: 0.7871\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 13/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.7622 - accuracy: 0.8057"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 0.7659 - accuracy: 0.8057"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"21/62 [=========>....................] - ETA: 0s - loss: 0.7652 - accuracy: 0.8063"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"30/62 [=============>................] - ETA: 0s - loss: 0.7634 - accuracy: 0.8067"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"39/62 [=================>............] - ETA: 0s - loss: 0.7621 - accuracy: 0.8072"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"49/62 [======================>.......] - ETA: 0s - loss: 0.7614 - accuracy: 0.8070"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"58/62 [===========================>..] - ETA: 0s - loss: 0.7609 - accuracy: 0.8067"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.7605 - accuracy: 0.8067\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 14/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.7082 - accuracy: 0.8145"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 0.7119 - accuracy: 0.8214"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"20/62 [========>.....................] - ETA: 0s - loss: 0.7114 - accuracy: 0.8226"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 0.7097 - accuracy: 0.8233"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"39/62 [=================>............] - ETA: 0s - loss: 0.7084 - accuracy: 0.8240"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"48/62 [======================>.......] - ETA: 0s - loss: 0.7078 - accuracy: 0.8240"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"57/62 [==========================>...] - ETA: 0s - loss: 0.7074 - accuracy: 0.8238"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.7070 - accuracy: 0.8238\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 15/20\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.6588 - accuracy: 0.8320"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 0.6624 - accuracy: 0.8367"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 0.6623 - accuracy: 0.8376"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 0.6605 - accuracy: 0.8385"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"38/62 [=================>............] - ETA: 0s - loss: 0.6593 - accuracy: 0.8393"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"46/62 [=====================>........] - ETA: 0s - loss: 0.6588 - accuracy: 0.8394"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"55/62 [=========================>....] - ETA: 0s - loss: 0.6585 - accuracy: 0.8393"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.6580 - accuracy: 0.8393\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 16/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.6139 - accuracy: 0.8525"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
" 9/62 [===>..........................] - ETA: 0s - loss: 0.6170 - accuracy: 0.8525"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"18/62 [=======>......................] - ETA: 0s - loss: 0.6175 - accuracy: 0.8527"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 0.6157 - accuracy: 0.8537"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"36/62 [================>.............] - ETA: 0s - loss: 0.6146 - accuracy: 0.8543"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"45/62 [====================>.........] - ETA: 0s - loss: 0.6140 - accuracy: 0.8545"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"55/62 [=========================>....] - ETA: 0s - loss: 0.6137 - accuracy: 0.8543"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.6134 - accuracy: 0.8542\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 17/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.5730 - accuracy: 0.8672"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 0.5764 - accuracy: 0.8659"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"21/62 [=========>....................] - ETA: 0s - loss: 0.5761 - accuracy: 0.8660"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 0.5747 - accuracy: 0.8666"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"39/62 [=================>............] - ETA: 0s - loss: 0.5735 - accuracy: 0.8673"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"49/62 [======================>.......] - ETA: 0s - loss: 0.5731 - accuracy: 0.8673"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"57/62 [==========================>...] - ETA: 0s - loss: 0.5730 - accuracy: 0.8671"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.5727 - accuracy: 0.8670\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 18/20\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.5359 - accuracy: 0.8760"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 0.5391 - accuracy: 0.8761"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"19/62 [========>.....................] - ETA: 0s - loss: 0.5391 - accuracy: 0.8760"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 0.5377 - accuracy: 0.8766"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"38/62 [=================>............] - ETA: 0s - loss: 0.5365 - accuracy: 0.8773"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"47/62 [=====================>........] - ETA: 0s - loss: 0.5361 - accuracy: 0.8775"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"56/62 [==========================>...] - ETA: 0s - loss: 0.5359 - accuracy: 0.8774"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.5357 - accuracy: 0.8773\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 19/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.5022 - accuracy: 0.8828"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
" 9/62 [===>..........................] - ETA: 0s - loss: 0.5049 - accuracy: 0.8834"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"18/62 [=======>......................] - ETA: 0s - loss: 0.5054 - accuracy: 0.8836"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"28/62 [============>.................] - ETA: 0s - loss: 0.5039 - accuracy: 0.8847"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"36/62 [================>.............] - ETA: 0s - loss: 0.5029 - accuracy: 0.8855"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"45/62 [====================>.........] - ETA: 0s - loss: 0.5024 - accuracy: 0.8859"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"55/62 [=========================>....] - ETA: 0s - loss: 0.5023 - accuracy: 0.8860"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.5021 - accuracy: 0.8860\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 20/20\n",
"\r",
" 1/62 [..............................] - ETA: 0s - loss: 0.4716 - accuracy: 0.8965"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"11/62 [====>.........................] - ETA: 0s - loss: 0.4745 - accuracy: 0.8938"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"21/62 [=========>....................] - ETA: 0s - loss: 0.4743 - accuracy: 0.8938"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"29/62 [=============>................] - ETA: 0s - loss: 0.4730 - accuracy: 0.8944"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"39/62 [=================>............] - ETA: 0s - loss: 0.4719 - accuracy: 0.8950"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"49/62 [======================>.......] - ETA: 0s - loss: 0.4717 - accuracy: 0.8952"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"57/62 [==========================>...] - ETA: 0s - loss: 0.4716 - accuracy: 0.8951"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
"62/62 [==============================] - 0s 6ms/step - loss: 0.4715 - accuracy: 0.8951\n"
]
},
{
"data": {
"text/plain": [
"<tensorflow.python.keras.callbacks.History at 0x7f65706f69b0>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"word2vec.fit(dataset, epochs=20, callbacks=[tensorboard_callback])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tensorboard now shows the Word2Vec model's accuracy and loss."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!tensorboard --bind_all --port=8081 --load_fast=false --logdir logs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run the following command in **Cloud Shell:**\n",
"\n",
"<code>gcloud beta compute ssh --zone <instance-zone> <notebook-instance-name> --project <project-id> -- -L 8081:localhost:8081</code> \n",
"\n",
"Make sure to replace <instance-zone>, <notebook-instance-name> and <project-id>.\n",
"\n",
"In Cloud Shell, click *Web Preview* > *Change Port* and insert port number *8081*. Click *Change and Preview* to open the TensorBoard."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QvURkGVpXDOy"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**To quit the TensorBoard, click Kernel > Interrupt kernel**."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TaDW2tIIz8fL"
},
"source": [
"## Lab Task 4: Embedding lookup and analysis"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Zp5rv01WG2YA"
},
"source": [
"Obtain the weights from the model using `get_layer()` and `get_weights()`. The `get_vocabulary()` function provides the vocabulary to build a metadata file with one token per line. "
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:11.793637Z",
"iopub.status.busy": "2021-01-21T03:01:11.792261Z",
"iopub.status.idle": "2021-01-21T03:01:11.801164Z",
"shell.execute_reply": "2021-01-21T03:01:11.800655Z"
},
"id": "_Uamp1YH8RzU"
},
"outputs": [],
"source": [
"# TODO 4a -- your code goes here"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gWzdmUzS8Sl4"
},
"source": [
"Create and save the vectors and metadata file. "
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:11.806879Z",
"iopub.status.busy": "2021-01-21T03:01:11.805849Z",
"iopub.status.idle": "2021-01-21T03:01:12.149051Z",
"shell.execute_reply": "2021-01-21T03:01:12.148328Z"
},
"id": "VLIahl9s53XT"
},
"outputs": [],
"source": [
"out_v = io.open('vectors.tsv', 'w', encoding='utf-8')\n",
"out_m = io.open('metadata.tsv', 'w', encoding='utf-8')\n",
"\n",
"for index, word in enumerate(vocab):\n",
" if index == 0: continue # skip 0, it's padding.\n",
" vec = weights[index] \n",
" out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n",
" out_m.write(word + \"\\n\")\n",
"out_v.close()\n",
"out_m.close()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1T8KcThhIU8-"
},
"source": [
"Download the `vectors.tsv` and `metadata.tsv` to analyze the obtained embeddings in the [Embedding Projector](https://projector.tensorflow.org/)."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-21T03:01:12.155357Z",
"iopub.status.busy": "2021-01-21T03:01:12.154229Z",
"iopub.status.idle": "2021-01-21T03:01:12.156768Z",
"shell.execute_reply": "2021-01-21T03:01:12.156132Z"
},
"id": "lUsjQOKMIV2z"
},
"outputs": [],
"source": [
"try:\n",
" from google.colab import files\n",
" files.download('vectors.tsv')\n",
" files.download('metadata.tsv')\n",
"except Exception as e:\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iS_uMeMw3Xpj"
},
"source": [
"## Next steps\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BSgAZpwF5xF_"
},
"source": [
"This tutorial has shown you how to implement a skip-gram Word2Vec model with negative sampling from scratch and visualize the obtained word embeddings.\n",
"\n",
"* To learn more about word vectors and their mathematical representations, refer to these [notes](https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf).\n",
"\n",
"* To learn more about advanced text processing, read the [Transformer model for language understanding](https://www.tensorflow.org/tutorials/text/transformer) tutorial.\n",
"\n",
"* If you’re interested in pre-trained embedding models, you may also be interested in [Exploring the TF-Hub CORD-19 Swivel Embeddings](https://www.tensorflow.org/hub/tutorials/cord_19_embeddings_keras), or the [Multilingual Universal Sentence Encoder](https://www.tensorflow.org/hub/tutorials/cross_lingual_similarity_with_tf_hub_multilingual_universal_encoder)\n",
"\n",
"* You may also like to train the model on a new dataset (there are many available in [TensorFlow Datasets](https://www.tensorflow.org/datasets)).\n"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "word2vec.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 0
}