(upbeat music)
- Text embeddings and semantic search.
In this video we'll explore how Transformer models
represent text as embedding vectors
and how these vectors can be used to find similar documents
in a corpus.
Text embeddings are just a fancy way of saying
that we can represent text as an array of numbers
called a vector.
To create these embeddings we usually use
an encoder-based model like BERT.
In this example, you can see how we feed three sentences
to the encoder and get three vectors as the output.
Reading the text, we can see that walking the dog
seems to be most similar to walking the cat,
but let's see if we can quantify this.
The trick to do the comparison
is to compute a similarity metric
between each pair of embedding vectors.
These vectors usually live in a very high-dimensional space,
so a similarity metric can be anything that measures
some sort of distance between vectors.
One very popular metric is cosine similarity,
which uses the angle between two vectors
to measure how close they are.
In this example, our embedding vectors live in 3D
and we can see that the orange and Grey vectors
are close to each other and have a smaller angle.
Now one problem we have to deal with
is that Transformer models like BERT will actually return
one embedding vector per token.
For example in the sentence, "I took my dog for a walk,"
we can expect several embedding vectors, one for each word.
For example, here we can see the output of our model
has produced 9 embedding vectors per sentence,
and each vector has 384 dimensions.
But what we really want is a single embedding vector
for each sentence.
To deal with this, we can use a technique called pooling.
The simplest pooling method
is to just take the token embedding
of the special CLS token.
Alternatively, we can average the token embeddings
which is called mean_pooling and this is what we do here.
With mean_pooling the only thing we need to make sure
is that we don't include the padding tokens in the average,
which is why you can see the attention_mask being used here.
This gives us a 384 dimensional vector for each sentence
which is exactly what we want.
And once we have our sentence embeddings,
we can compute the cosine similarity
for each pair of vectors.
In this example we use the function from scikit-learn
and you can see that the sentence "I took my dog for a walk"
has indeed a strong overlap with "I took my cat for a walk".
Hooray! We've done it.
We can actually take this idea one step further
by comparing the similarity between a question
and a corpus of documents.
For example, suppose we embed every post
in the Hugging Face forums.
We can then ask a question, embed it,
and check which forum posts are most similar.
This process is often called semantic search,
because it allows us to compare queries with context.
To create a semantic search engine
is actually quite simple in the datasets library.
First we need to embed all the documents.
And in this example, we take a small sample
from the squad dataset and apply
the same embedding logic as before.
This gives us a new column called embeddings,
which stores the embeddings of every passage.
Once we have our embeddings,
we need a way to find nearest neighbors for a query.
The datasets library provides a special object called FAISS
which allows you to quickly compare embedding vectors.
So we add the FAISS index, embed a question and voila,
we've now found the 3 most similar articles
which might store the answer.
(upbeat music)