- 在本视频中，我们将研究解码器架构。 - In this video, we'll study the decoder architecture.
一种流行的仅包含解码器架构 An example
的例子是 GPT-2。 of a popular decoder only architecture is GPT-2.
为了了解解码器的工作原理 In order to understand how decoders work
我们建议您观看有关编码器的视频。 we recommend taking a look at the video regarding encoders.
它们与解码器极为相似。 They're extremely similar to decoders.
可以使用解码器 One can use a decoder
执行与编码器相同的大多数任务 for most of the same tasks as an encoder
尽管通常会有一点性能损失。 albeit with generally a little loss of performance.
让我们采用相同的方法 Let's take the same approach we have taken
用编码器试试 with the encoder to try
并了解在编码器和解码器之间 and understand the architectural differences
的架构差异。 between an encoder and decoder.
我们将使用一个使用三个词的小例子。 We'll use a small example using three words.
我们通过解码器传递它们。 We pass them through their decoder.
我们检索每个单词的数值表示。 We retrieve a numerical representation for each word.
这里举例来说，解码器对三个词进行转换。 Here for example, the decoder converts the three words.
Welcome to NYC，这三个数字序列。 Welcome to NYC, and these three sequences of numbers.
解码器针对每个输入词汇 The decoder outputs exactly one sequence
只输出一个数列。 of numbers per input word.
这个数值表示也可以 This numerical representation can also
称为特征向量（feature vector）或特征传感器（feature sensor）。 be called a feature vector or a feature sensor.
让我们深入了解这种表现形式。 Let's dive in this representation.
它包含了每个通过解码器 It contains one vector
传递的单词的一个向量。 per word that was passed through the decoder.
这些向量中的每一个单词 Each of these vectors is a numerical representation
都是一个数值表示。 of the word in question.
这个向量的维度 The dimension of that vector is defined
由模型的架构所决定。 by the architecture of the model.
解码器与编码器的主要区别在于 Where the decoder differs from the encoder is principally
具有自注意力机制。 with its self attention mechanism.
它使用所谓的掩蔽自注意力。 It's using what is called masked self attention.
例如，在这里，如果我们关注 “to” 这个词 Here, for example, if we focus on the word "to"
我们会发现它的向量 we'll see that is vector is absolutely unmodified
完全未被 NYC 单词修改。 by the NYC word.
那是因为右边所有的话，即 That's because all the words on the right, also known
单词的右侧上下文都被屏蔽了 as the right context of the word is masked rather
而没有从左侧和右侧的所有单词中受益。 than benefiting from all the words on the left and right.
所以双向上下文。 So the bidirectional context.
解码器只能访问一个上下文 Decoders only have access to a single context
可以是左上下文或右上下文。 which can be the left context or the right context.
掩蔽自注意力机制不同于 The masked self attention mechanism differs
自注意力机制 from the self attention mechanism
通过使用额外的掩码在单词的两边 by using an additional mask to hide the context
来隐藏上下文 on either side of the word
通过隐藏上下文中的单词 the words numerical representation will not be affected
单词数值表示不会受到影响。 by the words in the hidden context.
那么什么时候应该使用解码器呢？ So when should one use a decoder?
像编码器这样的解码器可以用作独立模型 Decoders like encoders can be used as standalone models
因为它们生成数值表示。 as they generate a numerical representation.
它们还可以用于各种各样的任务。 They can also be used in a wide variety of tasks.
然而，解码器的力量在于方式。 However, the strength of a decoder lies in the way.
一个词只能访问其左侧上下文 A word can only have access to its left context
因为它只有左侧的上下文信息。 having only access to their left context.
它们天生擅长文本生成 They're inherently good at text generation
即在已知的词序列基础上生成一个单词 the ability to generate a word
或单词序列的能力。 or a sequence of words given a known sequence of words.
这被称为 This is known
因果语言建模或自然语言生成。 as causal language modeling or natural language generation.
下面是一个展示因果语言模型的工作原理的示例。 Here's an example of how causal language modeling works.
我们从一个词 my 开始， We start with an initial word, which is my
我们将其用作解码器的输入。 we use this as input for the decoder.
该模型输出一个数字向量 The model outputs a vector of numbers
这个向量包含有关序列的信息 and this vector contains information about the sequence
这里的序列是一个单词。 which is here a single word.
我们对该向量 We apply a small transformation
应用一个小的转换 to that vector so that it maps
使其映射到模型已知的所有单词 to all the words known by the model, which is a mapping
这个映射我们稍后会看到，称为语言模型头部信息 that we'll see later called a language modeling head.
我们发现模型认为 We identify that the model believes
接下来最有可能的单词是 “name”。 that the most probable following word is name.
然后我们把这个新单词加到原始的序列 my 后面 We then take that new word and add it
我们得到了 my name。 to the initial sequence from my, we are now at my name.
这就是自回归（auto-regressive）的作用所在。 This is where the auto regressive aspect comes in.
自回归模型。 Auto regressive models.
我们使用它们过去的输出作为输入和接下来的步骤。 We use their past outputs as inputs and the following steps.
我们再次执行完全相同的操作。 Once again, we do the exact same operation.
我们通过解码器投射那个序列 We cast that sequence through the decoder
并检索最有可能的后续词。 and retrieve the most probable following word.
本例中就是 “is” 这个词，我们重复操作 In this case, it is the word "is", we repeat the operation
直到我们满意为止，从一个词开始。 until we're satisfied, starting from a single word.
我们现在已经生成了一个完整的句子。 We've now generated a full sentence.
我们决定就此打住，但我们也可以继续一段时间。 We decide to stop there, but we could continue for a while.
例如，GPT-2 的最大上下文大小为 1,024。 GPT-2, for example, has a maximum context size of 1,024.
我们最终可以生成多达 1,024 个单词 We could eventually generate up to a 1,024 words
并且解码器仍然会对这个序列的前几个单词 and the decoder would still have some memory
有一些记忆。 of the first words in this sequence.