The topic Understanding Transformers Part 12: Building the Decoder Layers is currently the subject of lively discussion — readers and analysts are keeping a close eye on developments.
This is taking place in a dynamic environment: companies’ decisions and competitors’ reactions can quickly change the picture.
In the previous article, we just began with the concept of decoders in a transformer.
Just like before, we use the same sine and cosine curves to get positional values based on the embedding positions.
These are the same curves that were used earlier when encoding the input.
Since the <EOS> token is in the first position and has two embedding values, we take the corresponding positional values from the curves.
As a result, we get 2.70 and -0.34, which represent the <EOS> token after adding positional encoding.
Next, we add the self-attention layer so the decoder can keep track of relationships between output words.
The self-attention values for the <EOS> token are -2.8 and -2.3.
Note that the weights used in the decoder’s self-attention (for queries, keys, and values) are different from those used in the encoder.
Now, we add residual connections, just like we did in the encoder.
So far, we have seen how self-attention helps the transformer understand relationships within the output sentence.
However, for tasks like translation, the model also needs to understand relationships between the input sentence and the output sentence.
Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.
Templates let you quickly answer FAQs or store snippets for re-use.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment’s permalink.
For further actions, you may consider blocking this person and/or reporting abuse
Thank you to our Diamond Sponsors for supporting the DEV Community
Google AI is the official AI Model and Platform Partner of DEV
DEV Community — A space to discuss and keep up software development and manage your software career
Built on Forem — the open source software that powers DEV and other inclusive communities.
We’re a place where coders share, stay up-to-date and grow their careers.
