The topic Understanding Transformers Part 5: Queries, Keys, and Similarity is currently the subject of lively discussion — readers and analysts are keeping a close eye on developments.
This is taking place in a dynamic environment: companies’ decisions and competitors’ reactions can quickly change the picture.
In the previous article, we explored the self-attention concept for transformers, in this article we will go deeper into how the comparisons are performed.
We have already added positional encoding to the words “Let’s” and “go”.
The first step is to multiply the position-encoded values for the word “Let’s” by a set of weights.
Next, we repeat the same process using a different set of weights, which gives us another value (for example, 3.7).
We do this twice because we started with two position-encoded values representing the word “Let’s”.
These resulting values together represent “Let’s” in a new form.
Now, we use these query values to measure similarity with other words, such as “go”.
To do this, we first create a new set of values for each word, similar to how we created the query values.
We will use these key values along with the query values to calculate how similar “Let’s” is to “go”.
We will explore how this similarity is calculated in the next article.
Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.
Templates let you quickly answer FAQs or store snippets for re-use.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment’s permalink.
For further actions, you may consider blocking this person and/or reporting abuse
Thank you to our Diamond Sponsors for supporting the DEV Community
Google AI is the official AI Model and Platform Partner of DEV
DEV Community — A space to discuss and keep up software development and manage your software career
Built on Forem — the open source software that powers DEV and other inclusive communities.
We’re a place where coders share, stay up-to-date and grow their careers.
