◑ applying · kind model · level 3 · 52.17h
- Implementa: Attention
- Requiere: Self-Attention · Multi-Head Attention · Embeddings
- Cita: Attention Is All You Need
Self-attention + feed-forward blocks with residual connections and no recurrence, so the whole sequence trains in parallel. The backbone of every modern LLM and of the Open Intelligence Stack work.
In deep learning, the transformer is a family of artificial Neural Network architectures based on the Multi-Head Attention mechanism, in which input data such as text, images, or audio, is converted to a sequence of numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel Multi-Head Attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished.
Enlaces
- Implementa: Attention
- Requiere: Self-Attention · Multi-Head Attention · Embeddings
- Cita: Attention Is All You Need