To raised perceive MLA and likewise make this text self-contained, we are going to revisit a number of associated ideas on this part earlier than diving into the main points of MLA.
MHA in Decoder-only Transformers
Notice that MLA is developed to speedup inference velocity in autoregressive textual content technology, so the MHA we’re speaking about underneath this context is for decoder-only Transformer.
The determine beneath compares three Transformer architectures used for decoding, the place (a) exhibits each the encoder and decoder proposed within the authentic “Consideration is All You Want” paper. Its decoder half is then simplified by [6], resulting in a decoder-only Transformer mannequin proven in (b), which is later utilized in many technology fashions like GPT [8].
These days, LLMs are extra generally to decide on the construction proven in (c) for extra secure coaching, with normalization utilized on the enter slightly then output, and LayerNorm upgraded to RMS Norm. This can function the baseline structure we are going to talk about on this article.
Inside this context, MHA calculation largely follows the method in [6], as proven within the determine beneath:
Assume we now have n_h consideration heads, and the dimension for every consideration head is represented as d_h, in order that the concatenated dimension shall be (h_n · d_h).
Given a mannequin with l layers, if we denote the enter for the t-th token in that layer as h_t with dimension d, we have to map the dimension of h_t from d to (h_n · d_h) utilizing the linear mapping matrices.
Extra formally, we now have (equations from [3]):
the place W^Q, W^Okay and W^V are the linear mapping matrices:
After such mapping, q_t, k_t and v_t shall be break up into n_h heads to calculate the scaled dot-product consideration:
the place W^O is one other projection matrix to map the dimension inversely from (h_n · d_h) to d:
Notice that the method described by Eqn.(1) to (8) above is only for a single token. Throughout inference, we have to repeat this course of for every newly generated token, which includes plenty of repeated calculation. This results in a way referred to as Key-Worth cache.
Key-Worth Cache
As advised by its identify, Key-Worth cache is a way designed to speedup the autoregressive course of by caching and reusing the earlier keys and values, slightly than re-computing them at every decoding step.
Notice that KV cache is usually used solely throughout the inference stage, since in coaching we nonetheless must course of the complete enter sequence in parallel.
KV cache is usually applied as a rolling buffer. At every decoding step, solely the brand new question Q is computed, whereas the Okay and V saved within the cache shall be reused, in order that the eye shall be computed utilizing the brand new Q and reused Okay, V. In the meantime, the brand new token’s Okay and V may even be appended to the cache for later use.
Nevertheless, the speedup achieved by KV cache comes at a price of reminiscence, since KV cache typically scales with batch dimension × sequence size × hidden dimension × variety of heads, resulting in a reminiscence bottleneck when we now have bigger batch dimension or longer sequences.
That additional results in two methods aiming at addressing this limitation: Multi-Question Consideration and Grouped-Question Consideration.
Multi-Question Consideration (MQA) vs Grouped-Question Consideration (GQA)
The determine beneath exhibits the comparability between the unique MHA, Grouped-Question Consideration (GQA) [10] and Multi-Question Consideration (MQA) [9].
The essential thought of MQA is to share a single key and a single worth head throughout all question heads, which might considerably cut back reminiscence utilization however may even impression the accuracy of consideration.
GQA will be seen as an interpolating technique between MHA and MQA, the place a single pair of key and worth heads shall be shared solely by a gaggle of question heads, not all queries. However nonetheless it will result in inferior outcomes in comparison with MHA.
Within the later sections, we are going to see how MLA manages to hunt a steadiness between reminiscence effectivity and modeling accuracy.
RoPE (Rotary Positional Embeddings)
One final piece of background we have to point out is RoPE [11], which encodes positional info instantly into the eye mechanism by rotating the question and key vectors in multi-head consideration utilizing sinusoidal capabilities.
Extra particularly, RoPE applies a position-dependent rotation matrix to the question and key vectors at every token, and makes use of sine and cosine capabilities for its foundation however applies them in a novel method to obtain rotation.
To see what makes it position-dependent, take into account a toy embedding vector with solely 4 parts, i.e., (x_1, x_2, x_3, x_4).
To use RoPE, we firstly group consecutive dimensions into pairs:
- (x_1, x_2) -> place 1
- (x_3, x_4) -> place 2
Then, we apply a rotation matrix to rotate every pair:
the place θ = θ(p) = p ⋅ θ_0, and θ_0 is a base frequency. In our 4-d toy instance, which means that (x_1, x_2) shall be rotated by θ_0, and (x_3, x_4) shall be rotated by 2 ⋅ θ_0.
For this reason we name this rotation matrix as position-dependent: at every place (or every pair), we are going to apply a unique rotation matrix the place the rotation angle is set by place.
RoPE is extensively utilized in fashionable LLMs attributable to its effectivity in encoding lengthy sequences, however as we will see from the above method, it’s position-sensitive to each Q and Okay, making it incompatible with MLA in some methods.
