Showing posts with label Rotary Positional Embedding. Show all posts
Showing posts with label Rotary Positional Embedding. Show all posts

Sunday, March 17, 2024

Use of Long Text Sequences with LLM’s Trained on Shorter Text Sequences - ALiBi & RoFORMER



Training large language models (LLMs) on longer sequences poses challenges in computational resources, model complexity, gradient propagation, and overfitting. These include increased memory requirements due to self-attention mechanisms, longer training times, difficulty in scaling Transformers for very long sequences, challenges in capturing long-term dependencies, risk of vanishing or exploding gradients, and potential overfitting to training data. Solutions like linear biases, RoFormer, and RoPE improve handling of long-range dependencies, enhance model generalization, and incorporate positional information for better performance in NLP tasks. For Example:

Attention with linear Biases

Improved Handling of Long-Range Dependencies. Traditional attention mechanisms struggle with capturing long-range dependencies in text due to the quadratic increase in computational complexity with sequence length. Linear biases help to mitigate this by effectively incorporating positional information, thus enhancing the model’s ability to maintain context over long distances within the text. 


Improved Model Generalization: By more effectively encoding positional information, RoFormer helps LLMs to generalize better across different tasks and datasets. This results in enhanced performance on a wide range of NLP tasks, including text classification, machine translation, and semantic analysis. 
Enhanced Positional Encoding: RoPE uniquely integrates positional information with the token embeddings, preserving the relative distances between tokens. This method enables the model to better understand and utilize the order of words or tokens, which is crucial for many language understanding and generation tasks.

Video Tutorial -1

Video Tutorial -2

Video Tutorial -3

  1. Su, Jianlin, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. "Roformer: Enhanced transformer with rotary position embedding." Neurocomputing 568 (2024): 127063.
  2. Press, Ofir, Noah A. Smith, and Mike Lewis. "Train short, test long: Attention with linear biases enables input length extrapolation." arXiv preprint arXiv:2108.12409 (2021).
  3. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Ɓukasz Kaiser, and Illia Polosukhin. "Attention is all you need." Advances in neural information processing systems 30 (2017).