LoRA: Low-Rank Adaptation of Large Language Models

Abstract

We propose Low-Rank Adaptation (LoRA), which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. LoRA reduces the number of trainable parameters by 10,000x and the GPU memory requirement by 3x compared to full fine-tuning.

Key Findings

• 1Reduced trainable parameters by 10,000x while maintaining quality
• 2Decreased GPU memory requirements by 3x compared to full fine-tuning
• 3Showed that low-rank updates capture task-specific adaptations effectively
• 4Enabled fine-tuning of large models on consumer hardware
• 5Introduced a technique that is composable and switchable at inference time

Impact & Significance