GiVA: Gradient-Informed Bases for Vector-Based Adaptation

As model sizes continue to grow, parameter-efficient fine-tuning methods have emerged as a powerful alternative to full fine-tuning. While LoRA has become a widely adopted approach among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these approaches typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces Gradient-Informed Bases for Vector-Based Adaptation (GiVA), a gradient-based initialization strategy for vector-based adaptation. GiVA successfully combines the extreme parameter efficiency of vector-based approaches with training times comparable to LoRA. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. It consistently outperforms existing vector-based adaptation, reducing the rank requirements by a factor of eight ($8\times$), and achieves performance competitive with LoRA.