Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation

Parameter-efficient fine-tuning methods have emerged as a promising solution for adapting pre-trained models to various downstream tasks. While these methods perform well in single-task learning, extending them to multi-task learning exacerbates common challenges, such as task interference and negative transfer, due to the limited number of trainable parameters. To address these issues, we introduce progressive task-specific multi-task adaptation, a novel parameter-efficient approach for multi-task learning. This approach introduces adapter modules that are shared in early layers and become increasingly task-specific in later layers. Additionally, we propose a gradient-based approach for computing task similarity and use this measure to allocate similar tasks to the shared adapter modules. Applied to the Swin Transformer and Pyramid Vision Transformer on PASCAL, NYUD-v2, and CelebA, our method outperforms prior parameter-efficient multi-task approaches with fewer trainable parameters.