Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models