Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models