CROME: Cross-Modal Adapters for Efficient Multimodal LLM