Cross-Entropy Is All You Need To Invert the Data Generating Process