Learn from your own latents and not from tokens: A sample-complexity theory