-
Entropy and cross-entropy: why your loss function is an information measure
Every time a language model predicts the next token, it makes a bet about the future. If it assigns a high probability to the token that actually appears, the bet was good. If it assigns a tiny probab