arXiv cs.CV
· Papers
Token-Level Entropy Reveals Demographic Disparities in Large Language Models
arXiv:2501.19337v5 Announce Type: replace-cross Abstract: A name alone measurably reshapes a language model's next-token distribution before a single token is sampled. We measure full-vocabulary Shannon entropy of the next-token distribution across six open-weight model families on 5,760 sentence-completion prompts in