HF Daily Papers
· Papers
The Galaxy's Guide to the Tokenizer: A Benchmark for Scientific Foundation Models
Tokenization is central to adapting scientific data for transformer-based foundation models, yet its impact on learned representations remains poorly understood. We compare four tokenization strategies, Affine, AIM, JetFormer, and VQ-VAE, within a unified transformer framework for astronomical imagi