HF Daily Papers
· Papers
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that con