Skip to content
r/LocalLLaMA · Communities

Gigatoken: A new open source tokenizer ~100x faster than Tiktoken, -500-1000x faster than Huggingface

submitted by /u/Thrumpwart [link] [comments]

Read original