X · @huggingface
· X / Twitter
RT Daniel van Strien: datatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs — just shipped 0.10.0! – JobsPipelineExecutor: …
RT Daniel van Striendatatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs — just shipped 0.10.0!- JobsPipelineExecutor: run pipelines on @huggingface Jobs: fan-out, multi-stage dependencies, retries, resume. No Slurm cluster needed- HF storage buckets as a DataFolder: read, write, and log strai