Pretraining 数据构建
介绍预训练阶段的数据构建方法与流程
- 参考资料
- HuggingFace FineWeb 数据集构建手册：FineWeb: decanting the web for the finest text data at scale - a Hugging Face Space by HuggingFaceFW
- HuggingFace FineWeb 数据集构建手册的论文：The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- 处理 PDF 获取数据：FinePDFs: Liberating 3T of the finest tokens from PDFs - a Hugging Face Space by HuggingFaceFW
- 扩展不同语言：Scaling FineWeb to 1000+ languages: Step 1: finding signal in 100s of evaluation tasks - a Hugging Face Space by HuggingFaceFW
- Developing an LLM: Building, Training, Finetuning
这篇文档有帮助吗？