This is the latest version of this eprint.
Niklaus, J., Yamaguchi, A. orcid.org/0000-0001-8327-7598, Štefánik, M. et al. (9 more authors) (Accepted: 2026) How can we synthesize high-quality pretraining data? A systematic study of prompt design, generator model, and source data. In: Conference On Language Models 3 proceedings. The Third Annual Conference on Language Modeling (COLM 2026), 06-09 Oct 2026, San Francisco, CA, United States. COLM. (In Press)
Abstract
Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase, a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.
Metadata
| Item Type: | Proceedings Paper |
|---|---|
| Authors/Creators: |
|
| Copyright, Publisher and Additional Information: | © 2026 The Author(s). |
| Dates: |
|
| Institution: | The University of Sheffield |
| Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
| Date Deposited: | 18 Nov 2026 13:00 |
| Last Modified: | 18 Nov 2026 13:00 |
| Status: | In Press |
| Publisher: | COLM |
| Refereed: | Yes |
| Related URLs: | |
| Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:247943 |
Available Versions of this Item
-
How can we synthesize high-quality pretraining data? A systematic study of prompt design, generator model, and source data. (deposited 12 Nov 2026 15:58)
- How can we synthesize high-quality pretraining data? A systematic study of prompt design, generator model, and source data. (deposited 18 Nov 2026 13:00) [Currently Displayed]
Download
Filename: 2604.13977v2.pdf

CORE (COnnecting REpositories)
CORE (COnnecting REpositories)