Yamaguchi, A. orcid.org/0000-0001-8327-7598, Mi, M. and Aletras, N. orcid.org/0000-0003-4285-1965 (2026) Enhancing linguistic competence of language models through pre-training with language learning tasks. In: Liakata, M., Moreira, V.P., Zhang, J. and Jurgens, D., (eds.) Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 64th Annual Meeting of the Association for Computational Linguistics, 02-07 Jul 2026, San Diego, California, United States. Vol. 2. Association for Computational Linguistics, pp. 316-336. ISBN: 9798891763913.
Abstract
Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction. Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation. Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates their acquisition, while maintaining competitive performance on general reasoning tasks.
Metadata
| Item Type: | Proceedings Paper |
|---|---|
| Authors/Creators: |
|
| Editors: |
|
| Copyright, Publisher and Additional Information: | © 2026 ACL. This paper is licensed on a Creative Commons Attribution 4.0 International License. https://creativecommons.org/licenses/by/4.0/ |
| Keywords: | Language Studies; Linguistics; Cognitive and Computational Psychology; Language, Communication and Culture; Psychology; Basic Behavioral and Social Science; Behavioral and Social Science; Quality Education |
| Dates: |
|
| Institution: | The University of Sheffield |
| Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
| Date Deposited: | 22 Oct 2026 09:18 |
| Last Modified: | 22 Oct 2026 09:18 |
| Status: | Published |
| Publisher: | Association for Computational Linguistics |
| Refereed: | Yes |
| Identification Number: | 10.18653/v1/2026.acl-short.27 |
| Sustainable Development Goals: | |
| Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:246955 |
Download
Filename: 2026.acl-short.27.pdf
Licence: CC-BY 4.0


CORE (COnnecting REpositories)
CORE (COnnecting REpositories)