This is the latest version of this eprint.
Kumar, S., Yamaguchi, A. orcid.org/0000-0001-8327-7598 and Aletras, N. (2026) When tokenizers fail: Byte-level chunking for zero-shot transfer to low-resource languages. In: Lee, H.Y., Dev, S., Li, J.J, and Elliott, D., (eds.) Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 24-29 Oct 2026, Budapest, Hungary. Association for Computational Linguistics, pp. 48638-48651. ISBN: 9798891764590.
Abstract
Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models bypass this issue by processing raw UTF-8 characters, yet they create a granularity mismatch for word-level tasks in non-Latin scripts. Hierarchical byte-level architectures address this mismatch by grouping bytes into word-aligned chunks. However, these architectures require massive training data and suffer from representational misalignment when paired with frozen subword-based language models. In this paper, we propose an adapted hierarchical network framework that bridges this modality gap without extensive training. Our method initializes byte embeddings directly from the subword representations of a frozen base model. We apply a chunk alignment loss to project dynamically grouped byte chunks toward precomputed subword targets, and interleave lightweight part-of-speech (POS) supervision to guide boundary detection. Experiments across six languages demonstrate that our tokenizer-free approach improves performance for word-level morphological tasks, yielding up to a 13.3% improvement on POS tagging.
Metadata
| Item Type: | Proceedings Paper |
|---|---|
| Authors/Creators: |
|
| Editors: |
|
| Copyright, Publisher and Additional Information: | © 2026 Association for Computational Linguistics (ACL). Licensed on a Creative Commons Attribution 4.0 International License. (https://creativecommons.org/licenses/by/4.0/) |
| Dates: |
|
| Institution: | The University of Sheffield |
| Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
| Date Deposited: | 11 Nov 2026 15:13 |
| Last Modified: | 11 Nov 2026 15:39 |
| Status: | Published |
| Publisher: | Association for Computational Linguistics |
| Refereed: | Yes |
| Identification Number: | 10.18653/2026.emnlp-main.2139 |
| Related URLs: | |
| Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:247912 |
Available Versions of this Item
-
When tokenizers fail: Byte-level chunking for zero-shot transfer to low-resource languages. (deposited 11 Nov 2026 14:57)
- When tokenizers fail: Byte-level chunking for zero-shot transfer to low-resource languages. (deposited 11 Nov 2026 15:13) [Currently Displayed]
Download
Filename: 2026.emnlp-main.2139.pdf
Licence: CC-BY 4.0

CORE (COnnecting REpositories)
CORE (COnnecting REpositories)