
There is a more recent version of this eprint available. Click here to view it.
Kumar, S., Yamaguchi, A. orcid.org/0000-0001-8327-7598 and Aletras, N. (Submitted: 2026) When tokenizers fail: Byte-level chunking for zero-shot transfer to low-resource languages. [Preprint - arXiv] (Submitted)
Abstract
Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models bypass this issue by processing raw UTF-8 characters, yet they create a granularity mismatch for word-level tasks in non-Latin scripts. Hierarchical byte-level architectures address this mismatch by grouping bytes into word-aligned chunks. However, these architectures require massive training data and suffer from representational misalignment when paired with frozen subword-based language models. In this paper, we propose an adapted hierarchical network framework that bridges this modality gap without extensive training. Our method initializes byte embeddings directly from the subword representations of a frozen base model. We apply a chunk alignment loss to project dynamically grouped byte chunks toward precomputed subword targets, and interleave lightweight part-of-speech (POS) supervision to guide boundary detection. Experiments across six languages demonstrate that our tokenizer-free approach improves performance for word-level morphological tasks, yielding up to a 13.3% improvement on POS tagging.
Metadata
| Item Type: | Preprint |
|---|---|
| Authors/Creators: |
|
| Copyright, Publisher and Additional Information: | © 2026 The Author(s). This preprint is made available under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. (http://creativecommons.org/licenses/by-nc-sa/4.0/) |
| Keywords: | Information and Computing Sciences; Language, Communication and Culture; Linguistics |
| Dates: |
|
| Institution: | The University of Sheffield |
| Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
| Date Deposited: | 11 Nov 2026 14:57 |
| Last Modified: | 11 Nov 2026 14:57 |
| Status: | Submitted |
| Identification Number: | 10.48550/arxiv.2608.27658 |
| Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:247682 |
Available Versions of this Item
- When tokenizers fail: Byte-level chunking for zero-shot transfer to low-resource languages. (deposited 11 Nov 2026 14:57) [Currently Displayed]
Download
Filename: 2608.27658v1.pdf
Licence: CC-BY-NC-SA 4.0

CORE (COnnecting REpositories)
CORE (COnnecting REpositories)