When tokenizers fail: Byte-level chunking for zero-shot transfer to low-resource languages

Warning

There is a more recent version of this eprint available. Click here to view it.

This is a preprint and may not have undergone formal peer review

Kumar, S., Yamaguchi, A. orcid.org/0000-0001-8327-7598 and Aletras, N. (Submitted: 2026) When tokenizers fail: Byte-level chunking for zero-shot transfer to low-resource languages. [Preprint - arXiv] (Submitted)

Abstract

Metadata

Item Type: Preprint
Authors/Creators:
Copyright, Publisher and Additional Information:

© 2026 The Author(s). This preprint is made available under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. (http://creativecommons.org/licenses/by-nc-sa/4.0/)

Keywords: Information and Computing Sciences; Language, Communication and Culture; Linguistics
Dates:
  • Submitted: 27 August 2026
Institution: The University of Sheffield
Academic Units: The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield)
Date Deposited: 11 Nov 2026 14:57
Last Modified: 11 Nov 2026 14:57
Status: Submitted
Identification Number: 10.48550/arxiv.2608.27658
Open Archives Initiative ID (OAI ID):

Available Versions of this Item

Download

Filename: 2608.27658v1.pdf

Licence: CC-BY-NC-SA 4.0

Export

Statistics