
There is a more recent version of this eprint available. Click here to view it.
Xue, H., Yamaguchi, A. orcid.org/0000-0001-8327-7598 and Aletras, N. (Submitted: 2026) MultiHashFormer: Hash-based generative language models. [Preprint - arXiv] (Submitted)
Abstract
Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.
Metadata
| Item Type: | Preprint |
|---|---|
| Authors/Creators: |
|
| Copyright, Publisher and Additional Information: | © 2026 The Author(s). This preprint is made available under a Creative Commons Attribution 4.0 International License. (https://creativecommons.org/licenses/by/4.0/) |
| Keywords: | Information and Computing Sciences; Language, Communication and Culture; Linguistics |
| Dates: |
|
| Institution: | The University of Sheffield |
| Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
| Date Deposited: | 11 Nov 2026 14:40 |
| Last Modified: | 11 Nov 2026 15:15 |
| Status: | Submitted |
| Identification Number: | 10.48550/arxiv.2606.28057 |
| Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:247681 |
Available Versions of this Item
- MultiHashFormer: Hash-based generative language models. (deposited 11 Nov 2026 14:40) [Currently Displayed]
Download
Filename: 2606.28057v1.pdf
Licence: CC-BY 4.0

CORE (COnnecting REpositories)
CORE (COnnecting REpositories)