KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference

Language models (LMs) are enabling a new class of mobile and embedded AI applications, including meeting and video summarization, document analysis, and other multi-input workloads that require long-context reasoning. Running these models locally improves privacy, enables offline use, and lowers serving cost. However, on-device long-context inference is fundamentally constrained by a memory capacity wall: the key-value (KV) cache grows linearly with context length and batch size, quickly exceeding available memory. Existing KV-cache offloading schemes are designed to transfer cache data from GPU memory to CPU memory; however, they are not suitable for embedded and mobile systems, where the CPU and GPU (or NPU) typically share unified memory, and secondary storage offers limited, highly asymmetric I/O bandwidth. We present KVSwap, the first KV-cache management framework for resource-constrained mobile systems that uses storage, rather than CPU RAM, as the KV-cache backing store for long-context inference. KVSwap stores the full cache on disk, uses compact in-memory metadata to predict and preload future accesses, overlaps computation with hardware-aware storage transfers, and optimizes read patterns for device characteristics. Our evaluation shows that across representative LMs and storage types, KVSwap delivers higher throughput under tight memory budgets while maintaining generation quality over existing KV cache offloading schemes.