Local Inference of LLMs on Long Contexts: A Data Systems Perspective

Abstract

Large language models can solve complex problems across a variety of subjects, such as mathematics and coding. For such complex analytical tasks, models with a long context perform better. Serving models with long context is currently mostly done on powerful systems with multiple GPUs, which are too expensive for most users. An alternative is local inference, or serving a model on a user's own machine, which removes per-token costs and rate limits, besides keeping sensitive data private and giving users full control over the models they run. The common understanding, however, is that local LLMs need compression (e.g., quantization) to run on a single GPU. In this paper, we set out to study the limitations that prevent serving long-context models for single users, besides the capacity issue. Rather than characterizing a single model on one GPU, we analyze how different models behave across several GPU generations. This lets us uncover not just individual bottlenecks but broader trends in where inefficiencies arise. Our analysis confirms that, as expected, self-attention dominates the runtime for long-context inference. More unexpectedly, our evaluation reveals that the current structure of self-attention leaves key GPU resources significantly underutilized, and the newest GPU is often the least well served. Many of these findings are fundamentally data management problems, from paging the KV cache to improving cache utilization and synchronization in the GPU, which is why we bring this study to the Data Systems community. To engage this community, we present our findings as a series of experimental observations and formulate recommendations for future work.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Databases

• arXiv preprint · 20 October 2026

  TriGeoBench: A Multimodal Mathematical Geometry Benchmark with Figures and TikZ

  Jing Wang, Yitong Yang, Hao Dou and 6 others
• arXiv preprint · 21 October 2026

  Conflicted Records: Typed Adversarial Contracts for Auditable LLM Output Quality Enforcement

  Kardelen Subasi and Jarosław A. Chudziak
• arXiv preprint · 21 October 2026

  Getting a GRIP on Graph-Based Search for Learned Sparse Retrieval

  Vihan Lakshman, Franco Maria Nardini, Cosimo Rulli and 1 other
• arXiv preprint · 19 October 2026

  MultiCard: Multiple Attribute Cardinality Estimation via Deep Learning

  Mobina Koochaknia, Maryam Ramezani and Mohammad Hasan Shafaat
• arXiv preprint · 22 October 2026

  TT-SAX: A Two-Track Time Series Symbolization for Natural Language Shape Querying

  Zhao Tan, Xianglong Bao, Zhipeng Xi and 1 other

All new papers in Databases