arXiv preprint ·
BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1145/3845814.3850028
- Original
- Read it at arXiv
- Topics
Abstract
The rapid scaling of large language models (LLMs) has unlocked remarkable capabilities across various domains. However, deploying LLMs on mobile devices remains challenging because of their high computational and memory requirements. Low-bit quantization (e.g., 4-bit weights, 8-bit activations) can reduce model size while preserving accuracy, but its practical efficacy on mobile GPUs has been limited so far. In this paper, we identify that the performance gap stems primarily from frequent data type conversions between quantized and floating-point values, and dynamic and inefficient memory access patterns caused by the growing Key-Value (KV) cache during autoregressive decoding. We introduce BitLLM, a mobile inference engine that addresses type conversion overhead and suboptimal memory accesses in quantized LLMs. Our approach features an adaptive layout and packing strategy for the KV cache that ensures efficient memory access throughout LLM decoding. This is coupled with an integer-native execution pipeline that defers and overlaps type conversion outside the compute hot path, keeping the critical MatMul stages in the integer domain. Implemented on off-the-shelf mobile GPUs, BitLLM outperforms leading engines, i.e., LLAMA.CPP, MNN-LLM, LiteRT-LM, and MLC-LLM, achieving average speedups of 1.78$\times$, 1.73$\times$, 1.58$\times$, and 1.47$\times$, respectively, across various models and quantization levels. Under long-context generation, BitLLM reduces average energy consumption (per token) by up to 88.9%.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Hardware Architecture
DIMWIT: Directly Invoking Models Without Intensive Tuning
Ripple: Deadline-Aware CPU-GPU Sharing for Embodied Inference
Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading
GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading
LLM-PowerTrace: A Dataset of Time-Resolved GPU Power Traces for AI Compute
Get the few papers worth your time in Hardware Architecture every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.