arXiv preprint ·
The Weight Is Over - Interactive Diffusion on Consumer GPUs
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1145/3829339.3847852
- Original
- Read it at arXiv
- Topics
Abstract
On-device inference is booming, but the momentum is almost all in language models. Diffusion pipelines are memory hungry, latency-sensitive, and require orchestrating an embedder, a transformer, a decoder, and often further postprocessing that is not as standardized as LLM inference loops are. We navigate the trade-off between performance, quality, and model footprint to reach as many client devices in the wild as possible. We make three contributions: an embedding translator that maps a small text encoder into a large encoder space to cut weight and latency; a reproducible sweep recipe for navigating the speed/quality/memory triangle in diffusion pipelines; and an interactive on-device image generation editor achieving sub-second TTFI on recent GPUs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Hybrid GPU-CPU Retrieval for Personalized Search at Ultra-Large Scale
Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving
Cross-Platform vs Native Mobile Development: An Empirical Study of Software Quality Trade-offs
From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving
Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.