arXiv preprint ·
Shared Context, Separate Engines: Stateful Streaming Inference for Multimodal LLMs on Edge SoCs
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Edge SoCs combine CPUs, GPUs, and NPUs, yet runtimes for multimodal large language models (MLLMs) keep generation on one engine and leave others idle. Streaming workloads allow ingestion and answering to overlap, but backend-local KV state binds queries to the engine that built their history. Stage performance and co-running interference vary, so no single placement wins across SoCs and workloads. \name separates logical history from engine-specific KV state so any language engine can continue a stream. Stateful Streaming Generation overlaps ingestion and answering, while Concurrent Sequence Generation serves ready queries in parallel. Phase-level state continuation lets one engine prefill a query and another decode the answer, freeing the first to resume ingestion. State-Driven Execution Planning (SDEP) chooses paths by predicted completion time and sets each engine's share of queries using arrival rates and measured service times. Across M4 Max, RK3588, and QCS6490, \name beats the strongest single-engine baselines, even when they reuse context. On M4 Max, SDEP cuts live-stream p95 time to first token by $76\%$, with every first token within one second, and raises throughput under backlog by up to $35\%$ over Metal-only execution with context reuse. On two RK3588 streams, SDEP delivers $19.6\%$ higher request throughput than the best single engine with context reuse and $27.2\%$ more than parallel engines without shared state. Stateful Streaming Generation reaches up to $2.48\times$ serial throughput on the same engines. Every answer matches its serial reference.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Dream to Assist: Intent-Conditioned World Models for Shared Vehicle Control
Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning
AdaTutor: Adaptive Tutoring through Pedagogical Strategy Selection
Property-Based Testing for Robust Softmax and Cross-Entropy
Video World Models Need Predictable Latents
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.