Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
arXiv:2607.05475v1 Announce Type: cross Abstract: Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Ou
The proliferation of LLMs and the increasing demand for on-device AI inference are driving immediate research into optimizing performance on mobile hardware, especially NPUs, ahead of their widespread deployment.
This research highlights critical bottlenecks in energy efficiency for mobile LLM inference, directly impacting the feasibility, sustainability, and privacy benefits of ubiquitous on-device AI.
Understanding these hidden efficiency bottlenecks across various frameworks and hardware backends will enable more targeted optimization efforts, accelerating the development of energy-efficient mobile AI systems.
- · Mobile NPU manufacturers
- · On-device AI framework developers
- · Smartphone manufacturers
- · Edge AI companies
- · Inefficient mobile AI frameworks
- · Cloud-based LLM inference services (for some use cases)
- · Device manufacturers with poor NPU optimization
Improved energy efficiency will enable longer battery life for mobile devices running LLMs and expand the range of feasible on-device AI applications.
Enhanced privacy and reduced latency from on-device LLMs could shift user preferences away from cloud-dependent AI services in specific scenarios.
The increased capability of mobile devices to run complex AI models independently could foster new forms of distributed, personalized AI assistants and applications, potentially impacting data center growth for certain AI tasks.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI