source: PyTorch Blog: Fast, On Device Agentic AI with Muse Glimmer on ExecuTorch
level: technical
Meta introduced Muse Glimmer, an open-weight 30-billion-parameter model distilled from Muse Spark, designed for on-device agentic tasks. It runs on ExecuTorch, which supports NVIDIA GPUs and Macs with Apple silicon. ExecuTorch avoids rewriting models in other languages by exporting directly from PyTorch, handling backend-specific optimizations like Triton on CUDA and Metal on Apple Silicon. This approach simplifies deployment of complex architectures, multimodal inputs, and advanced decoding algorithms such as DFlash speculative decoding.
On an M5 Pro Mac, Muse Glimmer achieved 21.6 tokens per second in solo mode and 33.0 tokens per second with DFlash, a 52.8% speedup without quality loss. The model supports 128K+ token contexts, with only 13 of 52 layers being global and the rest using sliding windows for efficiency. ExecuTorch enables GGUF loading, k-quant execution, and a single model load serving multiple isolated conversations. Current limitations include no video input support, no cross-session prefix sharing, and no continuous batching.
ExecuTorch’s ahead-of-time compilation optimizes the full execution path, not just individual operations, making it suitable for edge devices. The release includes prebuilt PTE artifacts on Hugging Face for text-only and text-plus-image modalities, with and without DFlash. Developers can also build their own PTEs from GGUF checkpoints. The system integrates with agentic frameworks like Pi, enabling local coding agents that use tools for file editing, testing, and iterative reasoning. This approach reduces latency and keeps data on-device.
why it matters: On-device agentic AI reduces cloud dependency, lowers latency, and keeps sensitive data local, enabling responsive assistants for coding and multimodal tasks.
source: PyTorch Blog: Fast, On Device Agentic AI with Muse Glimmer on ExecuTorch