source: kdnuggets: 5 ways small language models are powering next-gen agents
level: technical
most agent work is narrow and repetitive, not broad or creative. nvidia research argues that small language models are better suited for these specialized tasks, like parsing commands or returning json. they are more reliable and economical than large general-purpose models for such jobs. large models still handle novel reasoning, but slms are becoming the default for routine agent operations.
slms can run directly on devices like phones and laptops, avoiding cloud latency. hardware advances, such as apple's a19 pro and m5 max chips, enable real-time inference. quantization reduces model size significantly, with a phi-4-mini model compressed to 1.2 gb while keeping over 95% performance. tools like ollama and microsoft's phi family support on-device deployment, useful when network connectivity is unreliable.
fine-tuning slms for tool calling achieves high accuracy at low cost. a fine-tuned slm reached 77.55% on the toolbench evaluation, outperforming larger prompted models. with 1,000 to 5,000 examples per tool, accuracy can exceed 95%. in heterogeneous systems, small models handle routine tasks while large models manage complex reasoning, cutting costs by up to 10 times and reducing latency. on-device slms also keep sensitive data local, essential for regulated industries like healthcare, where cloud use is often prohibited.
why it matters: using small language models in agents reduces costs, speeds up responses, and keeps data private, making ai practical for repetitive tasks and sensitive environments.
source: kdnuggets: 5 ways small language models are powering next-gen agents