Dans l'actualité
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
arXiv cs.AI · Publié le · 3 min de lecture
En 30 secondes
- Ce qui s'est passé
- PyroDash framework trains small language models to decide token-by-token when to hand off queries to large language models for cost-efficient collaborative inference.
- Pourquoi ça compte
- Relevant for engineers deploying LLM services who need to balance reasoning quality against inference costs in production systems.
- Vigilance
- Results demonstrated only on mathematical reasoning benchmarks; generalization to other domains and real-world deployment constraints remain unvalidated.
Écouter ce résumé
Extrait de l'article
-->
Computer Science > Computation and Language
arXiv:2607.20327v1 (cs)
[Submitted on 22 Jul 2026]
Title: PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
Authors: Niqi Lyu , Pengtao Shi , Wei Qiu , Jianlin Zhong , Sicong Xia , Jianyao Ma , Yicheng Ding
View a PDF of the paper titled PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference, by Niqi Lyu and 6 other authors
View PDF HTML (experimental)
Abstract: Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Ac
Extrait de l'original. Lisez l'article complet à la source.
- llm
- language model
- reasoning
- inference
The Agent Architect
Un pattern, un compromis, une panne de production racontée. Un brief hebdomadaire court pour ceux qui construisent des systèmes agentiques.
Un email par semaine, désinscription en un clic. Votre adresse ne sert qu'à envoyer le brief.