PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

-->

Computer Science > Computation and Language

arXiv:2607.20327v1 (cs)

[Submitted on 22 Jul 2026]

Title: PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

Authors: Niqi Lyu , Pengtao Shi , Wei Qiu , Jianlin Zhong , Sicong Xia , Jianyao Ma , Yicheng Ding

View a PDF of the paper titled PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference, by Niqi Lyu and 6 other authors

View PDF HTML (experimental)

Abstract: Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Ac

…