Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B

За 30 секунд

Что произошло

A controlled experiment shows simple repeated sampling outperforms or matches self-refining methods like Self-Refine and Reflexion when token budgets are equal across 1.5B to 7B models.

Почему это важно

Matters for engineers choosing inference strategies for math problems and evaluating whether complex reasoning methods justify their computational cost.

На что обратить внимание

Study limited to two math benchmarks with 150 questions each; results may not generalize to other domains, languages, or task types beyond mathematics.