Phantom Gains: Auditing Self-Improvement Against a Measured Null

За 30 секунд

Что произошло

Researchers identified seven measurement failures in language model self-improvement audits, showing reported gains often vanish when compared against proper frozen controls.

Почему это важно

Engineers evaluating self-training or fine-tuning improvements should care, especially when tracking problem-level gains and losses across model iterations.

На что обратить внимание

Standard practices like single greedy decoding and naive threshold repairs produce false positives on untrained models; proper null baselines from existing replicates are essential.