Phantom Gains: Auditing Self-Improvement Against a Measured Null

In 30 Sekunden

Was passiert ist

Researchers identified seven measurement failures in language model self-improvement audits, showing reported gains often vanish when compared against proper frozen controls.

Warum es zählt

Engineers evaluating self-training or fine-tuning improvements should care, especially when tracking problem-level gains and losses across model iterations.

Achtung

Standard practices like single greedy decoding and naive threshold repairs produce false positives on untrained models; proper null baselines from existing replicates are essential.