In the news
Toward a Theory of Evaluability
Apple Machine Learning Research · Published · 3 min read
In 30 seconds
- What happened
- Apple researchers formalized evaluability theory for black-box models when test data may have been used during training, characterizing when accurate evaluation remains feasible.
- Why it matters
- Matters for engineers evaluating large foundation models where held-out test set assumptions no longer hold, especially in production settings with data reuse.
- Watch out
- Results show accurate evaluation is only feasible when distributions are close to small support; query access helps but adds complexity to evaluation protocols.
- foundation model
- eval
The patterns behind this
Each one covers how the technique works, when it earns its cost, and where it breaks.
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.