DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

За 30 секунд

Что произошло

DataOrchestra framework tailors data processing to individual examples during LLM pretraining, deciding whether to drop, clean, or keep each chunk.

Почему это важно

Matters for engineers building LLMs who want to improve downstream performance by moving beyond uniform corpus-level data processing strategies.

На что обратить внимание

Paper shows gains across benchmarks but doesn't clarify computational overhead of the orchestrator itself or scalability to massive datasets.