OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

За 30 секунд

Что произошло

OSReward introduces a benchmark and open reward models to evaluate how well vision-language models judge whether computer-using agents completed tasks correctly.

Почему это важно

Matters for engineers building or evaluating autonomous agents that interact with computers, especially those using reinforcement learning with reward signals.

На что обратить внимание

State-of-the-art VLM judges show systematic leniency bias, misclassifying failures as successes. Reliable models are expensive; affordable open alternatives currently underperform commercial options.