Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs
Not provided
Abstract
This paper introduces OraRL, a method that enhances sample efficiency and scalability in reinforcement learning post-training for video multimodal large language models (MLLMs) by utilizing annotations as oracle rollouts.
Reality Card
OraRL improves the efficiency of reinforcement learning post-training for video MLLMs, achieving significant performance gains while reducing step time compared to existing methods.
OraRL requires just 2.2x the step time of SFT, significantly less than the 4.9x required by GRPO with chain-of-thought.
The integration of high-reward oracles can lead to advantage inversion, complicating the optimization process.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.