Reinforcing Video Reasoning with Focused Thinking

摘要

Recent advancements in reinforcement learning, particularlythrough Group Relative Policy Optimization (GRPO), have significantlyimproved multimodal large language models for complex reasoning tasks.However, two critical limitations persist: 1) they often produce unfo-cused, verbose reasoning chains that obscure salient spatiotemporal cues,and 2) binary rewarding fails to account for partially correct answers,resulting in high reward variance and ine!cient learning. In this pa-per, we propose TW-GRPO, a novel framework that enhances visualreasoning with focused thinking and dense reward granularity. Specifi-cally, we employ a token weighting mechanism that prioritizes tokenswith high informational density (estimated by intra-group informationentropy), suppressing redundant tokens like generic reasoning prefixes.Furthermore, we reformulate RL training by shifting from single-choice tomulti-choice QA tasks, where soft rewards enable finer-grained gradientestimation by distinguishing partial correctness. Additionally, we proposequestion-answer inversion, a data augmentation strategy to generatediverse multi-choice samples from existing benchmarks. Experimentsdemonstrate superior performance on several video reasoning and un-derstanding benchmarks. Notably, under identical training settings, ourmethod significantly outperforms the Video-R1, achieving an averageimprovement of 2.6% across four comparable benchmarks, e.g., +3.8%on MVBench, +2.4% on TempCompass, and +2.3% on VideoMME. Ourcodes are available at https://github.com/longmalongma/TW-GRPO.