On Test-Time Scaling for Vision-Language Models

Abstract

Test-time scaling is a paradigm where large models use ad-ditional compute at inference to achieve better performance, withoutchanging model weights. While it has been widely studied for Large Lan-guage Models (LLMs), its applicability to Large Vision-Language Mod-els (LVLMs) remains less explored and analyzed, with limited analysis ofwhether, when, and to what extent these approaches transfer to LVLMs.In this work, we ask a simple but fundamental question: can conven-tional test-time scaling methods developed for LLMs be directly appliedto LVLMs? We present the first comprehensive study of test-time scal-ing for LVLMs, spanning multiple models and model sizes, nine test-timescaling methods, and six diverse benchmarks. Our main findings is that1) different from previous findings, small, well-performing models benefitthe most from test-time scaling, enabling performance improvements ofup to around 30%, reaching large models performance, and often out-performing them, 2) LVLMs lose focus when given more compute thannecessary, and 3) Visual information is encoded early in the reasoningchain, after which the chain is dominated by text-only reasoning andthe contribution of image tokens drops significantly. Finally, we also pro-vide a global and fine-grained analysis on the quality and informationsufficiency of the reasoning chains produced. Overall, our findings andanalysis provide practical guidance and insights into LVLMs and theirdeployment in research and industry.