Reinforcement Learning for Multimodal Diffusion Language Models via Bidimensional Trajectory and Thought Optimization

摘要

Diffusion Large Language Models (dLLMs) have achievedremarkable success in pure language tasks and recently shown promis-ing results in multimodal understanding. However, existing post-trainingmethods for multimodal dLLMs have not yet fully exploited the uniquestructure and inference paradigm of diffusion models, leaving room formore tailored optimization strategies. We present Bi-VRL, a reinforce-ment learning framework specifically designed for multimodal dLLMs,featuring a bidimensional optimization strategy that enables com-prehensive optimization across both reasoning and diffusion timestepdimensions. For the diffusion timestep dimension, our trajectory-leveloptimization learns a value function over the entire diffusion path, en-abling holistic credit assignment across denoising timesteps. For the rea-soning dimension, our thought-level optimization decomposes outputsinto reasoning steps and delivers targeted rewards to enhance logicalsoundness. This core strategy is augmented by a fine-grained multimodalalignment mechanism that systematically evaluates both the logical cor-rectness of reasoning steps and their grounding in visual evidence. Ourframework is architecture-agnostic, successfully applied to models withboth discrete vision tokens (MMaDA) and continuous vision embed-dings (LLaDA-V). Extensive experiments across challenging multimodalreasoning benchmarks demonstrate that Bi-VRL significantly outper-forms strong supervised fine-tuning baselines, establishing effective post-training methodology for multimodal dLLMs. We will release the codeat https://github.com/Gen-Verse/dLLM-RL.