Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

Abstract

Reinforcement learning (RL) has become a standard tech-nique for post-training diffusion-based image synthesis models, as itenables learning from reward signals to explicitly improve desirable as-pects such as image quality and prompt alignment. In this paper, wepropose an online RL variant that reduces the variance in the modelupdates by sampling paired trajectories and pulling the flow velocityin the direction of the more favorable image. Unlike existing methodsthat treat each sampling step as a separate policy action, we considerthe entire sampling process as a single action. We experiment with bothhigh-quality vision language models and off-the-shelf quality metrics forrewards, and evaluate the outputs using a broad set of metrics. Ourmethod converges faster and yields higher output quality and promptalignment than previous approaches.