Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking

摘要

Although autoregressive (AR) models have demonstrated re-markable success in image generation, extending these models to layout-conditioned generation remains challenging due to the sparse natureof layout conditions and the risk of feature entanglement. We presentStructured Masking for AR-based Layout-to-Image (SMARLI), a novelframework that effectively integrates spatial layout constraints into theAR generation process. To equip AR models with layout control, a struc-tured masking strategy is applied to the attention computation to governthe interaction among the global prompt, layout, and image tokens. Thisdesign prevents the misassociation of different regions with their corre-sponding descriptions while enabling the sufficient injection of layoutconstraints into the generation process. To alleviate the exposure bias ofAR models and further enhance generation quality and layout accuracy,we incorporate a Group Relative Policy Optimization (GRPO) post-training scheme. We adapt it to the next-set-based paradigm and intro-duce a specifically designed layout reward, which is coordinated with animage quality reward to guide policy optimization in a balanced manner.Experimental results demonstrate that SMARLI seamlessly integrateslayout tokens with text and image tokens without compromising genera-tion quality, and the proposed masking strategy and post-training schemecan also be transferred to standard next-token-based AR models. Theproposed framework achieves superior layout control while maintainingthe structural simplicity and generation efficiency of AR models.