Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos

Abstract

We introduce Synthetic Visual Genome 2 (SVG2), alarge-scale panoptic video scene graph dataset. SVG2 contains over 636Kvideos with 6.6M objects, 52.0M attributes, and 6.7M relations, pro-viding an order-of-magnitude increase in scale and diversity over priorspatio-temporal scene graph datasets. To create SVG2, we design a fullyautomated pipeline that combines multi-scale panoptic segmentation,online–offline trajectory tracking with automatic new-object discovery,per-trajectory semantic parsing, and GPT-5-based spatio-temporal re-lation inference. Human verification of SVG2 annotation accuracy con-firms its reliability (objects: 93.8%, attributes: 88.3%, relations: 85.4%).Building on this resource, we train TraSeR, a trajectory-groundedvideo scene graph generation model. TraSeR augments VLMs with atrajectory-aligned token arrangement mechanism and new modules: anobject-trajectory resampler and a temporal-window resampler to convertraw videos and panoptic trajectories into compact spatio-temporal scenegraphs in a single forward pass. The temporal-window resampler bindsvisual tokens to short trajectory segments to preserve local motion andtemporal semantics, while the object-trajectory resampler aggregates en-tire trajectories to maintain global context for objects. On PVSG, VIPSeg,VidOR, and SVG2test , TraSeR outperforms the strongest open-sourcebaselines by 15∼20 points in relation detection, 20∼40 points in objectprediction, and 13 points in attribute prediction. It also surpasses GPT-5by 13 points in object prediction and 3 points in attribute prediction.When TraSeR’s generated scene graphs are sent to a VLM for videoquestion answering, it delivers a +1.5∼4.6 absolute accuracy gain overusing video alone or video augmented with Qwen2.5-VL’s generated scenegraphs, demonstrating the utility of explicit spatio-temporal scene graphsas an intermediate representation5 .