Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

Abstract

Graphical User Interface (GUI) grounding is essential for au-tonomous agents to map natural language instructions to precise screencoordinates. However, existing supervised fine-tuning and reinforcementlearning methods are constrained by the high cost of annotation, creatinga scalability bottleneck. In this paper, we introduce a label-free test-timetraining paradigm driven by two key insights: (1) confidence patterns incoordinate tokens are a better indicator than full-sequence confidence,and (2) in sparse GUI coordinate spaces, negative samples offer more re-liable learning signals than potentially noisy positive ones. We first pro-pose Confidence-Anchored Learning (CAL), which utilizes coordinate-token confidence to filter pseudo-labels and assign distance-based binaryrewards. Building on this, we develop Confidence-Anchored NegativeLearning (CANL), which exclusively optimizes the model using negativesamples to bypass the risks of incorrect positive samples. Experimen-tal results demonstrate that CANL-7B achieves 92.1% on ScreenSpot-V2. On more challenging ScreenSpot-Pro, CANL-7B reaches 33.8%, an8.9% absolute improvement over the base model. Our findings establishcoordinate-token confidence as a powerful alternative to manual annota-tions for scalable GUI agent development.