ProAct: Agentic Lookahead in Interactive Environments

摘要

LLM agents often fail in long-horizon tasks due to com-pounding simulation errors. We propose ProAct, a two-stage frameworkfor internalizing foresight. First, Grounded LookAhead Distillation (GLAD) performs supervised fine-tuning (SFT) on search-derived trajectories,compressing complex search trees into concise causal reasoning chains.This allows agents to learn lookahead logic without inference-time com-putational overhead. Second, we introduce Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator for policy-gradient al-gorithms (PPO/GRPO). By using lightweight rollouts to provide low-variance value signals, MC-Critic enables stable optimization without ex-pensive model-based approximations. Experiments on stochastic (2048)and deterministic (Sokoban) tasks show ProAct significantly boosts plan-ning accuracy. Notably, a 4B model trained with ProAct outperformsopen-source baselines and rivals state-of-the-art closed-source models,demonstrating strong generalization to unseen environments. The codesand models are available at https://github.com/GreatX3/ProAct.