paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Stories

Yang Yang, Boyun Xu, Shaofeng Liang, Yun Han, Zining Zhong, Songning Lai, Kaishen Yuan, Yutao Yue

Large language models can now generate fluent and complete stories, yet many outputs still feel formulaic and unnatural because of cliches, over-explanation, linear causal progression, and stereotyped endings, an immediately recognizable AI flavor. Existing detection and evaluation methods often stop at source labels or holistic scores, while revision methods typically target predefined issues through localized edits, limiting their ability to support multiple plausible revision strategies or guide story-wide changes in information release, causal organization, and ending treatment. We introduce CraftAlign, a framework that aligns AI stories with the craft of human storytelling by both assessing Human/AI writing patterns and providing revision guidance. CraftAlign comprises two learned modules and an inference-time guidance pipeline. A feature estimator built on Qwen3.5-9B predicts 304 explicit writing features spanning style and narrative. A class-conditional energy model scores the resulting feature configuration against Human and AI writing patterns, conditioning on the original writing prompt when available. At inference time, CraftAlign applies schema-valid structured perturbations, selects changes that move the feature configuration toward the Human writing pattern, and converts them into natural-language guidance for a separate editor to rewrite the full story. Experiments show that CraftAlign accurately distinguishes Human and AI writing patterns and that its guidance outperforms revision baselines across editors and in a human study.

arXiv

CRAFTS: Collaborative Role-Adaptive Fine-Tuning of LLM Agents for Chemical Process Simulation

Ziyun Zhang, Yuxin Lin, Eldin Wee Chuan Lim, Xinghao Ding

Constructing an executable chemical-process model remains manually intensive. Chemical engineers translate underspecified requests into coupled decisions about unit operations, thermodynamics, streams, specifications, degrees of freedom (DoF), initialization, solver repair, and optimization; one error can invalidate the model. CRAFTS mirrors the staged workflow of chemical engineers by decomposing simulation building into bounded subtasks assigned to seven bounded roles, with deterministic IDAES/Pyomo gates between stages. Given a natural-language request, process flowsheet diagram (PFD) evidence, and curated chemical-engineering knowledge, Input Understanding and Intent recover requirements, constraints, and process semantics; visual, topology, and specification specialists translate them into typed simulator contracts; and Debug and Optimization support bounded repair and eligible optimization. Fine-tuning is applied to the three schema-critical visual, topology, and specification roles, while the remaining roles use untuned Qwen. The resulting VisualGraphIR, TopologyIR, SpecIR, BuildPlan, and SolveReport expose unit, port, thermodynamic, numerical, and execution decisions. Compatible constructors, property packages, and runners are attached only after semantic artifacts pass engineering gates. We introduce OpenIDAES-450, a 450-case IDAES process- simulation dataset, and evaluate the complete seven-role LangChain/LangGraph workflow through solve and eligible optimization on its frozen 82-case held-out split. CRAFTS completes the prescribed validation and execution contract for for 91.5% of cases and achieves unit, stream, and directed-connection F1 scores of 0.815, 0.791, and 0.782. These results demonstrate the effectiveness of role specialization, typed intermediate representations, and deterministic engineering gates for reliable automated process-model construction.

arXiv

Imprecise Belief Fusion Improves Multi-agent Social Learning

Zixuan Liu, Jonathan Lawry, Michael Crosscombe

In social learning, agents learn not only from direct evidence but also through interactions with their peers. We investigate the role of imprecision in such interactions and ask whether it can improve the effectiveness of the collective learning process. To that end we propose a model of social learning where beliefs are equivalent to formulas in a propositional language, and where agents learn from each other by combining their beliefs according to a fusion operator. The latter is parametrised so as to allow for different levels of imprecision, where a more imprecise fusion operator tends to generates a more imprecise fused belief when the two combined beliefs differ. In this context we describe both difference equation models and agent-based simulations of social learning under a variety of conditions and with different initial biases. The results presented suggest that for populations with a strong initial bias towards incorrect beliefs some level of imprecision in fusion can improve learning accuracy across a range of learning conditions. Furthermore, such benefits of imprecision are consistent with a stability analysis of the fixed points of the proposed difference equation models.

arXiv

High-Stakes Decisions with Language Models: Insights from Emergency Triage

Khurram Yamin, Christopher Kelly, Bryan Wilder, Eric Horvitz

High-stakes decisions under uncertainty, such as medical emergency triage, require more than accurate predictions. They depend on estimating the likelihood of alternative outcomes while explicitly weighing the consequences of different actions, principles that have long formed the foundation of medical diagnosis and decision making. Yet language models are increasingly used for high-stakes clinical recommendations without explicit specification of the utilities governing these decisions. Here we show that emergency triage with language models can be understood within a probabilistic decision framework, providing a case study of a broader decision-analytic paradigm for steering, evaluating, and deploying language models in high-stakes settings. Using clinical vignettes from a structured evaluation of a consumer triage system, we analyze recommendations for treatment under alternative utility functions that specify the relative costs of missed emergencies and unnecessary escalation. We find that capable language models adjust recommendations in response to stated utilities, revealing that the same underlying predictions can support markedly different decision policies. These findings show that effective deployment depends not only on improving predictions but also on making decision objectives explicit. More broadly, they suggest that language models for high-stakes applications should be understood and evaluated as probabilistic decision systems whose recommendations depend jointly on predictive performance and explicit utilities.

arXiv

402Pilot: An x402 Decision Layer for Autonomous Agent Micropayments

Yin Li, Yanbo He, Boo-Ho Yang, Rav Lawana, Ziyue Li, Wei Zeng, Jing Tang, Fugee Tsung

Programmable-payment protocols such as x402 enable per-request micropayments, but they do not determine which payable service an autonomous agent should buy under a finite wallet. We formulate this buyer-side problem as agent-native payment decision-making: contextual provider selection under wallet pressure, chosen-only paid feedback, and changing market conditions. We propose 402Pilot, a protocol-agnostic buyer-side decision layer between autonomous agents and payment execution that implements purchasing policies for selecting among payable providers. We instantiate it with PA-DCT, a payment-aware discounted contextual Thompson-sampling policy that adapts purchasing decisions under wallet pressure while learning from post-payment feedback. To evaluate buyer-side payment policies, we introduce 402Pilot-Bench, a frozen-replay benchmark spanning 823 tasks, five heterogeneous provider pipelines, and three market regimes, each evaluated over 30 paired seeds. PA-DCT achieves the strongest fixed-wallet adaptive trade-off among non-oracle policies: it maintains competitive service quality while spending only 39 to 43 percent of the wallet and reallocates spending as market conditions change. It attains the best non-oracle PA-gap/T under the price shock and the best mean and worst-case ranks across the nine scenario-metric combinations of quality, ROI, and PA-gap/T. Comparisons with learning baselines and component ablations further support the effectiveness and design of the proposed decision policy. These results suggest that programmable payment must be complemented by buyer-side decision-making capable of learning service value and adapting purchasing decisions accordingly.

arXiv

Context Compaction Theory

Hayder Tirmazi, Sam Markelon, Allison Bishop, Michael Mitzenmacher

Large Language Models (LLMs) have a bounded context window. The context window is the maximum input size an LLM can consume for a single inference. AI agents rely on a process called context compaction to fit their state within the context window when calling an LLM. Despite its ubiquity, context compaction has received essentially no formal analysis. In this paper, we initiate a formal study of context compaction. We first introduce a framework consisting of two games that capture the two algorithmic strategies for context compaction used by contemporary AI agents in practice. The Context Selection Game models context compaction algorithms that select a subset of an agent's accumulated state to retain. The Context Generation Game models context compaction algorithms that summarize an agent's state by an arbitrary message of bounded length. We then prove an equivalence between the Context Generation Game and one-way communication complexity. The minimum context compaction budget for answering a set of queries within a target error is equal to the one-way communication complexity of the induced communication problem at the same error. Known bounds from communication complexity therefore transfer directly to context compaction. We also show that the Context Selection Game corresponds to a restricted class of one-way communication protocols. Any gap between selection and generation is therefore a gap between two classes of communication protocols. We prove that there exists a set of queries for which generation needs strictly less budget than selection. The equivalence between the Context Generation Game and one-way communication also lets us measure how well a deployed context compaction algorithm performs on a query relative to the optimal strategy. As an example, we present a case study that evaluates Anthropic's context compaction endpoint on set membership queries.

arXiv

Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Models

John Scoville, Shengzhuang Chen, Yejin Bang, Stefan Winzeck, Jonathan Richard Schwarz

Recent meta-reasoning frameworks improve LLM reasoning by wrapping chain-of-thought generation in an iterative control loop, allowing more effective backtracking, termination of reasoning loops, and injection of promising reasoning patterns, among other strategy adjustments. Despite promising results, methods often rely on backward-looking reward functions, utilize coarse search actions, or require additional reasoning controller training requiring many-shot supervision. We introduce Cognitive Demand Steering (CDS), a training-free meta-reasoning framework equipped with residual demand assessment: at each step, an LLM-based progress evaluator characterizes the residual reasoning required to arrive at a solution rather than merely evaluating the previous step. This allows a meta-controller to select reasoning interventions comprising both general-purpose exemplars and actions (e.g., general guidance for quantitative reasoning) that directly tackle this forward-looking demand signal. This shift eliminates the need for any trained component while enabling zero-shot transfer across models and tasks with no adaptation. Rather than relying on coarse characterizations, we employ cognitive scales to both design interventions as well as profile initial problem complexity and residual demand signal over 16 dimensions motivated by cognitive science (e.g., attention and scan, learning and abstraction, spatio-physical reasoning), giving the controller a fine-grained vocabulary for diagnosing. Averaged across three frontier LLMs and six reasoning benchmarks, CDS improves accuracy by $21.9\%$ over direct calls and $9\%$ over standard CoT reasoning, with the largest gains on difficult mathematics and coding tasks.

arXiv

Auditing Semantic Gains in Sequential Recommendation: A Lightweight Recovery Test

Kong Wang, Zhongke He, Xiang Chen, Hongwei Zeng, Kai Deng, Long Wang, Kehua Yang

Recent semantic and generative-retrieval recommenders report substantial improvements over ID-only sequential baselines, but it remains unclear whether these gains arise from language-model reasoning, semantic-ID generation, end-to-end semantic architectures, stronger offline item representations, or complementary semantic and collaborative signals. We investigate this attribution ambiguity through LIME-Rec, a lightweight and auditable recovery test. LIME-Rec combines three independent experts: a SASRec sequential expert, an ItemCF co-occurrence expert, and a semantic expert based on frozen BAAI/bge-base-en-v1.5 item embeddings. Their full-catalog scores are normalized per user and combined through auditable score-level fusion followed by bounded history calibration. The fusion gate and calibration head are fitted on validation data only, require no serving-time language-model inference, and keep each expert contribution separately inspectable. On Amazon Beauty, Toys, and Sports, LIME-Rec achieves R@10 scores of 0.0996, 0.1105, and 0.0593, outperforming the strongest comparison baseline by 7.0%-12.0%. Three-expert fusion without history calibration consistently outperforms calibrated SASRec, showing that calibration alone does not explain the recovery. Randomly permuting item-text embeddings across item IDs reduces R@10 by 13.6%-17.5%, indicating that the gains depend on genuine item-text correspondence rather than additional representation capacity. These results suggest that lightweight recovery from offline item representations and transparent fusion should be ruled out before improvements are attributed to serving-time language modeling, semantic-ID generation, or heavier semantic machinery.

arXiv

Learning What to Remember and What to Internalize in LLM Self-Evolution via Adaptive Memory-Parameter Coordination

Tianyun Ji, Zhenya Huang, Jiayu Liu, Zirui Liu, Yu Su, Hongbin Pei

Large language model agents increasingly operate in dynamic environments where tool interfaces, APIs, and user requirements change after deployment. Existing self-evolution methods mainly follow two paradigms: harness-based approaches, which externalize feedback into editable memories or skills for rapid adaptation, and parameter-based approaches, which internalize experience into model parameters for deeper capability improvement. However, using either mechanism alone creates a trade-off between flexibility and performance. This paper asks how an agent can coordinate both channels to achieve robust self-evolution. We present COVE, a unified agent self-evolution framework that combines harness-based and parameter-based learning through task-aware routing, stage-aware scheduling, and knowledge optimization. Through this design, COVE treats self-evolution not as indiscriminate accumulation of experience, but as a coordinated process that matches tasks and knowledge types to appropriate learning mechanisms. Experiments across multiple task categories show that COVE outperforms single-channel evolution strategies, demonstrating more robust and efficient improvement under changing environments.

arXiv

CT-PrepAgent: Bounded Policy and Controlled Execution for Adaptive CT Data Preparation

Xiaolin Fan, Yue Pei, Yingying Zhang, Haogang Zhu

Heterogeneous computed tomography (CT) acquisitions and diverse downstream task requirements limit the transferability of fixed data preparation workflows across data sources and tasks. Existing approaches typically rely on manually designed or dataset-specific rules, making it difficult to accommodate changes in acquisition conditions and analytical objectives without manual intervention. Large language model (LLM)-based agents have shown promise for automating medical workflows, yet their potential for adaptive CT data preparation remains largely unexplored. To bridge this gap, we propose CT-PrepAgent, which enables adaptive CT data preparation through a bounded policy and controlled deterministic execution. Deterministic inspection constructs structured data--task profiles, from which a policy decides an eligible DICOM series or predefined preprocessing profile, while the controlled execution flow guards, resolves, executes, and verifies the decision with bounded recovery when enabled and safe quarantine otherwise. Across three public CT segmentation tasks, CT-PrepAgent derived data-task adaptive preprocessing decisions and achieved the highest macro-average Dice. On two private raw-DICOM cohorts, CT-PrepAgent increased verified output yield from 61.7\% to 70.0\% and yielded similar registration metrics on common verified outputs. Controlled fault and replay tests validate bounded recovery, safe quarantine, and policy-free replay under tested fault and drift settings.

arXiv

Perspectives on Tsallis Statistics for Artificial Intelligence

Kleyton Da Costa, Bernardo Modenesi

Tsallis statistics generalizes Boltzmann-Gibbs statistical mechanics through a single real parameter $q$ that controls the weight assigned to rare and frequent events. Originally proposed to describe physical systems with long-range correlations, multifractal geometry, and heavy-tailed fluctuations, the framework has become a recurring ingredient in modern artificial intelligence (AI): it underlies sparse attention mechanisms (\textsc{sparsemax} and $α$-\textsc{entmax}), maximum-entropy reinforcement learning with controllable exploration, robust and heavy-tailed probabilistic models, and a family of generalized loss functions and regularizers. This paper offers a structured perspective on where Tsallis statistics meets AI. We first review the mathematical core: $q$-entropy and its variational (maximum-entropy) foundation, the $q$-exponential and $q$-logarithm, the $q$-central limit theorem, $q$-Gaussian distributions, and their dynamical origin in superstatistics, emphasizing the properties that matter for machine learning. We then survey applications across softmax generalization, reinforcement learning, sequential and graph neural models, generative and probabilistic modeling, loss design, and optimization, extracting the recurring design pattern in each case: a tunable interpolation between dense/uniform and sparse/peaked behavior governed by $q$. We further argue that the heavy-tailed weight spectra and gradient-noise statistics empirically observed in deep networks are themselves nonextensive signatures, placing modern learning dynamics within the scope of $q$-statistics. Finally, we discuss methodological pitfalls, the relationship to information geometry and $q$-exponential families, and open directions, arguing that $q$ should be treated as a learnable inductive bias rather than a fixed hyperparameter.

arXiv

Reputation-driven Cooperation in Lattice-based Decentralized Federated Learning through Evolutionary Game Theory

Phuc Hoang Truong Huynh, Dung Tran Vinh, Khoa Duc Anh Lam, An Nghiem Nguyen Truong, Uyen Nha Tran Bui, Khang Nguyen Dinh, Bao Nguyen Le Gia, Minh Le Nguyen Nhat, Manh Hong Duong, The Anh Han, Thi Ai Thao Nguyen, and Le Hong Trang

Decentralized Federated Learning (DFL) has emerged as an optimal privacy-preserving solution; however, it remains vulnerable to opportunistic behaviors due to the absence of a central coordinator. While Evolutionary Game Theory (EGT) serves as a powerful framework for analyzing such behaviors, existing studies often assume that agents possess perfect rationality and maintain static strategies. To address these limitations, this paper proposes a novel EGT framework designed to analyze strategic evolution and enhance overall system performance. The primary contributions of this work are threefold: First, we model peer-to-peer (P2P) interactions on a lattice network structure under the assumption of bounded rationality. Second, we formulate a comprehensive payoff matrix incorporating training costs, communication overhead, and cooperative rewards, while tailoring a strategy update rule that captures spatial propagation dynamics. Third, we integrate a reputation-based reward-and-punishment mechanism to effectively deter free-riding behaviors. Simulation results demonstrate that the framework significantly outperforms the baseline. Specifically, it increases average accuracy from approximately 70% to 82%, elevates cooperation frequency to approach 100% (compared to below 5% in the baseline), and drops accuracy variance from around 0.40 to 0.002, thereby accelerating uniform convergence and ensuring system stability.

arXiv

Co-evolution of social reward and punishment under institutional interventions

Van An Nguyen, Vuong Khang Huynh, Hoai Thuong Nguyen, Duc Tin Duong, An Nguyen Gia, Tat Kien Nguyen, Huu Loi Bui, My Nguyen Tra, Ho Nam Duong, Ba Thanh Phan, Thanh Vo, Dinh Anh Trung Hoang, Adeela Bashir, Zhao Song, Manh Hong Duong, Le Hong Trang, The Anh Han

We investigate how peer and institutional incentives jointly shape the evolution of cooperation, social welfare, and enforcement efficiency in social dilemmas. In a Prisoners Dilemma with four strategies, unconditional cooperators (C), defectors (D), social punishers (SP), and social rewarders (SR), we allow decentralised peer incentives and centralised institutional incentives to act simultaneously, with the institution able to reward or punish any subset of strategies. In infinite well-mixed populations, we analyse the resulting four-strategy replicator dynamics, and in structured populations we use agent-based simulations on square lattices to study spatial effects and network reciprocity. Intervention schemes are evaluated by equilibrium states and evolutionary flow for infinite well-mixed populations, by cooperation levels and social welfare for structured populations, defined as aggregate population payoff net of institutional cost. We find that peer punishment most strongly promotes cooperation, whereas peer reward is more beneficial for social welfare. Institutionally rewarding peer incentive strategies substantially improves both cooperation and welfare, while subsidising unconditional cooperators has little impact. Under institutional punishment, directly penalising defectors is the only consistently effective policy; punishing peer incentive strategies dismantles decentralised incentives, reduces cooperation, and harms social welfare, showing that maximising cooperation does not necessarily optimise overall societal benefit. Our findings provide design principles for institutions seeking to balance cooperation promotion with welfare maximisation.

arXiv

Talking to Digital Twins: Selective Disclosure and Belief Measurement in Financial Social Media

Boone Bowles, Raymond Duch, Sorin Sorescu

Social media affect financial markets, but public posts by financial media personas are voluntary disclosures. What is not disclosed is therefore usually unobserved. We address this measurement problem by conducting repeated, real-time interviews of "digital twins" built from monitored finfluencers' X accounts under a fixed protocol. The interviews recover stock-level public-persona belief proxies even when no public recommendation is made. Because the interviews are generated and archived before the relevant return windows, the design avoids the look-ahead bias that arises when LLMs are queried ex post. The evidence shows that information obtained from these digital-twin interviews predicts the cross section of large-cap stock returns in the expected direction. Repeated real-time interviews therefore show how selective disclosure can be turned into measurable panels of market views.

arXiv

The Graph Language: How Knowledge Graphs Speak to Large Language Models

Giuseppe Pirrò

Large Language Models (LLMs) excel at reasoning but benefit from grounding provided by Knowledge Graphs (KGs). However, integrating these paradigms is challenging. We introduce GRALAN, which enables KGs to speak directly in the LLM's semantic space through relational tokens that preserve graph structure. GRALAN-s trainable language mediator generates structured tokens for any frozen LLM, creating a foundation for knowledge-intensive applications. We demonstrate its effectiveness in question-answering by re-framing the task as entity classification over question-focused subgraphs. Experiments show that GRALAN significantly outperforms existing methods, particularly on complex multi-hop reasoning tasks, establishing a new paradigm for KG-LLM integration that maintains structural fidelity while leveraging LLMs' reasoning capabilities.

arXiv

Hybrid Lagrangian-Eulerian Model for Lagrangian Fluid Simulation

Ruoyan Li, Wei Wang, Yizhou Sun

Pure Lagrangian neural simulators offer geometric flexibility and exact advection, making them well-suited for modeling moving domains and free surfaces. However, the absence of a fixed global reference frame introduces two severe limitations: a spatial bottleneck, in which model capacity is wasted on uniform regions because the dense particle neighborhoods required for stable gradients are applied indiscriminately, and rapid temporal drift, caused by purely local message passing that lacks a global anchor. Inspired by classical hybrid numerical solvers, we propose a Hybrid Lagrangian-Eulerian neural simulator that augments Lagrangian dynamics with an Eulerian representation. To address the spatial bottleneck, we introduce adaptive downsampling that eliminates kinematic redundancy, preserving micro-scale details on particles while aggregating compressed features onto Eulerian nodes to resolve large-scale dynamics. To counter temporal drift, we employ a cross-attention mechanism that queries these Eulerian features, using the fixed grid as a stable spatial anchor to correct trajectory deviations at every timestep. Comprehensive experiments show that this hierarchical, cross-attended design substantially suppresses error accumulation, establishing a new state-of-the-art for accuracy and rollout stability in Lagrangian fluid simulation.

arXiv

PATH-Bench: Path-Dependent Evaluation of Lifelong Agents

Xidong Yang, Xingyi Zhang, Wenhao Li, Wenyan Liu, Junjie Sheng, Yun Hua, Wei Yin, Tao Fang, Chuyun Shen, Xiangfeng Wang

Lifelong LLM agents increasingly adapt through external learning states that store past interactions as retrievable memories or reusable skills, yet existing benchmarks rarely account for how the path of accumulated experience shapes what agents transfer and retain. In this work, we establish PATH-Bench, a benchmark for path-dependent evaluation of lifelong agents. PATH-Bench estimates directed task relationships via multi-model in-context learning, constructs probe-centered sequences with controlled helpful and interfering histories, and repeatedly evaluates probe tasks to measure average performance, forward transfer, backward transfer, and forgetting. We evaluate eight representative agents on single-turn code generation and multi-turn tool-use tasks under positive- and negative-dominant histories. Benchmark results show that experience utility depends jointly on how experience is represented and on the task's interaction structure, that strong transfer does not ensure retention, and that later experience can reshape gains acquired earlier in the learning path. Based on these findings, we propose Selective Experience Use (SEU), an agent harness that regulates how path-accumulated experience influences each new task, admitting helpful items while filtering out potential interference. SEU consistently reduces forgetting while improving forward transfer in the majority of settings. The PATH-Bench provides both a controlled evaluation framework and actionable guidance for designing more selective and robust lifelong agents.

arXiv

Fighting Fire with Fire: On the Feasibility of Protecting Exercises Against AI Cheating

Tobias Braun, Jonas Grebe, Louis Rethfeld, Marcus Rohrbach

The widespread adoption of generative AI enables students to outsource cognitive effort to increasingly capable assistants, creating an illusion of competence while undermining the independent reasoning that education aims to cultivate. We investigate whether adversarial machine learning can be repurposed to protect educational exercises against such corrosive reliance. Our approach uses multimodal multiple-choice questions whose visual components can be protected with subtle visual perturbations that steer AI solvers toward designated incorrect answers. These responses form a statistical fingerprint: students who blindly copy a solver reproduce the induced answer pattern more frequently than genuine students. We study the feasibility of this paradigm under realistic black-box assistant assumptions using three of the most common state-of-the-art multimodal language models: Anthropic's Claude, Google's Gemini, and OpenAI's ChatGPT. By using accessible surrogate models, we optimize adversarial perturbations that induce consistent response patterns. Those patterns enable principled detection through statistical hypothesis testing. These findings establish both the promise and the limitations of fighting machine-assisted reasoning with the vulnerabilities of the machines themselves.

arXiv

Inter-Residue Geometry Attention for Antibody-Specific Epitope Prediction

Chuanliu Fan, Nan Yu, Junjie Wu, Guohong Fu

Antibody-specific epitope prediction aims to identify which antigen residues are recognized by a given antibody, a task that depends on the three-dimensional complementarity between antibody CDRs and the antigen surface. Existing methods usually leverage PLM embeddings and inject structure through additional graph, surface, or point-cloud encoders, where the positional mechanism inside attention remains largely tied to one-dimensional sequence order. For proteins, the analogue of a token offset is not only sequence separation, but also the three-dimensional displacement between residues after folding. This raises a question, can folded residue geometry serve as the positional mechanism of attention itself? We propose Local-Frame 3D Rotary Position Encoding (LF3DRoPE), which expresses inter-residue displacements in backbone-defined local frames and injects them directly into rotary attention. This design preserves continuous directional geometry while ensuring invariance to global $\mathrm{SE}(3)$ transformations. On the AsEP benchmark, LF3DRoPE achieves state-of-the-art $\mathrm{MCC}$ on both ratio and epitope-group splits. Ablations and rigid transformation tests show that local three-dimensional geometry provides information beyond sequence-order attention while preserving invariance to arbitrary global coordinate systems. Mutation ranking results further indicate that LF3DRoPE captures antigen-specific structural compatibility.

arXiv

Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth

Kehan Wang

Depth-routing residual architectures allow Transformer layers to retrieve earlier representations instead of inheriting only the immediately preceding state. Existing Block Attention Residuals, however, use a single content-dependent depth mixture to construct the inputs to queries, keys, and values. This design couples two functionally different decisions: queries and keys determine where attention matches, whereas values determine what content is retrieved. We therefore ask whether matching and content retrieval should be forced to read from the same depth. We introduce Role-Decoupled Attention Residuals (RD-AttnRes), a minimal extension that shares one depth route between queries and keys while learning an independent value route over the same residual sources. Tying the two routing queries exactly recovers the parent architecture, while decoupling them adds only one model-width vector per layer and introduces no additional token-to-token attention operation. We evaluate RD-AttnRes using a frozen, paired pretraining protocol on FineWeb-Edu with five matched seeds for both 120M- and 343M-parameter models and a 2.0B-token training budget. RD-AttnRes improves validation negative log-likelihood in all 10 matched comparisons. The mean reductions are 0.0301 and 0.0247, corresponding to perplexity reductions of 2.97 percent and 2.43 percent at 120M and 343M parameters, respectively. Early-budget controls indicate that neither the additional parameter count, duplicated routing execution, nor a fixed value route reproduces the improvement. Routing diagnostics further reveal persistent divergence between the query-key and value depth distributions. These results suggest that, within the evaluated training regime, attention matching and content retrieval benefit from distinct reads over the residual hierarchy.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.