NeurIPS 2026 Papers

Speaker identification benchmarks evaluate how accurately a model can predict a ground-truth speaker label for a given utterance, treating voice identity as a property of the recording's producer. However, these benchmarks do not test whether a model aligns with human perception of voice identity. Whether a human would hear two voices and interpret them as the same speaker or not has consequences for intellectual property rights, cybersecurity, and everyday life. We make two contributions. First, we introduce the Voice Identity Perception benchmark (VIPBench), built from the identity judgments of English-speaking crowdworkers on stratified celebrity voice pairs. VIPBench consists of 124,876 same/different identity judgments from 1,290 participants on 9,800 voice pairs across 100 speakers, spanning real recordings, AI voice clones generated by a state-of-the-art Text-to-Speech (TTS) system, and continuously morphed voices. Our second contribution is to define four evaluation tasks: predicting the listener same-speaker agreement rate; binary same/different classification against the majority listener vote, evaluated for both ranking and calibration; alignment between the model's and listeners' speaker-similarity structures; and whether a predictor fit on real speech still works on voice clones and morphs. We report baselines for ten publicly available speech representations. We find that the perception target re-orders model rankings relative to metadata: supervised embeddings (trained on metadata speaker labels) still outperform self-supervised models (which learn voice structure without identity supervision), and no model reaches the noise ceiling. VIPBench enables systematic evaluation of speaker representations against human voice-identity perception across real, cloned, and morphed speech, motivating speaker models trained on perceptual judgments directly.

While LLMs excel at isolated coding tasks, their ability to perform autonomous, long-horizon machine learning (ML) development remains bottlenecked by poor strategic planning. Because current benchmarks rely on outcome-based metrics, they obscure process-level failures, making it impossible to diagnose where agent strategies diverge from human expertise. To solve this, we introduce TraceML, the first large-scale dataset of paired human and agent trajectories for long-horizon ML tasks. TraceML contains 4{,}672 Kaggle trajectories across 134 MLE-bench competitions, with 430 expert humans paired head-to-head with 207 LLM-agent runs on 7 of those competitions. Using a unified extraction schema to analyze step-by-step behavior, we reveal systematic gaps: agents execute local edits well but exhibit unstructured exploration, poor experiment prioritization, and weak long-term memory attribution compared to experts. Translating these diagnostic insights into action, we design a novel, planning-oriented agent harness based on human priors that significantly improves both process efficiency and final competitive performance.

The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain constrained to isolated scenarios, narrow action spaces, or synthetic data, failing to capture the holistic nature of authentic human behavior. To bridge this gap, we introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data, integrating long-horizon, cross-scenario, and heterogeneous behavioral patterns into a unified framework. Based on this benchmark, we first provide empirical evidence that previous datasets with isolated scenarios suffer from tunnel vision, whereas real-world decision-making relies on long-term, cross-scenario causal chains. Extensive evaluations of state-of-the-art LLMs reveal that current models struggle to accurately simulate these complex behaviors, with performance plateauing even as context windows expand. Crucially, a systematic comparison between simulated and authentic behaviors uncovers a fundamental structural bias: LLMs tend to converge toward a positive average person, exhibiting hyper-activity, persona homogenization, and a utopian bias. This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.

Long-horizon RTL agents often collapse under context pollution, lacking the active, compartmentalized debugging strategies of human engineers. We introduce OneVeriAgent, a human-like framework modeled directly on expert cognitive workflows. First, mimicking how engineers probe waveform viewers, Agentic Temporal Exploration (ATE) enables dynamic, targeted signal queries, distilling dense simulator dumps into sparse, causal traces. Second, reflecting how designers compartmentalize tasks, a task-scoped Orchestrator enforces strict context isolation, preventing raw tool outputs from polluting the global reasoning trace. On the CVDP benchmark, OneVeriAgent achieves 95.8\% Pass@1 with Claude Opus 4.6, matching state-of-the-art proprietary swarm. Finally, we propose Tool-Validated Self-Distillation (TVSD) to bridge the open-source capability gap. By fine-tuning on self-generated, syntactically valid trajectories without external correctness oracles, TVSD lifts Gemma4-31B-Instruct's Pass@1 from 61.1\% to 74.8\%, demonstrating that fully open-weight RTL agents are viable.

Unsupervised medical anomaly detection aims to identify images or regions that deviate from the learned normal distribution. Reconstruction-based methods are a dominant paradigm, generating a normal-looking reference for each test image and detecting anomalies via input--reconstruction discrepancy. However, most existing methods follow a one-shot reconstruction-and-comparison pipeline, making anomaly scoring vulnerable to reconstruction failures: abnormal regions may be preserved, while normal anatomical structures may be distorted. Moreover, they often lack coordinated control from structural and semantic normality, which can lead to anatomical inconsistency and semantic drift. We propose S2Agent, a structure- and semantic-guided multi-agent framework that reformulates medical anomaly detection as feedback-driven normality reconstruction. S2Agent decomposes the process into three collaborative agents. The Planner derives sample-specific matched-tree structural priors and normal-only semantic claims from normality knowledge. The Reconstructor generates a normality-oriented reference image under their joint guidance. The Detector verifies the reconstruction in matched-tree and semantic-claim spaces, returning feedback to refine structural weights and semantic constraints for the next round. Through this closed loop, S2Agent iteratively corrects structural deviation and semantic drift, suppresses abnormal or unsupported content, and improves the reliability of anomaly scoring. Experiments on three public medical benchmarks demonstrate the effectiveness of the proposed closed-loop guidance.

Machine learning (ML) predictions are increasingly being used to guide decision-making, giving rise to the problem of decision-focused learning (DFL) where predictors are optimized for downstream decision quality rather than accuracy alone. However, existing work assumes a single decision-maker optimizing in isolation. This paper formalizes *strategic* decision-focused learning, where an ML system predicts an exogenous state that some agents observe before playing a game. For example, a park ranger may predict wildlife locations to allocate anti-poaching patrols against strategic poachers. While the exogenous state is unaffected by agent actions, predictions influence agents' strategies and the resulting equilibrium. We find that strategic considerations fundamentally change the learning problem. In particular, we show the prediction accuracy–equilibrium payoff landscape can be non-monotonic---i.e., better predictions can degrade performance. We propose algorithmic approaches to address these challenges and validate them across benchmarks in wildlife conservation and infrastructure protection. Our theory and experiments highlight the importance of accounting for strategic interactions when designing predictors.

We argue that sport represents the next grand challenge for Artificial Intelligence, a domain rich enough to expose the limits of current approaches and structured enough to drive rigorous scientific progress in understanding and modelling human physical skill. Despite rapid advances in language, vision, and robotic manipulation, the physical intelligence required to learn and adapt complex motor skills remains poorly understood and largely unaddressed. We identify two intertwined gaps: the absence of models for human physical skill, and the absence of the scientific agenda needed to build them. We propose concrete directions for how the community can begin to close both.

Predictive models are often deployed through existing decision policies that stakeholders are reluctant to change unless a risk constraint requires intervention. We study risk-controlled post-processing: given a deterministic baseline policy, choose a new policy that maximizes agreement with the baseline subject to a chance constraint on a user-specified loss. At the population level, we show that the optimal policy has a threshold structure: it follows the baseline except on contexts where switching to the oracle fallback policy yields a large reduction in conditional violation risk. At the finite-sample level, given a fitted fallback policy and score, we develop a post-processing algorithm that uses calibration data to select a threshold. Leveraging tools from algorithmic stability and stochastic processes, we show that under regularity conditions, in the i.i.d. setting, the expected excess risk of the post-processed policy is $O(\log n/n)$. In the special case when an exact-safe fallback policy is available, the algorithm achieves precise expected risk control under exchangeability. In this setting, we also give high-probability near-optimality guarantees on the post-processed policy. Experiments on a COVID-19 radiograph diagnosis task, an LLM routing problem, and a synthetic multiclass decision task show that targeted post-processing can meet or nearly meet risk budgets while preserving substantially more agreement with the baseline than score-blind random mixing.

Audits of human-AI complementarity target the wrong estimand. They ask whether the human knows more than the model; deployment asks whether that knowledge would change the action. Residual predictive expertise is not residual decision value. Deployment rewards actions, and a posterior movement that does not cross a reward-induced action boundary changes nothing; a signal can be informative under log-loss and leave the deployed action unchanged. We formalize the missing quantity as **boundary regret**: the regret of holding the model's action after observing the human. In any finite-action Bayesian decision problem, the value of consulting the human equals expected boundary regret, formally verified in Lean 4. The identity turns when humans help into a geometric question: which decision facets does the human signal move belief across? Complementarity is therefore reward-relative. A given human signal can be valuable under one reward matrix and worthless under another, the human's knowledge unchanged; the boundary moved, not the expertise. The gap is widest precisely where asymmetric costs dominate: high-stakes clinical and operational decisions. Audits built on residual predictive expertise overstate the value of human review and route scarce review toward cases where the human is informative but the action will not move. Boundary regret is the quantity to measure.

Respecting a-priori known symmetries of underlying data is a key principle in designing efficient neural architectures. Convolutional Neural Networks (CNNs) are a canonical example, encoding translation equivariance. Extending this inductive bias to richer symmetry groups such as $\mathrm{SE}(2)$ in the most effective way possible is an ongoing architectural and computational challenge. Building on classical invariant theory, we derive invariant feature maps that are provably optimal: minimal and information-preserving with respect to the symmetry group. Our construction is designed to integrate seamlessly into standard architectures, requiring no modification to nonlinearities or normalization layers. As a concrete implementation, we introduce a plug-and-play module for CNNs that enforces $\mathrm{SE}(2)$ invariance, capturing both translations and rotations within standard architectures. The resulting models are both efficient and expressive. Experiments show competitive or improved accuracy compared to state-of-the-art equivariant methods at significantly lower computational cost. Finally, we provide a theoretical and empirical analysis of the relationship between equivariance and invariance, offering insight into their roles in controlling robustness and flexibility.

As AI systems are increasingly deployed in high-stakes domains such as healthcare, effectively integrating human expertise with AI outputs is critical for reliable decision-making. A substantial body of theoretical work studies optimal mechanisms for human-AI collaboration (HAC), focusing on when to involve humans and how to combine their expertise with AI predictions. However, these approaches largely overlook a key factor highlighted by empirical evidence: human judgments are impressionable and can vary depending on how AI outputs are presented. For example, when AI systems defer decisions to humans, revealing or withholding the AI prediction can significantly influence human judgments. Moreover, existing work neglects deployment-time variability, where conditions, particularly the cost of eliciting human input, may differ substantially from those observed during training. To address these challenges, we introduce a rigorous and theoretically grounded framework for human-AI collaboration, termed \emph{collaborative omnimechanism design}, based on a novel and computationally efficient contextual bandit variant of boosting for outcome indistinguishability. In particular, we reduce a calibration-type outcome indistinguishability condition to projected smooth calibration, providing novel time-efficiency result, bypassing the high-dimension curse induced by complicated collaboration methods. Our framework provides a principled characterization of human impressionability and introduces a unified optimization scheme that enables robust and adaptable decision-making across diverse query costs and evaluation objectives without retraining. We further validate our theoretical results through comprehensive experiments.

Large language models are increasingly deployed as teams of agents that hold different private information about a shared task. How should such agents concisely communicate in a way that promotes aggregation of decision relevant information? Communicating only proposed actions can discard decision-relevant uncertainty: two agents may prefer the same action for different reasons. We study an alternative interface in which agents communicate calibrated beliefs over actions. We first analyze a simple synthetic setting that captures the structure of multi-round reasoning under partial information. In this setting, we show theoretically and empirically that communicating unbiased probabilities can be strictly more powerful than communicating actions. We then test the same principle in a collaborative maze-solving task with trained transformers and pretrained LLM agents. Our results demonstrate that probability communication outperforms action communication when the exchanged predictions preserve calibrated uncertainty. When communicated probabilities are biased we show that the benefits of probability communication can disappear. We give a post-hoc conversation calibration intervention that consistently improves decision-making by correcting these biases. These results suggest that probabilistic communication is useful for multi-agent collaboration not merely because it transmits more information, but because calibrated uncertainty gives other agents a reliable object to update on.

Judging image quality is not only ecologically relevant to everyday humans tasks, but also underpins many machine vision tasks such as image generation. This paper proposes a framework to understand the inherent perceptual space underlying image quality judgment in humans. We propose a multi-dimensional observer model that represents images as distributions in a latent perceptual space and that models human judgment as comparing noisy samples. Being constrained by neural representations in the primate ventral stream and fit to large-scale behavioral data, the model enables analysis of perceptual structure while matching the predictive power of existing metrics. Using this model, we find that the perceptual spaces needed to account for image quality judgment in humans are extremely low-dimensional compared to the image space even when considering its sparsity. The exact structure of the space (e.g., dimensionalities, information encoded) varies between low-level and high-level quality judgments, suggesting that, despite a shared retinal encoding in the beginning, humans selectively construct task-dependent perceptual spaces in visual decision making.

It is widely agreed that when AI models assist decision-makers in high-stakes domains, they should communicate the confidence of their predictions. However, empirical evidence suggests that decision-makers often struggle to determine when to trust a prediction based solely on this communicated confidence. In this context, recent theoretical and empirical work suggests a positive correlation between the utility of AI-assisted decision-making and the degree of alignment between the AI confidence and the decision-makers' confidence in their own predictions. Crucially, these findings do not yet elucidate the extent to which this alignment influences the complexity of learning to make optimal decisions through repeated interactions. In this paper, we address this question in the canonical case of binary predictions and binary decisions. We first show that this problem is equivalent to a two-armed online contextual learning problem with full feedback, and establish a lower bound of $\Omega ( \sqrt{|\mathcal{H}| \cdot |\mathcal{B}| \cdot T} )$ on the expected regret any learner can attain, where $\mathcal{H}$ and $\mathcal{B}$ denote the sets of possible human and AI confidence values. We then demonstrate that, by leveraging the alignment between AI and human confidence, a learner can attain an expected regret of $\mathcal{O}(\sqrt{|\mathcal{H}| \cdot T\log T})$ and, in the regime where $\mathcal{B}$ is countable and $\sqrt{|\mathcal{H}|} = \mathcal{O}(\log T)$, a non-trivial generalization of the Dvoretzky-Kiefer-Wolfowitz inequality improves the regret bound to $\mathcal{O}(\sqrt{T\log T})$. Taken together, these results reveal that alignment can reduce the complexity of learning to make decisions with AI assistance. Experiments on real data from two different human subject studies where participants solve simple decision-making tasks assisted by AI models with different degrees of alignment complement our theoretical results.

We propose HuPER, a human-inspired framework that models phonetic perception as adaptive inference over acoustic--phonetic evidence and linguistic knowledge. HuPER first learns an acoustic-grounded phone recognizer from limited human-annotated data and transcript-only speech: canonical G2P transcriptions are treated as auxiliary linguistic cues rather than ground truth, and are corrected into realized phone proxies for self-training. At inference time, HuPER supports multiple perceptual routes: it can rely on bottom-up phone evidence when the signal is clear, incorporate explicit expectations when a reference is available, or use lexical constraints when acoustic evidence is weak. With only 100 hours of training data, HuPER achieves state-of-the-art phonetic feature error rates on five English benchmarks and strong zero-shot transfer to 95 unseen languages. HuPER also improves robustness on weak-evidence and disordered speech, demonstrating the benefit of adaptive, multi-path phonetic perception under diverse acoustic and task conditions. All training data, models, and code are open-sourced. Code and demo are available at https://github.com/HuPER29/HuPER.

Data collection is a critical component of modern machine learning pipelines. In many settings, multiple agents, such as hospitals or research labs, can collaborate to share the burden of data collection rather than each bearing it alone. But how should the work be divided among these agents fairly? Studying this question is complicated by heterogeneity in agents' data collection costs and data quality, and by the fact that fairness itself is subjective and use-case-dependent. We study these challenges in the context of PAC mean estimation, where agents wish to estimate an unknown scalar $\mu$ to within accuracy $\epsilon$ with probability at least $1-\delta$, via samples from distributions with common mean $\mu\,$. Agents may incur different costs to collect their samples, and their distributions may have different variances (data qualities). Our first contribution is a framework that casts collaborative data collection as a bargaining problem, where agents specify a notion of utility (e.g., negative cost incurred, or cost savings relative to working alone) and a welfare function over agent utilities. By maximizing the welfare over the feasible set of data collection amounts (i.e. there is enough data to satisfy $(\epsilon,\delta)$-PAC mean estimation), we obtain a fair division of work. Drawing on classical fairness axioms from the microeconomics literature, the family of admissible welfare functions takes a specific one-parameter form that includes utilitarian, egalitarian, and Nash bargaining as special cases. When agents' noise variances (data qualities) are known, this yields a clean characterization of the fair division. Our second contribution addresses the more challenging setting where variances are unknown. As the feasible set of data collection amounts itself depends on the variances, it is impossible to specify a fair division of work upfront. We propose an online algorithm that combines plug-in variance estimates with carefully calibrated forced sampling, and show that it is asymptotically optimal: the ratio of the achieved welfare to the welfare of the true fair division converges to 1 almost surely as either $\epsilon$ or $\delta$ goes to 0. We further show that our estimator asymptotically exactly meets the specified error tolerance and failure probability.

We propose **EverAnimate**, an efficient post-training method for long-horizon animated video generation that preserves visual quality and character identity. Long-form animation remains challenging because highly dynamic human motion must be synthesized against relatively static environments, making chunk-based generation prone to accumulated drift: (i) low-level quality drift, such as progressive degradation of static backgrounds, and (ii) high-level semantic drift, such as inconsistent character identity and view-dependent attributes. To address this issue, **EverAnimate** restores drifted flow trajectories by anchoring generation to a persistent latent context memory, consisting of two complementary mechanisms. *(i) Persistent Latent Propagation* maintains a context memory across chunks to propagate identity and motion in latent space while mitigating temporal forgetting. *(ii) Restorative Flow Matching* introduces an implicit restoration objective during sampling through velocity adjustment, improving within-chunk fidelity. With only lightweight LoRA tuning, **EverAnimate** outperforms state-of-the-art long-animation methods in both short- and long-horizon settings: at 10 seconds, it improves PSNR/SSIM by 8%/7% and reduces LPIPS/FID by 22%/11%; at 90 seconds, the gains increase to 15%/15% and 32%/27%, respectively. Anonymous project page: [https://anonymous-xxx-lab.github.io/](https://anonymous-xxx-lab.github.io/).

When large language models (LLMs) are used for creative tasks where the evaluation is largely subjective, two crucial activities are ideation (in which a list of candidate ideas is produced) and pruning (in which the initial candidates are prioritized through a process of comparison). In this framework, the subjective results are influenced by inherent biases in the LLM's generation and evaluation, and it is therefore important to understand how these subjective biases operate. We define and explore a set of stylized tasks that allow us to highlight the effects of these subjective biases in LLM evaluation. In the process, we find evidence for several key underlying principles. First, there are multiple subjective LLM biases at play, and to understand the overall evaluation we must analyze the relative strengths of these biases and their interactions. Second, different prompts -- even when they seem superficially similar -- can have the effect of making different subjective LLM biases more or less salient. And third, there are complex interactions between the LLM's initial idea-generation phase and the subsequent decisions that go into prioritizing and pruning these ideas.

NIGHTS is a recent benchmark for human perceptual similarity over objects, scenes, and composition. DreamSim, the supervised metric introduced with NIGHTS, and later supervised metrics trained on the same data report large gains over self-supervised baselines. The standard NIGHTS evaluation uses the unanimous-agreement subset of a released 100,000-triplet corpus, leaving weak-majority and split-vote triplets outside the headline benchmark. We evaluate eleven perceptual similarity methods on the full released corpus, analyzing unanimous, weak-majority, and split-vote triplets separately. On the 64,420 non-tie triplets, cosine distance on a frozen self-supervised backbone reaches 74.1% accuracy without training, LoRA fine-tuning reaches 77–79%, and the DreamSim ensemble reaches 80.7%. A split-half reproducibility estimate from the triplet vote counts shows that majority labels reproduce only 70.1% of the time overall, and only 33–50% on weak-majority triplets. The extra accuracy comes from matching the collected votes, not from showing that the metric would better match a new set of human judgments. The evaluation target is also unstable across task formulations: NIGHTS’ 2AFC similarity labels and JND same/different labels disagree on 51–64% of shared stimuli. Fine-tuning changes the feature basis as well, increasing foreground-only fragility and making DreamSim the most fragile method under rotation, color, and blur. NIGHTS supports a narrower conclusion: cosine distance on self-supervised features captures the stable signal currently measurable in these labels. Additional claims of human alignment require a redesigned benchmark with more votes per triplet, axis-specific similarity questions, per-subject statistics, and ablation tests.