paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

Decided Upstream, Written Late: Locating and Pricing the Cross-Lingual Refusal Circuit of a Multilingual MoE

Ramakrishna P. Kompella, Aadit Mahajan

Safety alignment in multilingual models is uneven: a model that reliably refuses a harmful request in English will often comply with the same request in a lower-resource language. We trace this gap mechanistically in sarvam, an Indic-multilingual mixture-of-experts reasoning model, and find it is not a failure to detect harm. Harm is encoded as an internal direction that is nearly language-invariant in mid-network (English-vs-Indic cosine ${\approx}0.9$ at $L11$), and steering that direction upstream causally controls refusal. But the detection direction is orthogonal to the change that actually writes the refusal, which is late and assembled over the course of generation rather than read off in a single forward pass. We attribute the write to a specific, localizable circuit, a mixture-of-experts writer held in check by an attention opposer and price every way of intervening on it: damping the opposer is cheap and effective, amplifying the writer is a cost wall, and surgical edits to the responsible heads do nothing. The circuit's organization, and the gradient method that exposes it, recur in a second, unrelated MoE model, while the lever's strength is architecture-specific. The result is a cost-measured map of where a multilingual safety repair can land, and what it costs

arXiv

Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards

Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang, Junming Zhang, Ranjie Duan, Qiaolin Xia, Hao Wang, Yu Lu, Haibo Shi, Xingjun Ma

Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Existing external rewards provide either sparse outcome supervision or richer feedback from process annotations and LLM judges. Outcome rewards scale readily but cannot distinguish grounded retrieval from redundant search, whereas richer signals require costly annotation or inference during training. Internal rewards based on policy-side signals such as entropy, likelihood, or information gain are graded and inexpensive to evaluate, yet mainly reflect model confidence rather than evidence grounding. We propose Search-G1, a representation-based intrinsic reward framework that measures the operational grounding of an agent's answers through two intervention-calibrated readouts. A prompt-state readout predicts closed-book sufficiency, whose complement defines policy-relative retrieval necessity; an answer-commit readout estimates evidence reliance from answer-stage sensitivity to evidence deletion. Together, they provide additional credit to correct searched trajectories when retrieval is estimated necessary and the answer is evidence-sensitive, favor correct direct answers when closed-book knowledge suffices, and penalize repeated search. After calibration, reward scoring requires neither process annotations nor LLM-as-judge inference during policy optimization. Because reinforcement learning changes policy representations, Search-G1 periodically refits both readouts on trajectories from the latest checkpoint, allowing the reward to co-evolve with the policy. Experiments across multiple search-based question-answering benchmarks and two model scales show that Search-G1 improves the grounding--search-cost trade-off, producing shorter response-side trajectories at competitive task accuracy. Code is available at https://github.com/Rosy0912/Search-G1.

arXiv

Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent

Chenyu Wang, Yi Liu, Baoqing Li, Min Tu, Diping Song

Scaling supervision for multi-turn medical agents is difficult because expert dialogue annotation is costly and clinical conversations are privacy-restricted. We introduce Guideline-as-Oracle (GAO), which compiles American Academy of Ophthalmology guidance into a 70-row operational rule table and uses it as the sole source of instance-level supervision for 3,000 training dialogues, reserving human labeling for evaluation. Because converting rules into dialogues is itself a design problem, we catalog eight construction strategies, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair, and characterize the evidential status of each: labeling mechanism, null, confounded, or evaluated only as a package. Fine-tuning a 9B backbone on this corpus yields GAO-Triage, improving agreement with a 201-case operational reference from 61.7% to 74.1% (exact McNemar p=0.0046) and emergent-case recall from 9.5% to 69.0%; the gains persist across a second seed and patient simulator. None of the seven general-purpose systems we test dominates GAO-Triage on both metrics, and GAO-Triage requires no frontier model at inference time. Permuting label-dialogue assignments collapses the model to a constant-routine predictor, indicating that the signal lies in guideline-derived assignment rather than dialogue surface form. Label repair coincides with the disappearance of a late-training safety degradation.

arXiv

Douyin Multimodal Embedding Model Technical Report

Haonan Chen, Chu Li, Zhicheng Wang, Yuanwei Liu, Yuanjiang Wang, Shaohua Jiang, Zhicheng Dou

Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.

arXiv

Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

Hailay Kidu Teklehaymanot, Debela Desalegn Yadeta, Wolfgang Nejdl

Multilingual pre-trained language models such as XLM-R perform well for major languages but struggle with low-resource Ge'ez-script languages, largely due to high out-of-vocabulary (OOV) rates and excessive subword fragmentation from Latin-script-centric tokenizers. We introduce VEXMLM, a vocabulary-extended variant of XLM-R targeting Amharic and Tigrinya. We train language-specific SentencePiece tokenizers on curated monolingual corpora, extend XLM-R's vocabulary with 30k Ge'ez-script subwords, and initialize their embeddings via subword averaging. VEXMLM undergoes two-stage training: (1) continued masked language modeling on the curated corpora and (2) supervised fine-tuning on question answering, named entity recognition, and sentiment analysis. VEXMLM substantially outperforms XLM-R and Glot500 across all evaluated tasks on Amharic and Tigrinya, with particularly strong gains on out-of-vocabulary entity recognition. Critically, improvements on Amharic and Tigrinya transfer to 17 languages in Africa. VEXMLM demonstrates that vocabulary expansion and tokenizer adaptation provide an effective, computationally efficient path to improve multilingual models for underrepresented languages without retraining from scratch. Resources: GitHub repository | Hugging Face models

arXiv

DominoTree: Conditional Tree-Structured Drafting with Domino for Speculative Decoding

Saw S. Lin, Jyh-Shing Roger Jang

Speculative decoding accelerates LLM inference by drafting tokens and verifying them in parallel. Block-diffusion drafters such as DFlash model only per-position marginals, and tree methods such as DDTree expand candidate trees from those marginals. The released Domino drafter adds a GRU-based causal correction making each draft token's distribution path-dependent, a structure DDTree's factorized formulation cannot represent. We introduce DominoTree, a training-free best-first draft tree scored by Domino's conditional (non-factorized) correction along each root-to-node path, made practical by restricting the per-node correction to a candidate top-M. We evaluate it on eight benchmarks in a single-stream harness, and in SGLang, where it runs as an out-of-tree plugin against AR, DFlash, EAGLE-3 and Domino under identical flags. DominoTree attains the highest mean accepted length in every serving cell - two model sizes, single-request and concurrent load, context to 32K - and the highest Overall accepted length at every temperature in the research harness (21 of 24 per-dataset cells). A three-arm decomposition holding drafter, budget and verifier fixed separates the gain from applying the correction at all (+10.1% accepted length) from that of recomputing it along each candidate's realized path (+4.7% more), the part this paper adds. Where the round is verify-dominated, throughput follows: up to 7.3x over AR on Qwen3-8B, beating the released Domino decoder at its CUDA-graph best at every temperature, and inside SGLang winning single-request throughput by +12% over Domino on Qwen3-8B. On HELMET long context it beats Domino by +29-36% accepted length and +10-34% throughput at every length and both model sizes. Past a memory-constrained card's admission cap the chain wins goodput, and at our longest context, where prefill dominates, our lead over EAGLE-3 narrows to a tie.

arXiv

The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents

Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He

A self-evolving agent retires its bad skills by watching them fail, so what happens when the judge cannot see the failures? Skill retirement is the structural constraint that keeps a growing library from drifting below the no-skill baseline, but its guarantee assumes an unbiased reward, which is false for the LLM judges that reference-free tasks require. We show that a biased judge does not merely add noise; it \emph{silently switches off the curator}. We make this precise with a corrupted-reward analysis, then a behavioral study on a reference-free report-writing testbed with a code-generation cross-check, injecting corruption on top of a deterministic reward to isolate the causal channel. Symmetric noise leaves retirement intact, but \emph{false-pass} bias (failures slipping through as passes) disables contribution-based retirement past a sharp threshold (here a false-pass rate of $0.45$) that no amount of data can cross. Separating genuine retirement from cap-eviction churn shows this \emph{mechanism} failure is universal, holding across domains and failure rates and sparing only near-zero-false-pass, verifier-like graders. The downstream \emph{outcome}, though, is regime-dependent: eval quality degrades only where the same corruption also starves skill synthesis, and otherwise holds steady, so the disabled curator is \emph{silent}, surfacing in no aggregate metric. The contribution is a behavioral safety result, not a performance one. A cheap defect-injection audit then tells an operator, before deployment, which side of the threshold their judge occupies.

arXiv

Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

Yu Li, Xiuyu Li, Mingyang Yi, Jiaxing Wang, Liangxu Zhang, Zhaolong Xing, Zhen Chen

Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is "transferring" these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.

arXiv

Can LLMs Reliably Self-Report Adversarial Prefills, and How?

Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim

Prior work shows that large language models (LLMs) exhibit varying degrees of introspective capability on benign tasks. We extend the question to safety contexts and examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack. Across ten open-weight instruction-tuned LLMs from 3B to 70B parameters and four safety benchmarks, no model reliably recognizes its own compromised outputs, with models claiming intent on prefilled responses at an average rate of 25.3%. Introspective signal stems primarily from reasoning about safety and refusal. Orthogonalizing models' weights against the refusal direction collapses the gap between claim rates on prefilled and natural outputs to near zero, though the direction is not its unique mediator. Framing the question as internal intention versus external tampering elicits qualitatively different responses on the same models. Training models to mimic correct introspective answers or optimize an introspective objective can improve the accuracy of introspection, but such training does not transfer to the tampering probe and counterintuitively raises attack success rate under adversarial prefill on most models, amounting to a partial mitigation. These findings outline mechanisms underpinning the observed introspective signals in safety contexts and highlight risks in the reliability of LLM self-reports.

arXiv

Wind Turbine Maintenance Log Labelling Framework: LLM-Driven Data Correction and Enrichment via Semantic Extraction of Reliability Intelligence

Max Malyi, Jonathan Shek, Alasdair McDonald, Andre Biscaya

As wind turbine fleets age, data-driven reliability engineering and maintenance optimisation are essential to manage lifecycle expenditure and support asset life extension. Historical maintenance records offer a vital source of field evidence, yet their analytical use is impeded by inconsistent system codes, generic categorical fields, and unstructured technician text. This paper presents a topology-aware large language model (LLM) workflow for reviewing legacy labels, extracting candidate maintenance and failure-mode taxonomies, and assigning structured semantic fields at record level. The workflow processed 16,316 maintenance records from 280 turbines across 32 onshore wind farms, spanning 9.2 years of operational history. It combines system-specific batch synthesis with granular labelling, deterministic exclusions, structured outputs, record-level provenance, and explicit review routes. Of 2,984 records targeted by three system-code tasks, 2,178 proposed labels met the operational acceptance rule of a 'High' self-reported confidence tier and no human-review flag. Accepted maintenance-type and action labels were assigned to 14,251 and 13,179 records, respectively. Failure-mode evidence profiles were assigned to 11,662 records; 3,441 records were classified as containing insufficient information, and 1,213 records were excluded as 'Not applicable' by deterministic workflow rules. The resulting fields reveal changes in system and maintenance-type distributions, a broader component-level action vocabulary, and topology-specific candidate evidence profiles. The recorded API expenditure was $368.86, or $0.0226 per processed record, and the total wall-clock duration was 6.83 hours. The provenance-linked outputs constitute candidate semantic evidence for subsequent multi-source event reconstruction, exposure-based reliability analysis, and failure modes and effects analysis (FMEA).

arXiv

H$^{2}$MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer

Maryam Haghifam, Zifan He, Jason Cong, Yizhou Sun

Transformer-based LLMs achieve strong results on many language tasks; however, long inputs remain challenging because context windows are finite, and prefill latency and memory grow rapidly with prompt length. Flat token-stream processing and chunk-based retrieval can therefore spend substantial computation and context budget on text unrelated to the query. Offline-indexed RAG additionally introduces external storage and index management overhead, and typically appends retrieved evidence as raw text, increasing prefill cost and latency. H^{2}MT makes long-context inference structure-aware: it builds a semantic hierarchy offline, computes a memory embedding for each node via bottom-up post-order aggregation, and routes queries coarse-to-fine at inference to prune irrelevant branches early. On LongBench QA (NarrativeQA, HotpotQA, QASPER) and two structured technical-document settings, H MT achieves favorable quality efficiency trade-offs, delivering competitive ROUGE-L and F1 (where applicable) with lower peak GPU memory and time-to-first-token (TTFT) than prompt compression, memory-token methods, and retrieval-augmented generation baselines.

arXiv

Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades

Donghyuk Jung, Youngwon Choi

We analyze how automatic speech recognition (ASR) errors propagate through ASR--LLM cascades in Korean spoken question answering (SQA), focusing on downstream semantic failures that conventional ASR metrics cannot fully capture. Our analysis shows that the relative downstream degradation caused by ASR errors is consistent across LLMs with different absolute performance, suggesting that cascade degradation largely tracks ASR-stage information loss. We further identify single-character ASR errors as a particularly salient source of information loss in Korean, where even a minimal transcription difference can change the intended question and degrade downstream QA performance. Finally, an auxiliary comparison shows that a large audio language model outperforms an ASR--LLM cascade with an approximately matched language backbone in noisy Korean SQA, indicating the potential of direct audio input to mitigate transcript-induced information loss.

arXiv

FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers

Sihan Wang, Jiayi Zhao, Qingyan Cao, Hongbo Yao, Lin Shu

Activation steering has emerged as a lightweight approach for modifying language model behavior without parameter updates, yet existing methods remain brittle: unstable across layers and prone to disturbing behavior unrelated to the target concept. We trace these failures to a hidden assumption shared by widely-used methods such as CAA, ActAdd, and ITI: that the intermediate activation space is Euclidean. We show this assumption is fundamentally flawed. The metric that actually governs how a hidden-state perturbation changes the output is the Fisher information metric of the softmax layer, pulled back to the intermediate layer through the Jacobian of the intervening layers. From it we derive a closed-form steering direction, applied to a hidden state at an intermediate layer, that reaches a target concept change with the least non-target distortion. The framework is sharpest in the early and middle intermediate layers, where the metric is strongly non-Euclidean and geometric correction matters most. We evaluate it on three verb-morphology concepts: third-person-singular, progressive, and past-tense inflection, following standard counterfactual-concept evaluation. On GPT-2 Small, this non-Euclidean geometry is borne out empirically, and our method lowers off-target KL divergence by median factors of 1.4--6.5x against individual steering baselines. On Llama-3-8B and Qwen3-8B, it lowers off-target KL by median factors of 1.8--3.6x against individual baselines at the early and middle layers. These results show that geometric correction retains its advantage on larger models with more complex internal structure.

arXiv

SOD: Step-wise On-policy Distillation for Small Language Model Agents

Qiyong Zhong, Mao Zheng, Mingyang Song, Xin Lin, Jie Sun, Houcheng Jiang, Xiang Wang, Junfeng Fang

Tool-integrated reasoning (TIR) is difficult to scale to small language models due to instability in long-horizon tool interactions and limited model capacity. While reinforcement learning methods like group relative policy optimization provide only sparse outcome-level rewards. Recently, on-policy distillation (OPD) has gained popularity by supplying dense token-level supervision from a teacher on student-generated trajectories. However, our experiments indicate that applying OPD to TIR leads to a critical failure mode: erroneous tool calls tend to cascade across subsequent reasoning steps, progressively amplifying student-teacher divergence and rendering the teacher's token-level supervision increasingly unreliable. To address this, we propose SOD, a step-wise on-policy distillation framework for small language model agents, which adaptively reweights distillation strength at each step based on step-level divergence. Therefore, SOD can attenuate potentially misleading teacher signals in high-divergence regions while preserving dense guidance in well-aligned states. Experiments on challenging math, science, and code benchmarks show that SOD achieves up to 20.86% improvement over the second-best baseline. Notably, our 0.6B student achieves 26.13% on AIME 2025, demonstrating effective transfer of agentic reasoning to lightweight models. Our code is available at https://github.com/YoungZ365/SOD.

arXiv

TCMIIES: A Browser-Based LLM-Powered Intelligent Information Extraction System for Academic Literature

Hanqing Zhao

The rapid growth of academic publications has created a need for tools that extract structured knowledge from unstructured scientific texts. Although large language models (LLMs) can perform natural language understanding and information extraction, existing solutions often require specialized infrastructure, programming expertise, or fine-tuned domain-specific models, which limits their accessibility for researchers in specialized fields. This paper describes TCMIIES (Traditional Chinese Medicine Information Intelligent Extraction System), a browser-based, zero-installation platform that uses commercial LLM APIs to perform structured information extraction from academic literature. The system employs a schema-guided prompting framework with automatic system prompt generation, allowing researchers to define custom extraction schemas through a graphical interface without programming. TCMIIES features a pure front-end architecture that processes all information locally in the browser, supports five major LLM providers (DeepSeek, OpenAI, Qwen, Zhipu AI, and custom OpenAI-compatible endpoints), implements concurrent batch processing with automatic retry mechanisms, and provides intelligent field mapping for Chinese academic databases including CNKI and Wanfang. Evaluation across multiple extraction scenarios in Traditional Chinese Medicine research shows structured output compliance rates exceeding 94\% and extraction accuracy approaching but below expert-level agreement ($κ=0.82$ as reference). The system offers a flexible, privacy-preserving, and cost-effective solution for domain researchers who need to process literature at scale.

arXiv

Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting

Chi Liu, Xin Chen, Xu Zhou, Fangbo Tu, Srinivasan Manoharan

Large Language Models (LLMs) have achieved remarkable success, underpinning diverse AI applications. However, they often suffer from performance degradation due to factors such as catastrophic forgetting during Supervised Fine-Tuning (SFT), quantization, and pruning. In this work, we introduce a performance recovery framework based on Self-Distillation Fine-Tuning (SDFT) that effectively restores model capabilities. Complementing this practical contribution, we provide a rigorous theoretical explanation for the underlying recovery mechanism. We posit that an LLM's generative capability fundamentally relies on the high-dimensional manifold constructed by its hidden layers. To investigate this, we employ Centered Kernel Alignment (CKA) to quantify the alignment between student and teacher activation trajectories, leveraging its invariance to orthogonal transformations and scaling. Our experiments demonstrate a strong correlation between performance recovery and manifold alignment, substantiating the claim that self-distillation effectively aligns the student's high-dimensional manifold with the optimal structure represented by the teacher. This study bridges the gap between practical recovery frameworks and geometric representation theory, offering new insights into the internal mechanisms of self-distillation.

arXiv

Chronos: The AI Co-Historian

Lorenz Hufe, Niclas Griesshaber, Gavin Greif, Sebastian Oliver Eck, Pieter Francois, Wojciech Samek, Christian Schroeder de Witt, Philip Torr

AI is increasingly supporting, accelerating, and automating scientific discovery across subjects. Yet, the adoption of AI in historical research remains limited due to the lack of specialised solutions for historians. To change this, we introduce Chronos, an AI Co-Historian designed to support historians. It allows researchers to create and customize research workflows through natural-language interaction and share these as Chronos-Extensions with others. Chronos specifically addresses the need of historians for a tool that is specialised, non-technical, highly customizable, and facilitates extensive task evaluation. As a first extension, we introduce Chronos-Extract, which enables researchers to automate the targeted extraction of information from image scans of historical sources. We benchmark Chronos-Extract on three historical source corpora and find that it achieves high task-accuracy across primary sources spanning three centuries and diverse languages, layouts, and typefaces. Chronos is openly available and ready for historians to use on their own primary and secondary sources.

arXiv

Supporting Calibrated Reliance in Human-AI Collaboration: Different Strategies for Different Tasks

Ruth Cohen, Lu Feng, Ayala Bloch, Sarit Kraus

As AI systems increasingly support human decision making, a central challenge is determining what information helps people recognize when to rely on AI predictions and when to question or override them. Across three controlled human-subject studies spanning abstract visual reasoning with RAVEN matrices and deductive logical reasoning with LSAT problems, we examine how different forms of AI support affect human--AI team performance. A multi-stage reveal study shows that AI predictions and explanations can affect objective accuracy and subjective confidence differently. In visual reasoning, LLM explanations do not improve accuracy beyond the predicted answer alone, and no additional support format significantly outperforms prediction-only support; predicted probabilities show the highest descriptive accuracy and error recovery, while a derived selective-automation policy provides a higher-performing reference benchmark. In language-based logical reasoning, by contrast, LLM explanations yield the highest accuracy and error recovery, outperforming expert-written explanations and probability-based support. These results show that no single support strategy is universally effective. Human--AI interfaces should instead be designed to support calibrated reliance and effective error recovery by matching the form of assistance to the task and the evidence available to users.

arXiv

TSQueryBench: LLM-as-a-Judge for Time Series Explanations

Preetham Sivalingam, Murari Mandal, Dhruv Kumar, Saurabh Deshpande

Natural language explanations of time series data are increasingly produced by foundation models in high stakes domains, making factual correctness critical. Evaluating such explanations differs fundamentally from standard natural language generation: correctness requires verifying numerical claims against structured data rather than similarity to reference text. While LLM as a Judge has emerged as a scalable paradigm for text evaluation, its applicability to numerically grounded time series explanations remains unstudied. We introduce TSQueryBench, a controlled synthetic benchmark of 500 time series instances across 10 query types, each paired with correct, partially correct, and incorrect explanations. We evaluate six large language models across four tasks: explanation generation, relative ranking, independent scoring, and multi anomaly detection. Our central finding is a consistent generation evaluation asymmetry: models that fail to generate numerically correct explanations nonetheless reliably identify or score correct ones. These results show that rubric guided LLM evaluation is substantially more reliable than generation for time series reasoning, supporting LLM judges as scalable evaluators in numerically grounded settings. Code and data: https://github.com/Prxxthxm/TSQueryBench/

arXiv

ContextClaim: A Context-Driven Paradigm for Verifiable Claim Detection

Yufeng Li, Rrubaa Panchendrarajan, Arkaitz Zubiaga

Automated fact-checking pipelines typically begin with a filtering stage that decides which claims are worth verifying, given that the later evidence retrieval and verification components are expensive to apply at scale. A central task in this stage is verifiable claim detection, which asks whether a statement is in principle checkable against external evidence. Prior work on this task, as well as on the closely related notion of check-worthiness, conditions its decisions only on the claim sentence itself. We argue that this is restrictive, because deciding whether a statement is checkable often depends on identifying the entities and events it mentions, and on whether external information about them is actually available in the first place. Motivated by how downstream verification systems rely on retrieved evidence, we move retrieval upstream into the detection stage and introduce ContextClaim. Given an input claim, the approach identifies entity mentions, queries Wikipedia as a structured background source, and uses large language models to compress the retrieved material into short contextual summaries that are then passed to a classifier. Experiments are conducted on two domains and genres, namely the CheckThat! 2022 Twitter collection and the PoliClaim corpus of political debates, and cover both encoder and decoder only models under fine-tuning, zero-shot, and few-shot settings. The added context yields gains on verifiable claim detection in several configurations, although the size of the improvement varies with the dataset, the backbone model, and the training setup. We further find that the same retrieved summaries are useful beyond detection. Feeding them into a downstream verification model on FEVER improves verification F1. Component level analyses, human annotation, and error inspection further clarify the conditions under which retrieved context helps, and where it does not.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.