paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignment

Hexiang Gu, Qifan Yu, Yuan Liu, Zikang Li, Saihui Hou, Jian Zhao, Zhaofeng He

As a multimodal communication medium that integrates images and text, memes often convey implicit harmful content through metaphors, satire, and humor, making harmful meme detection a complex and challenging task. Although recent studies have achieved considerable progress in detection accuracy and model interpretability, large-scale, high-quality datasets for harmful memes remain scarce. Moreover, existing methods still exhibit notable limitations in identifying implicit risks and understanding fine-grained semantics. To address these challenges, we construct MemeMind, a large-scale dataset for harmful meme detection. MemeMind comprises a broad collection of publicly available memes and adopts a rigorous and comprehensive taxonomy of harmful content developed in accordance with widely recognized international standards and contemporary Internet contexts. In addition, the dataset provides detailed structured Chain-of-Thought (CoT) reasoning annotations to support fine-grained analysis of harmfulness, implicit intentions, and underlying semantics in memes. Building upon MemeMind, we further propose MemeGuard, a reasoning-oriented multimodal framework for harmful meme detection. MemeGuard employs a three-stage training strategy to progressively enhance the model's visual understanding, multimodal reasoning, and harmful content discrimination capabilities, thereby improving both detection accuracy and the interpretability of model decisions. Extensive experimental results demonstrate that MemeGuard outperforms existing state-of-the-art methods on the MemeMind dataset, providing a solid foundation for future research on harmful meme detection and multimodal content safety.

arXiv

Reliable Proof Generation with LLMs via Analogical Retrieval and Symbolic Verification: A Case Study in Euclidean Geometry

Oren Sultan, Eitan Stern, Dafna Shahaf

Large language models (LLMs) struggle with formal domains that require rigorous logical deduction and symbolic reasoning, such as mathematical proof generation. We propose a neuro-symbolic approach centered on the hypothesis that structurally analogous problems often admit similar proofs. As a proof-of-concept, we focus on SAT-level geometry problems. Our approach is two-fold: (1) We retrieve analogous problems and use their proofs to guide the LLM, and (2) a formal verifier evaluates the generated proofs and provides feedback, helping the model fix incorrect proofs. Our complete pipeline substantially improves proof accuracy across model families, achieving 68%-96% accuracy compared with 10%-44% for LLM-only baselines that use neither analogy retrieval nor verifier feedback. When comparing against models with the same verifier feedback and inference budget, analogical guidance improves accuracy from 88% to 96% for GPT-5, 78% to 86% for Claude Sonnet 4.6, 72% to 86% for Gemini-Flash-2.5, and 52% to 80% for OpenAI o1. More broadly, shifting to LLMs that generate provably correct conclusions has the potential to dramatically improve their reliability, accuracy and consistency, unlocking complex tasks and critical real-world applications that require trustworthiness.

arXiv

AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study

Mostafa Faghih Shojaei, Rahul Gulati, Benjamin A. Jasperson, Shangshang Wang, Simone Cimolato, Manas Vardhan, Dangli Cao, Willie Neiswanger, Krishna Garikipati

We introduce AI University (AI-U), a flexible framework for AI-driven course content delivery that adapts to a course's instructional style. AI-U combines a fine-tuned large language model (LLM) with retrieval-augmented generation (RAG) and a reasoning synthesis model to generate style-aligned responses from lecture videos, notes, and textbooks. Using a graduate-level finite-element-method (FEM) course as a case study, we present a pipeline to synthesize course-grounded training data, fine-tune an open-source LLM with Low-Rank Adaptation (LoRA), and apply RAG-based synthesis. Our evaluation---combining cosine similarity, LLM-based assessment, expert review, and user studies---shows improved alignment with course materials relative to the base model. We have also developed a prototype web application, available at https://my-ai-university.com, that enhances AI-generated responses with references to relevant sections of the course material and clickable links to time-stamped video lectures. Our expert model is found to be higher scoring by a quantitative measure on 86% of test cases. An LLM judge also preferred our expert model to its base model under both evaluation prompts. Human evaluation by advanced users showed a preference for our expert model approximately twice as often as for the base model. The FEM course instructor found our expert model to achieve better alignment with class-specific content than a recent closed-weight model when both were combined with the reasoning synthesis model. AI-U offers a practical approach to developing course-specific learning assistants using fine-tuned and retrieval-augmented LLMs. By presenting our framework in an FEM class---central to training PhD and master's students in engineering science---we offer a template with potential for extension across STEM fields.

arXiv

A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges

Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, Guangyao Shi

Large vision-language models (VLMs) have evolved rapidly from contrastive image-text encoders and adapter-based assistants into natively multimodal foundation models that support long-context reasoning, agentic workflows, and, increasingly, unified perception-generation-action loops. Since 2025, the frontier has consolidated around a small number of large pretraining families, including GPT, Gemini, Claude, Grok, Qwen, Gemma, DeepSeek, Kimi, and MiniMax, while evaluation has shifted from short-form visual question answering toward spatial, temporal, embodied, and calibration-sensitive benchmarks. In this survey, we provide an updated overview of VLMs through 2026 with an emphasis on three developments. First, we trace the architectural evolution from contrastive or bridged two-tower models to LLM-backbone adapter models, native multimodal-input models, omni-modal unified input/output models, and emerging world-action models. Second, we summarize how benchmarks expanded from classical OCR, VQA, and chart understanding toward long-video reasoning, embodied evaluation, reward-model judging, and domain-specific decision making. Third, we review the growing role of post-training, including supervised fine-tuning, preference optimization, and reinforcement learning methods such as GRPO-style multimodal alignment. Rather than exhaustively listing every release, we focus on representative frontier families, influential benchmarks, and the major open challenges that remain in hallucination, safety, efficiency, data quality, and multimodal alignment.

arXiv

NeST: Neighborhood-aware semantic alignment and temporal modulation for LLM based time series forecasting

Jayanie Bogahawatte, Sachith Seneviratne, Maneesha Perera, Saman Halgamuge

Adapting Large Language Models (LLMs) trained on discrete text data, to forecast continuous time series signals is challenging. While finetuning the LLMs enables such adaptation, effectively integrating both textual and time series information in the prompt is critical. Current LLM-based time series forecasting methods combine the two modalities through simple concatenation or parameter heavy cross-attention. Moreover, existing methods embed time series data using decomposition techniques that may inadequately capture complex temporal dynamics. To address these limitations, we propose neighborhood-aware semantic alignment and temporal modulation based framework (NEST) to formulate a new text-integrated time series prompt to finetune the LLM. First, we generate neighborhood-aware text prototypes that are optimized to represent local neighborhoods of pretrained word token embeddings of the LLM. Second, we align them with temporal representations of the time series input using a nearest-neighbor contrastive objective, after which the top-k most relevant text prototypes are retrieved. Third, we introduce text prototype conditioned temporal modulation that uses the retrieved text prototypes to adaptively scale and shift time series features. Extensive experiments demonstrate that NeST consistently outperforms state-of-the-art methods across eight benchmarks, achieving an average 1.2\% reduction in MSE for long-term forecasting. In addition, it demonstrates strong generalization, yielding an average 4.9\% reduction in MSE in zero-shot forecasting. Beyond benchmark datasets, NeST also delivers robust performance on a real-world distributed photovoltaic power forecasting task across nine datasets, improving the average R$^2$ score by 3.3\%. These findings demonstrate the effectiveness and generalizability of NeST for adapting LLMs to diverse time series forecasting tasks.

arXiv

A Survey on Human-AI Collaboration with Large Foundation Models

Vanshika Vats, Marzia Binta Nizam, minghao liu, Ziyuan Wang, Richard Ho, Mohnish Sai Prasad, Vincent Titterton, Sai Venkat Malreddy, Riya Aggarwal, Yanwen Xu, Lei Ding, Jay Mehta, Nathan Grinnell, Li Liu, Sijia Zhong, Devanathan Nallur Gandamani, Xinyi Tang, Rohan Ghosalkar, Celeste Shen, Rachel Shen, Nafisa Hussain, Kesav Ravichandran, James Davis

As the capabilities of artificial intelligence (AI) continue to expand rapidly, Human-AI (HAI) Collaboration, combining human intellect and AI systems, has become pivotal for advancing problem-solving and decision-making processes. The advent of Large Foundation Models (LFMs) has greatly expanded its potential, offering unprecedented capabilities by leveraging vast amounts of data to understand and predict complex patterns. At the same time, realizing this potential responsibly requires addressing persistent challenges related to safety, fairness, and control. This paper reviews the crucial integration of LFMs with HAI, highlighting both opportunities and risks. We structure our analysis around four areas: human-guided model development, collaborative design principles, ethical and governance frameworks, and applications in high-stakes domains. Our review shows that successful HAI systems are not the automatic result of stronger models but the product of careful, human-centered design. By identifying key open challenges, this survey aims to give insight into current and future research that turns the raw power of LFMs into partnerships that are reliable, trustworthy, and beneficial to society.

arXiv

RedLight-VLA: Models for traffic-rule grounding and behavioral emphasis in driving policies

Bala Murali Manoghar Sai Sudhakar, Sourab Bapu Sridhar, Sandipan Das, Rahul Ahuja, Meda Lazar, Ashish Garg, Pratik Likhar, Senthil Yogamani

Behavior-cloned Vision-Language-Action (VLA) driving policies struggle with rare rule-governed maneuvers at signalized intersections. Braking and launching examples contribute little to averaged trajectory loss, while fused representations lack explicit supervision for the governing traffic-light and stop-line state. We present RedLight-VLA, a training objective that uses expert futures and automatically generated perception targets without additional manual rule annotation. First, trajectory-derived behavioral reweighting (BR) emphasizes rare deceleration and acceleration using rotation-invariant longitudinal dynamics and a scale-preserving reduction that exactly recovers the baseline when disabled. Second, parallel auxiliary (AUX) heads ground traffic-light and stop-line state in continuous post-fusion rule tokens, without autoregressive language generation or changes to the trajectory decoder. We evaluate on a curated set of 20 s sequences with a 5 s prediction horizon. Controlled variants share the same backbone, training data, decoder, and evaluation population. Against an otherwise identical VLA baseline, RedLight-VLA reduces red-light stop-line overshoot from 7.3% to6.8%, reduces stop-line velocity error by 12.7%, and improves 3 s trafficlight-sliced ADE/FDE from 0.274/0.964 m to 0.247/0.897 m. Green-light false stops increase from 3.2% to 3.9%; however, combining BR with AUX supervision mitigates the larger increase observed for AUX alone (4.0%). The combined model also improves non-traffic-light ADE/FDE from 0.268/0.956 m to 0.241/0.876 m and outperforms either mechanism alone on all four sliced displacement measures.

arXiv

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu

Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks. Existing benchmarks such as AndroidWorld and MobileWorld provide strong foundations for mobile agent evaluation, but their application coverage and task design do not yet fully capture the diversity and complexity of realistic mobile use. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios. GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. We evaluate eight frontier models and find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements. We further conduct controlled ablation studies of agent harness choices, including context retention and explicit state tracking, under a shared environment, model setting, and task taxonomy. Results show that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models. Overall, GMA complements existing benchmarks by expanding application coverage and task complexity, providing a challenging testbed for evaluating mobile agents and studying how harness design supports reliable execution in complex mobile workflows.

arXiv

Test-Time Scaling for Scientific Equation Discovery

Haowei Lin, Hubert Lim, Xiangyu Wang, Letian Huang, Di He

Test-time scaling (TTS) improves language model reasoning by allocating additional test-time compute, but prior work mainly studies closed-ended tasks such as math and coding. We study TTS for automated equation discovery, an open-ended setting where models search over candidate equations and rely on observed datapoints for feedback. We formulate LLM-driven equation discovery as an iterative search process that unifies Best-of-N, sequential refinement, tree search, and evolution-style methods under a common compute-allocation view. To isolate allocation effects from prompt engineering and other heuristics, we compare minimal parallel controllers under fixed budgets. On LLM-SRBench equation-discovery tasks, we find that search width is the dominant allocation parameter: the best width in our sweep generally increases with the compute budget, while the population--branching split and controller choice matter less. Appropriate width selection also improves wall-clock efficiency by increasing parallelism. These results suggest that, given an informative verifier, controlling exploration and exploitation is central to scaling LLM-based equation discovery.

arXiv

Bounded Precision-Geometry Scaling for Robust Multi-Task Learning under Loss Scale Mismatch

Krishna Subedi

Multi-task learning often combines losses that span several orders of magnitude, causing homoscedastic uncertainty weighting to degrade severely. We propose Bounded Precision-Geometry Scaling (BPGS), a method that maps each task's log-variance through a bounded sigmoid parameterisation anchored to detached batch loss statistics, and decouples network optimisation from uncertainty optimisation. Its normalised task weights are provably invariant to uniform rescaling under non-degenerate loss scales. We evaluate BPGS on synthetic stress tests and three real-world benchmarks: NYUv2 dense prediction, Yeast multi-label classification, and RF1 multi-target regression. Under pure loss rescaling from $\times 1$ to $\times 1000$, its macro score changes from 0.777 to 0.778, whereas Kendall weighting drops from 0.780 to 0.637; $\ell_1$-normalising Kendall's weights does not close the gap. On NYUv2, BPGS records the lowest depth absolute relative error (0.223), depth RMSE (0.790), and total loss (1.891) among all compared methods, including Nash-MTL. Sensitivity studies on batch size and calibration show small variation across the tested ranges, and runtime overhead relative to Kendall is under 1%. BPGS posts the highest Yeast micro-F1 (0.616) and is competitive on RF1, though PCGrad leads RMSE and MAE there. These findings establish BPGS as a scale-robust alternative to homoscedastic uncertainty weighting, notably effective when loss-scale disparities dominate multi-task optimisation.

arXiv

Semantic Slots for Video Object-Centric Learning

Khalil Sabri, Guillaume-Alexandre Bilodeau, Nicolas Saunier, Wassim Bouachir

Video Object-Centric Learning (OCL) has traditionally focused on refining the encoder architecture to ensure temporal consistency. In this paper, we argue that the primary bottleneck lies in the decoder. We show that traditional decoders force slots to be spatially anchored, hindering their ability to adapt to motion. We propose SemanticSlots, which uses a Transformer-based decoder that leverages image context, relieving slots from encoding boundary precision and spatial location. This allows slots to function as semantic queries that are inherently object position invariant, retrieving matching features rather than memorizing coordinates. More importantly, this property allows slots computed from a single frame to decompose subsequent video frames, eliminating the need for complex temporal predictors or auxiliary temporal losses. Results on YouTube-VIS show that SemanticSlots improves upon VideoSAUR by 31 points in mBO and outperforms current state-of-the-art methods by 21 points, achieving 86.6% ARI and 62.8% mBO.

arXiv

Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Michael Jerge, Joseph Pelczar, Justin Downes

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar $p \le 0.001$) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.

arXiv

Extending the Horizon of Early Diagnosis: Lung Cancer Prediction with Vision Transformers

Olivera Kotevska, Ian Goethert, Michael McGee, Maria Mahbub, Sean R. Wilkinson, Rowena Yip, Myvizhi Esai Selvan, Zeynep H. Gumus, Claudia Henschke, Robert J. Klein, Providencia Morales, Samuel M Aguayo, Ioana Danciu, Mayanka Chandrashekar

Lung cancer remains a leading cause of cancer-related mortality worldwide, and early diagnosis is critical for improving survival. However, early-stage malignancies can be subtle on chest X-rays, creating challenges for radiologists. This study evaluates Vision Transformers (ViTs) for predicting lung cancer one to two years before clinical diagnosis. We analyzed 259,361 chest X-rays from 91,020 imaging studies at the Jamaica Plains VA Hospital in Boston, MA. The dataset showed extreme class imbalance, approximately 1:150 cancer to non-cancer, which was addressed using hybrid under- and over-sampling and class-weighted loss optimization. Three ViT configurations were evaluated: a model trained from scratch, an ImageNet-pretrained model, and a Corona-pretrained model fine-tuned on the lung cancer dataset. Transfer learning improved performance, with pretrained models exceeding the scratch baseline by 6-10 percentage points in AUC and about 10-12 percent in balanced accuracy. ImageNet-pretrained models showed the most stable overall performance, while Corona-pretrained models achieved higher sensitivity in some settings but greater variability. Moderate resampling ratios, including 1:1 undersampling and 1.5:2 oversampling, provided favorable trade-offs between sensitivity, precision, and computational efficiency, reducing runtime by up to 70 percent without major performance loss. These findings demonstrate the potential of ViTs for early lung cancer risk prediction from routine chest X-rays. Although performance remains below clinical deployment thresholds, the results support further development of ViT-based triage systems to flag high-risk patients for earlier evaluation.

arXiv

presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Search

Weixuan Ding, Shang Liu, Hanyu Pei, Zeyan Liu

Object placement is critical in image composition, requiring spatially and semantically coherent positioning of objects within diverse scenes. Existing approaches typically rely on hand-crafted rules or supervised learning on limited datasets, which restricts their generalization and interpretability, especially in open-world scenarios involving novel objects and scenes. In this work, we reformulate open-world object placement as a heuristic search task guided by reasoning from a Multimodal Large Language Model (MLLM). We introduce \textsf{presto}, a zero-shot, training-free framework that operates within an imaginary action space to iteratively refine object position and scale. Our coarse-to-fine search strategy ensures fast convergence, and we evaluate two decision-making variants: Metric-guided Selection and MLLM-as-a-judge. Experiments across multiple benchmarks show that \textsf{presto}~achieves state-of-the-art performance, particularly in previously unseen, open-world settings. Human studies further reveal that the MLLM-as-a-judge variant produces more perceptually coherent placements than metric-driven approaches, highlighting a gap between standard evaluation metrics and human visual judgment.

arXiv

CHIMERA Challenge: Biochemical Recurrence Prediction in Prostate Cancer Patients using multimodal datasets

Robert N. Spaans, Catherine Chia, Tongjie Wang, Adam Kowalewski, Parandzem Khachatryan, Domingos Oliveira, Khrystyna Faryna, Jean-Paul A. van Basten, Geert Litjens, Nadieh Khalili

Biochemical recurrence (BCR), defined as any detectable prostate-specific antigen level after prostatectomy with confirmatory elevation, is widely used as a surrogate endpoint and typically assessed using clinical and pathological variables. Currently, no standardized benchmark exists for multimodal prognostic modeling in urological cancers, partly because curating heterogeneous multimodal data remains challenging. We developed the CHIMERA Challenge, a multimodal benchmark integrating preoperative mpMRI, post-prostatectomy histopathology, patient characteristics, and clinician-derived variables from 267 patients across two institutions. The dataset comprises 801 MRI sequences, 13 clinical variables per case, and 942 WSIs. Training (n=95), validation (n=23), and test (n=149) splits were established and hosted on the Grand Challenge platform. Baseline clinical and pathological characteristics did not differ significantly across splits. Models were evaluated on predicting time to BCR using the C-index. Post-challenge analyses tested how each model type performed when clinician-derived variables were withheld or randomized. Unimodal clinical models achieved the highest test C-index of 0.7402 but proved sensitive to the integrity of these variables, with performance collapsing toward chance (C approximately 0.50) when they were randomized. Multimodal models retained near-baseline performance when these variables were withheld (delta C at most 0.04), indicating their ability to recover prognostic signal directly from imaging data. CHIMERA is the first public, standardized multimodal benchmark for prostate cancer prognosis. Although models using only patient characteristics and clinician-derived variables yielded the highest leaderboard performance, multimodal models demonstrated greater robustness in clinically realistic scenarios where complete expert annotation is not guaranteed.

arXiv

MDFI: A Multi-Domain Features Integration for Compressed Video Quality Enhancement

Sang NguyenQuang, Hieu Bui Minh, Dang BuiDinh, Xiem HoangVan

The latest video coding standard, H.266/VVC, has demonstrated significant improvements in compression efficiency compared to H.265/HEVC. Despite its advanced coding techniques, H.266/VVC still faces challenges in meeting the increasing demand for higher perceptual quality and enhanced compression performance. To address these limitations, we propose MDFI (Multi-Domain Features Integration), a compressed video quality enhancement approach that features a novel Frame-Prediction Feature Transform (FPFT) module to process prediction information. Moreover, MDFI integrates a multi-domain feature fusion strategy that effectively combines spatiotemporal characteristics, cross-frequency representations, and compressed-domain prediction information to enhance decoded video quality. Additionally, we introduce a comprehensive dataset that encompasses uncompressed video sequences, corresponding reconstructed versions at multiple QP levels, and predicted frames generated from H.266/VVC compressed bitstreams, providing essential resources for developing and benchmarking video enhancement approaches. Extensive experiments demonstrate that our MDFI approach achieves superior performance to state-of-the-art methods in both objective metrics and visual quality, effectively mitigating video compression artifacts. The code is available at: https://github.com/dangdinh17/MDFI.git.

arXiv

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models

Chang Sun, Francesco Barbato, Matteo Caligiuri, Pietro Zanuttigh

Vision-Language Models (VLMs) exhibit strong zero-shot capabilities, making them an attractive solution for continual learning across diverse tasks. However, during continual adaptation, both catastrophic forgetting and zero-shot degradation occur, severely degrading performance. In this paper, we introduce TASSO, a new paradigm that efficiently preserves the latent space geometry while ensuring network plasticity. We achieve this with two complementary techniques: subspace learning and geometry-aware knowledge distillation. Specifically, we first learn a sequence of task-specific low-rank projectors, which we use to project the latent representations before optimizing cross-entropy. Secondly, we employ a geodesic-distance-based loss that distills knowledge from the previous-task model while effectively preserving the latent space geometry. These design choices not only avoid unnecessary parameter updates along the full embedding dimensions but also improve learning by focusing on task-specific manifolds. Moreover, the geometry-aware distillation provides strong regularization and significantly reduces both catastrophic forgetting and zero-shot degradation throughout the continual learning sequence. Experimental results with the CLIP vision language model in the multi-domain task incremental and class incremental learning benchmarks demonstrate clear improvements over state-of-the-art methods in mitigating forgetting and preserving zero-shot capabilities.

arXiv

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer, Fadi Boutros

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.

arXiv

Reliability- and Anatomy-Consistency-Aware Multimodal Learning for Robust Fracture Classification from Bangladeshi Radiographs

Musa Tur Farazi, K G Subarno Bithi

Background: Multimodal fracture classifiers may benefit from patient and anatomical metadata, but they can also become brittle when contextual information is missing or mismatched. Methods: We studied 1493 radiographs from the Bangladeshi OrthoFrac-XR dataset using leakage-safe age, sex, bone type, and laterality. A ConvNeXt image encoder was combined with a clinical multilayer perceptron through concatenation, late fusion, reliability-gated residual fusion, and a hierarchical state-location formulation. We additionally introduced an anatomy-consistency gate that attenuates metadata corrections when an image-side anatomical prediction disagrees with the reported bone type. Results: Across five folds and three seeds, hierarchical residual fusion achieved a macro-F1 of 0.6046 +/- 0.0279, compared with 0.5727 +/- 0.0270 for image-only learning, while improving the Brier score from 0.5239 to 0.4948. In a five-fold robustness experiment, anatomy-consistency fusion reduced the macro-F1 loss under shuffled metadata from 0.0567 to 0.0203 relative to ordinary residual fusion, although its clean-data macro-F1 was lower. Without bone type at inference, auxiliary anatomy supervision improved macro-F1 from 0.5620 +/- 0.0330 to 0.5899 +/- 0.0289. Conclusions: Structured context improves fracture classification, and consistency-aware gating limits harm from mismatched metadata. The observed clean-performance-robustness trade-off and the absence of patient-level identifiers motivate external and prospective validation.

arXiv

Multimodal pseudo-CT synthesis for PET attenuation correction using separate modality encoding and topogram conditioning

Rory Bell, Artemis Bouzaki, Jiaming Cao, Jasmine Morrison, Chelsea Sargeant

We participated in the BIC-MAC Challenge with a multimodal 3D patch-based U-Net for pseudo-CT generation from NAC-PET, MRI, and 2D topograms. By using separate PET and MR encoders, multi-scale feature fusion, and FiLM-based topogram conditioning at the bottleneck, we obtain a model that integrates complementary cross-modal information while reducing reliance on precise voxel-wise correspondence between modalities. Our final submission can be found: https://github.com/rrr-uom-projects/BIC-MAC-MICCAI2026

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.