ACML 2024 Papers

Re-ranking is the last part of a multi-stage recommendation system, involving the reordering of lists based on historical user behavior to better align with user preferences. Offline Reinforcement Learning (RL) has been employed in both the prediction and ranking phases of recommendation systems to align with long-term objectives, surpassing the efficacy of supervised learning. However, extrapolation error is a common problem in offline RL, due to the biased distribution of features, which can lead to the reduction of recommendation accuracy. Consider that as users browse an e-commerce app, their preferences are influenced by previously recommended items or pages, therefore the history can be used to correct the bias of offline RL. This paper uses offline RL to model re-ranking and presents a re-ranking algorithm named Page and Item Sequential Decision for Re-ranking (PISDR) to improve accuracy by correcting bias at two levels (pages and items). PISDR employs sequential RL, leveraging a session-level data structure that encapsulates global information at the page level and item-level interrelationships. Additionally, PISDR utilizes a multi-tower critic network to assess various feedback metrics, including click-through rate, conversion rate, etc. which can raise actor network from the long-term reward. Experimental results validate the effectiveness of PISDR in significantly enhancing of Area Under Curve (AUC), Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) about 1.4% in generated re-ranking sequences when compared to current state-of-the-art re-ranking algorithms. Finally, as a consequence, our method achieves a significant improvement (2.59%) in terms of Click-Through Rate (CTR) over the industrial-level ranking model in online A/B tests.

Learning in biological systems involves the intricate modeling of diverse entities and their interrelations, leading to the evolution of logical knowledge networks with accumulating experience. Analogously, knowledge graphs serve as semantic representations of entity relationships, playing a vital role in natural language processing and graph representation learning. However, contemporary knowledge graph embedding models often neglect real-world event updates, while existing continual knowledge graph research predominantly relies on conventional learning methods that inadequately leverage graph structure, thereby compromising their continual learning capabilities. This study introduces a novel Continual Mask Knowledge Graph Embedding framework (CMKGE), designed to address these limitations. CMKGE integrates semantic attributes, network structure, and continual learning mechanisms to capture the dynamic evolution of knowledge. Inspired by biological signal propagation and Dale’s principle, we introduce a dual-mask mechanism for neuronal inhibition and activation. This mechanism automatically filters critical old knowledge, enhancing model plasticity and stability. Through comprehensive evaluations on four datasets, we demonstrate CMKGE’s superiority over state-of-the-art continual embedding models.

Ethical and privacy issues inherent in artificial intelligence (AI) applications have been a growing concern with the rapid spread of deep learning. Machine unlearning (MU) is the research area that addresses these issues by making a trained AI model forget about undesirable training data. Unfortunately, most existing MU methods incur significant time and computational costs for forgetting. Therefore, it is often difficult to apply these methods to practical datasets and sophisticated architectures, e.g., ImageNet and Transformer. To tackle this problem, we propose a lightweight and effective MU method. Our method identifies the model parameters sensitive to the forgetting targets and adds perturbation to such model parameters. We identify the sensitive parameters by calculating the Fisher Information Matrix (FIM). This approach does not require time-consuming additional training for forgetting. In addition, we introduce class-specific random signals called mnemonic code to reduce the cost of FIM calculation, which generally requires the entire training data and incurs significant computational costs. In our method, we train the model with mnemonic code; when forgetting, we use a small number of mnemonic codes to calculate the FIM and get the effective perturbation for forgetting. Comprehensive experiments demonstrate that our method is faster and better at forgetting than existing MU methods. Furthermore, we show that our method can scale to more practical datasets and sophisticated architectures.

We present a differentially private learner for half-spaces over a finite domain $\mathcal{X}^d\subseteq\mathbb{R}^d$ with sample complexity $\tilde{O}(d\cdot\log^*\vert\mathcal{X}\vert)$, which improves over $\tilde{O}(d^{2.5}\cdot 8^{\log^*\vert\mathcal{X}\vert})$, the state-of-the-art result of [Kaplan et al., 2020]. The building block of our result is a novel differentially private algorithm that learns half-spaces by iteratively learning thresholds on angles.

The goal of Synthetic Speech Detection (SSD) is to detect spoofing speech synthesized by text-to-speech and voice conversion. Most existing SSD methods focus only on mining frequency-wise dependency by customizing frequency-aggregation modules in SSD models. However, the instance-wise dependency is usually under-explored, which is critical for identifying the synthetic speech in a global view. In this paper, we propose a novel model-agnostic training strategy for SSD that exploits both local (frequency-wise) and global (instance-wise) contexts, which do not rely on a customized architecture and can be flexibly integrated into previous SSD models. Specifically, we propose an inter-frequency correlation module to capture the local context by reconstructing the masked frequency information from the unmasked frequency context. Meanwhile, an inter-instance correlation module is performed to explore the global context among different instances by promoting intra-class compactness and inter-class dispersion in the latent space. These two complementary modules operate from distinct contextual perspectives, leading to improvements in SSD performance. Extensive experiments show that our method significantly improves the performance of two state-of-the-art models on the 2019 dataset and 2021 dataset of ASVspoof.

Random projections are a cornerstone of high-dimensional computations. However, their analysis has proven both difficult and inadequate in capturing the empirically observed accuracy. To bridge this gap, this paper studies random projections from a novel perspective, focusing on data-dependent, that is, \emph{non-oblivious}, performance. The key contribution is the precise and data-dependent accuracy analysis of Rademacher random projections, achieved through elegant geometric methods of independent interest, namely, \emph{Schur-concavity}. The result formally states the following property: the less spread-out the data is, the better the accuracy. This leads to notable improvements in accuracy guarantees for data characterized by sparsity or distributed with a small spread. The key tool is a novel algebraic framework for proving Schur-concavity properties, which offers an alternative to derivative-based criteria commonly used in related studies. We demonstrate its value by providing an alternative proof for the extension of the celebrated Khintchine inequality.

Effectively capturing the spatiotemporal dependencies between joints is crucial for skeleton-based action recognition. However, existing methods do not consider the sparsity of skeleton data, which hinders the accurate capture of complex posture information and subtle action variations. Moreover, the locality of temporal features requires the model to focus on certain key features. Yet, most methods overlook the impact of temporal redundancy on feature focus, resulting in ineffective capture of significant temporal features. To address the issue of skeleton sparsity, we propose a Multilevel Position-aware Attention module (MPA) that explicitly leverages the relative positional information of the input data to enrich spatial information. To achieve a more effective focus on local temporal features, we develop a Multi-scale Temporal Excitation module (MTE). By scaling temporal features, the MTE module elevates the prominence of salient features and facilitates the capture of multi-scale features. Furthermore, we propose a Part Partition Encoding module (PPE) to aggregate joint data into part data, thereby providing the model with high-level information carried by the interactions between body parts. The MPA, MTE, and PPE are integrated into a unified framework called MPAE-Net. Extensive experimental results demonstrate that the MPAE-Net achieves state-of-the-art performance on two large-scale datasets, NTU RGB+D and NTU RGB+D 120.

Multivariate Time Series (MTS) forecasting involves analyzing the evolution and interrelationships of multiple variables over time. Effectively mining relationships between MTS variables remains challenging as variables may imply multiple relational patterns. Recently, graph-based approaches have exhibited substantial effectiveness in capturing relationships between MTS variables. However, these methods often adhere to the paradigm of capturing low-level pairwise relationships, which limits their ability to capture other high-level beyond pair-wise relational patterns. To address this issue, we present a synergistic graph learning framework that combines the modeling advantages of graphs and hypergraphs to uncover more comprehensive relational patterns. This framework mainly consists of two parts. Firstly, we introduced a Synergistic Relation Construction module, which incorporates dynamic graph and hypergraph structures to model low-level pairwise and high-level beyond pairwise relationships among variables, representing multi-level relational patterns through obtained adjacency matrices and incidence metrics. Additionally, we developed a Synergistic Relation Learning mechanism, that leverages novel synergistic graph and hypergraph convolutional networks to facilitate spatial dependency interactions across multi-levels, along with temporal convolutional networks to capture more comprehensive spatial-temporal dependencies. We conducted comprehensive experiments on four benchmark datasets, and experimental results demonstrate that our model outperforms the state-of-the-art performance. The source code will be available online.

Fake news detection has become a hot research topic in the multimodal domain. Existing multimodal fake news detection research utilizes a series of feature fusion networks to gather useful information from different modalities of news posts. However, how to form effective cross-modal features? And how cross-modal correlations impact decision-making? These remain open questions. This paper introduces MMFND, a multi-task guided multimodal fusion framework for fake news detection , which introduces multi-task modules for feature refinement and fusion. Pairwise CLIP encoders are used to extract modality-aligned deep representations, enabling accurate measurement of cross-modal correlations. Enhancing feature fusion by weighting multimodal features with normalised cross-modal correlations. Extensive experiments on typical fake news datasets demonstrate that MMFND outperforms state-of-the-art approaches.

Deep Unfolding Networks have emerged as a prominent strategy in compressed sensing image reconstruction, effectively merging optimization techniques with deep learning through end-to-end training of truncated inferences. Despite their advantages, these algorithms generally require extensive iterations and parameters, potentially limited by storage capacity. Additionally, the image-level transmission at each iterative step does not optimally harness the inter-scale feature information available. To address these issues, we introduce a novel approach in this paper: the $\textbf{M}$ulti-$\textbf{S}$cale $\textbf{D}$ual-$\textbf{A}$ttention $\textbf{U}$nfolding $\textbf{N}$etwork ($\textbf{MSDAUN}$) for compressed sensing image reconstruction. We propose a cross-stage multi-scale deep reconstruction module $\textbf{D}$ as an iterative process, which is composed of multiple attention sub-modules. These include Cross Attention Transformer($\textbf{CAT}$) Modules that enhance the reconstruction with multi-channel inertia, thereby facilitating feature-level transmission and robust information exchange. Concurrently, Texture Attention Transformer($\textbf{TAT}$) Modules are designed to meticulously extract salient reconstruction information, subsequently channeling it into the texture path to effectuate the precise prediction of textural regions, thereby contributing to the meticulous restoration of textural details. Our comprehensive experimental evaluation across diverse datasets confirms that MSDAUN surpasses existing state-of-the-art methods. This work presents significant potential for further advancements and applications in inverse imaging problems and optimization models.

Videos contain rich spatio-temporal information. Traditional methods for extracting motion, used in tasks such as action recognition, often rely on visual contents rather than precise motion features. This phenomenon is referred to as ’blind motion extraction’ behavior, which proves inefficient in capturing motions of interest due to a lack of motion-guided cues. Recently, attention mechanisms have enhanced many computer vision tasks by effectively highlighting salient visual areas. Inspired by this, we propose a modified Sigmoid function with learnable slope and shift parameters as an attention mechanism to modulate motion signals from frame differencing maps. This approach generates a sequence of attention maps that enhance the processing of motion-related video content. To ensure temporal continuity and smoothness of the attention maps, we apply pair-wise temporal attention variation regularization to remove unwanted motions (e.g., noise) while preserving important ones. We then perform Hadamard product between each pair of attention maps and the original video frames to highlight the evolving motions of interest over time. These highlighted motions, termed video motion prompts, are subsequently used as inputs to the model instead of the original video frames. We formalize this process as a motion prompt layer and incorporate the regularization term into the loss function to learn better motion prompts. This layer serves as an adapter between the model and the video data, bridging the gap between traditional ’blind motion extraction’ and the extraction of relevant motions of interest. We show that our lightweight, plug-and-play motion prompt layer seamlessly integrates into models like SlowFast, X3D, and TimeSformer, enhancing performance on benchmarks such as FineGym and MPII Cooking 2.

Generating executable code through natural language instructions to drive robotic movements is considered a crucial step towards achieving embodied intelligence. However, in the robotics domain, the scarcity of programming language data often necessitates manually encapsulating high-level APIs to enable Large Language Models(LLMs) to predict code correctly, which is time-consuming and incomplete. Therefore, this paper proposes a three-stage Middle Code Prediction(MCP) scheme, by injecting appropriate prompts at different stages, the LLMs can shift towards predicting middle code that it understands more easily. This middle code can then be converted into the final code through specific scripts, accomplishing the task of generating code in uncommon programming languages automatically and without the need for manually encapsulating high-level APIs. We tested our approach on Hospital Item Transport Dataset(HITD) and found that MCP could improve the mean accuracy of various baseline models to varying degrees, with an overall increase of 31%, while also enhancing the noise resistance of fine-tuned models. We conducted real-world experiments on industrial robotic arms, verifying the feasibility of MCP in scenarios with no API and partial API encapsulation. The method proposed in this paper provides a guideline for code generation in uncommon programming languages within the context of LLMs. Our experimental dataset is available at https://github.com/Ghbbbbb/MCP.

Analyzing time-series data that may contain personal information, particularly in the medical field, presents serious privacy concerns. Sensitive health data from patients is often used to train machine-learning models for diagnostics and ongoing care. Assessing the privacy risk of such models is crucial to making knowledgeable decisions on whether to use a model in production, share it with third parties, or deploy it in patients’ homes. Membership Inference Attacks (MIA) are a key method for this kind of evaluation, however time-series prediction models have not been thoroughly studied in this context. We explore existing MIA techniques on time-series models, and introduce new features, focusing on the seasonality and trend components of the data. Seasonality is estimated using a multivariate Fourier transform, and a low-degree polynomial is used to approximate trends. We applied these techniques to various types of time-series models, using datasets from the health domain. Our results demonstrate that these new features enhance the effectiveness of MIAs in identifying membership, improving the understanding of privacy risks in medical data applications.

Seismic motion simulations enable high-precision predictions but are computationally demanding. This study introduces a deep learning surrogate model using the MLP-Mixer architecture to address this challenge. Traditional models using independent Multi-layer Perceptrons (MLPs) fail to capture spatial correlations, while U-shaped Neural Operators (U-NOs) require high computational costs for high-resolution inputs and outputs. Our proposed model, the Multiple MLP-Mixer (Multi-MLP-Mixer), integrates global and local spatial information through multiple MLP-Mixer blocks and dual patch-wise affine transformations. We demonstrate the effectiveness of our method with simulation data from anticipated megathrust earthquakes in the Nankai Trough, achieving performance comparable to state-of-the-art models with significantly improved computational efficiency.

Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. However, their knowledge is confined to specific training datasets and may reflect the subjective emotional biases of the annotators. Furthermore, acquiring large amounts of labeled data is often challenging in real-world applications. In this paper, we systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task from three paradigms: 1) We fine-tune LVLMs on two CAER datasets, which is the most common way to transfer large models to downstream tasks. 2) We design zero-shot and few-shot patterns to evaluate the performance of LVLMs in scenarios with limited data or even completely unseen. In this case, a training-free framework is proposed to fully exploit the In-Context Learning (ICL) capabilities of LVLMs. Specifically, we develop an image similarity-based ranking algorithm to retrieve examples; subsequently, the instructions, retrieved examples, and the test example are combined to feed LVLMs to obtain the corresponding sentiment judgment. 3) To leverage the rich knowledge base of LVLMs, we incorporate Chain-of-Thought (CoT) into our framework to enhance the model’s reasoning ability and provide interpretable results. Extensive experiments and analyses demonstrate that LVLMs achieve competitive performance in the CAER task across different paradigms. Notably, the superior performance in few-shot settings indicates the feasibility of LVLMs for accomplishing specific tasks without extensive training.

Recently, prompt-based learning has become popular in many Natural Language Processing (NLP) tasks by converting the task into a cloze-style one to smooth out the differences between Pre-trained Language Models (PLMs) and the current task. However, as for relation classification, it is challenging to associate the natural language word that fills in the mask token with relation labels due to the rich semantic information in textual label, e.g. ”org:founded_by”. To address this challenge, this paper presents a novel prompt-based learning method, namely LabelPrompt, for the relation classification task. It is an extraordinary intuitive approach motivated by “GIVE MODEL CHOICES!”. Specifically, we first define additional tokens to represent the relation labels, which regard these tokens as the verbalizer with semantic initialisation and explicitly construct them with a prompt template method. Then, we address the inconsistency between predicted relations and given entities by implementing an entity-aware module that employs contrastive learning. Last, we conduct an attention query strategy to differentiate prompt tokens and sequence tokens. These strategies effectively improve the adaptation capability of prompt-based learning, especially when only a small labelled dataset is available. Extensive experimental results obtained on several bench-marking datasets demonstrate the superiority of our method, particularly in the few-shot scenario. Our code can be found at \url{https://github.com/xerrors/Labelprompt}.

Few-shot Named Entity Recognition (NER) aims to identify named entities using very little annotated data. Recently, prompt-based few-shot NER methods have demonstrated significant effectiveness. However, most existing methods employ multi-round prompts, which significantly increase time and computational costs. Furthermore, current single-round prompt methods are mainly designed for flat NER tasks and are not effective in handling nested NER tasks. Additionally, these methods do not to fully utilize the semantic information of entity labels through prompts. To address these challenges, we propose a novel Label-Prompt-based few-shot nested NER method named LPNER, which not only handles nested NER tasks but also efficiently extracts semantic information of entities through label prompts, thereby achieving more efficient and accurate NER. LPNER first designs a specialized prompt based on a span strategy to enhance label semantics and effectively combines multiple span representations using special mark to obtain enhanced span representations integrated with label semantics. Then, entity prototypes are constructed through prototype network for classifying candidate entity spans. We conducted extensive experiments on five nested datasets: ACE04, ACE05, GENIA, GermEval, and NEREL. In 1-shot and 5-shot tasks, LPNER’s $F_1$ scores mostly outperform baseline models.

The task of multi-hop link prediction within knowledge graphs (KGs) stands as a challenge in the field of knowledge graph analysis, as it requires the model to reason through and understand all intermediate connections before making a prediction. In this paper, we introduce the Knowledge Graph Large Language Model (KG-LLM), a novel framework that leverages large language models (LLMs) for knowledge graph tasks. We first convert structured knowledge graph data into natural language and then use these natural language prompts to fine-tune LLMs to enhance multi-hop link prediction in KGs. By converting the KG to natural language prompts, our framework is designed to learn the latent representations of entities and their interrelations. To show the efficacy of the KG-LLM Framework, we fine-tune three leading LLMs within this framework, including Flan-T5, LLaMa2 and Gemma. Further, we explore the framework’s potential to provide LLMs with zero-shot capabilities for handling previously unseen prompts. Experimental results show that KG-LLM significantly improves the models’ generalization capabilities, leading to more accurate predictions in unfamiliar scenarios.