ACML 2023 Papers

Models with similar performances exhibit significant disagreement in the predictions of individual samples, referred to as prediction churn. Our work explores this phenomenon in graph neural networks by investigating differences between models differing only in their initializations in their utilized features for predictions. We propose a novel metric called Influence Difference (ID) to quantify the variation in reasons used by nodes across models by comparing their influence distribution. Additionally, we consider the differences between nodes with a stable and an unstable prediction, positing that both equally utilize different reasons and thus provide a meaningful gradient signal to closely match two models even when the predictions for nodes are similar. Based on our analysis, we propose to minimize this ID in Knowledge Distillation, a domain where a new model should closely match an established one. As an efficient approximation, we introduce DropDistillation (DD) that matches the output for a graph perturbed by edge deletions. Our empirical evaluation of six benchmark datasets for node classification validates the differences in utilized features. DD outperforms previous methods regarding prediction stability and overall performance in all considered Knowledge Distillation experiments.

Medical visual question answering (Med-VQA) aims to correctly answer the medical question based on the given image. One of the major challenges is the scarcity of large professional labeled datasets for training, which poses obstacles to feature extraction, especially for medical images. To overcome it, we propose a method to learn transferable visual representation based on conditional denoising diffusion probabilistic model(conditional DDPM).Specifically, we collate a large amount of unlabeled radiological images and train a conditional DDPM with the paradigm of auto-encoder to obtain a model which can extract high-level semantic information from medical images.The pre-trained model can be used as a well initialized visual feature extractor and can be easily adapt to any Med-VQA systems. We build our Med-VQA system follow the state-of-the-art Med-VQA architecture and replace the visual extractor with our pre-trained model.Our proposal method outperforms the state-of-the-art Med-VQA method on VQA-RAD and achieves comparable result on SLAKE.

With the rapid development of deep learning and its pervasive usage on various low-power and resource-constrained devices, model compression methods are increasingly used to reduce the model size and computation cost. Despite the overall high test accuracy of the compressed models, our observation shows that an original model and its compressed version (e.g., via quantization) can have deviated prediction outputs on the same inputs. These behavior deviations on compressed models are undesirable, given that the compressed models may be used in reliability-critical scenarios such as automated manufacturing and robotics systems. Inspired by software engineering practices, this paper proposes CompD, a differential testing (DT)-based framework for detecting and repairing prediction deviations on compressed models and their plaintext versions. CompD treats original/compressed models as “black-box,” thus offering an efficient method orthogonal to specific different compression schemes. Furthermore, CompD can leverage deviation-triggering inputs to finetune the compressed models, largely “repairing” their defects. Evaluations show that CompD can effectively test and repair common models compressed by different schemes.

Despite much research, Graph Neural Networks (GNNs) still do not display the favorable scaling properties of other deep neural networks such as Convolutional Neural Networks and Transformers. Previous work has identified issues such as oversmoothing of the latent representation and have suggested solutions such as skip connections and sophisticated normalization schemes. Here, we propose a different approach that is based on a stratification of the graph nodes. We provide motivation that the nodes in a graph can be stratified into those with a low degree and those with a high degree and that the two groups are likely to behave differently. Based on this motivation, we modify the Graph Neural Network (GNN) architecture so that the weight matrices are learned, separately, for the nodes in each group. This simple-to-implement modification seems to improve performance across datasets and GNN methods. To verify that this increase in performance is not only due to the added capacity, we also perform the same modification for random splits of the nodes, which does not lead to any improvement.

This study introduces a new approach for open set recognition, wherein we propose a novel method utilizing uniformly distributed centers on a hypersphere. Each class in the proposed method is represented by a center, and these centers and features of the deep learning architecture are jointly learned from the training data in an end-to-end fashion. We ensure that the centers lie on the boundary of a hypersphere whose center is positioned at the origin. The class-specific samples are compelled by the proposed loss function to be closer to their respective centers. In open set recognition scenarios, an additional loss term is employed to separate the background samples from the known class centers. The assignment of test samples to classes is based on the Euclidean distances calculated from the learned class centers. Experimental results show that the proposed method yields the state-of-the-art accuracies on open set recognition datasets.

Predicting human driving behavior has always been an important area of autonomous driving research. Existing data on autonomous driving in this area is limited in both perspective and duration. For example, vehicles may block each other on the road, although data from vehicles behind them is useful for research. In addition, driving in this area is constrained by the road environment, and the host vehicle cannot observe the designated area for an extended period of time. To investigate the potential relationship between human driving behavior and traffic conditions, we provide a drone-collected video dataset, Deep Traffic, that includes: (1) aerial footage from a vertical perspective, (2) image and annotation capture for training vehicle destination detection and semantic segmentation model, (3) high-definition map data of the captured area, (4) development scripts for various features. Deep Traffic is the largest and most comprehensive dataset to date, covering both urban and high-speed areas. We believe that this benchmark dataset will greatly facilitate the development of drones to monitor traffic flow and study human driver behavior, and that the capacity of the traffic system is of great importance. All datasets and pre-training results can be downloaded from github project.

Temporal Point Processes (TPP) play an important role in predicting or forecasting events. Although these problems have been studied extensively, predicting multiple simultaneously occurring events can be challenging. For instance, more often than not, a patient gets admitted to a hospital with multiple conditions at a time. Similarly people buy more than one stock and multiple news breaks out at the same time. Moreover, these events do not occur at discrete time intervals, and forecasting event sets in the continuous time domain remains an open problem. Naïve approaches for extending the existing TPP models for solving this problem lead to dealing with an exponentially large number of events or ignoring set dependencies among events. In this work, we propose a scalable and efficient approach based on TPPs to solve this problem. Our proposed approach incorporates contextual event embeddings, temporal information, and domain features to model the temporal event sets. We demonstrate the effectiveness of our approach through extensive experiments on multiple datasets, showing that our model outperforms existing methods in terms of prediction metrics and computational efficiency. To the best of our knowledge, this is the first work that solves the problem of predicting event set intensities in the continuous time domain by using TPPs.

As a growing number of warehouse operators are moving from human-only to Collaborative human-robot Order Picking solutions, more efficient picker routing policies are needed, since the complexity of coordinating multiple actors in the system increases significantly. The objective of these policies is to match human pickers and robot carriers to fulfill picking tasks, optimizing pick-rate and total tardiness of the orders. In this paper, we propose to formulate the order picking routing problem as a more general combinatorial optimization problem known as Two-sided Online Bipartite Matching. We present an end-to-end Deep Reinforcement Learning approach to optimize a combination of pick-rate and order tardiness, and to deal with the uncertainty of real-world warehouse environments. To extract and exploit spatial information from the environment, we devise three different Graph Neural Network architectures and empirically evaluate them on several scenarios of growing complexity in a simulation environment we developed. We show that all proposed methods significantly outperform greedy and more sophisticated heuristics, as well as non-GNN-based DRL approaches. Moreover, our methods exhibit good transferability properties, even when scaling up test problem instances to more than forty times the size of the ones the models were trained on. Code is available at: \url{https://github.com/lbegnardi/DRL-TOBM-CPR}.

In this work, we study kernel regression that integrates with a modern deep neural network (DNN). The DNN projects the input into an embedding space, meanwhile a set of representative points is constructed in this embedding space. We build the regression using a finite set of kernels defined on the embedding space, where the DNN weights, the regression coefficients, and kernel hyperparameters are all learnable. We extend the model by introducing a location attention strategy and the multiple kernel technique. We provide effective ways to obtain representative points. The proposed model can be trained with an end-to-end learning algorithm with simple implementation. Simulation studies show that the proposed deep kernel regression is well scalable to large datasets and comparable to or superior to recent deep kernel models in various regression problems.

With the popularity of electronic mobile and online payment, the demand for detecting financial fraudulent transactions is increasing. Although numerous efforts are devoted to tackling this problem, there are still two key challenges that are not well resolved, \emph{i.e.}, the class imbalance ratio of test samples are extremely larger than that of training samples and amount of detected fraudulent transactions do not be considered. In this paper, we propose a simple and effective framework composed of majority and minority branches to address the above issues. The input samples of majority and minority branches come from vanilla and re-adjusted distribution, respectively. Parameters of each branch are optimized individually, by which the representation learning for majority and minority samples are decoupled. Besides, an extra loss re-weighted by amount is added in the majority branch to improve the recall amount of detected fraudulent transactions. Theoretical results show that under the proposed framework, minimizing the empirical risk is guaranteed to achieve small generalization risk on more imbalanced data with high probability. Experiments on real-world datasets from Tencent Wechat payments demonstrate that our framework achieves superior performance than competitive methods in terms of both number and money of detected fraudulent transactions.

Neural Networks (NN) are increasingly used for image classification in medical, transportation, and security devices. However, recent studies have revealed neural networks’ vulnerability against adversarial examples generated by adding small perturbations to images. These malicious samples are imperceptible by human eyes, but can give rise to misclassification by NN models. Defensive distillation is a defence mechanism in which the NN’s output probabilities are scaled to a user-defined range and used as labels to train a new model less sensitive to input perturbations. Despite initial success, defensive distillation was defeated by state-of-the-art attacks. A proposed countermeasure was to add noise in the inference time to hamper the adversarial attack which also decreased the model accuracy. In this paper, we address this limitation by proposing a two-phase training methodology to defend against adversarial attacks. In the first phase, we train architecturally diversified models individually using the cross-entropy loss function. In the second phase, we train the ensemble using a diversity-promoting loss function. Our experimental results show that our training methodology and noise addition in the inference time improved our ensemble’s resistance against adversarial attacks, while maintaining reasonable accuracy, compared to the state-of-the-art methods.

We consider the challenge of establishing relationships between samples in distinct domains, A and B, using supervised data that captures the intrinsic relationships within each domain. In other words, we present a semi-supervised setting in which there are no labeled mixed-domain pairs of samples. Our method is derived based on a generalization bound and incorporates supervised terms for each domain, a domain confusion term on the learned features, and a consistency term for domain-specific relationships when considering mixed-domain sample pairs. Our findings showcase the efficacy of our approach in two disparate domains: (i) Predicting protein-protein interactions between viruses and hosts by modeling genetic sequences. (ii) Forecasting link connections within citation graphs using graph neural networks.

In this work, we propose “Cell Variational Information Bottleneck Network (cellVIB)”, a convolutional neural network using information bottleneck mechanism, which can be combined with the latest feedforward network architecture in an end-to-end training method. Our Cell Variational Information Bottleneck Network is constructed by stacking VIB cells, which generate feature maps with uncertainty. As layers going deeper, the regularization effect will gradually increase, instead of directly adding excessive regular constraints to the output layer of the model as in Deep VIB. In each VIB cell, the feedforward process learns an independent mean term and a standard deviation term, and predicts the Gaussian distribution based on them. The feedback process is based on reparameterization trick for effective training. This work performs an extensive analysis on MNIST dataset to verify the effectiveness of each VIB cells mentioned above, and provides an insightful analysis on how the VIB cells affect mutual information. Experiments conducted on CIFAR-10 also prove that our network is robust against noisy labels during training and against corrupted images during testing. Then, we validate our method on PACS dataset, whose results show that the VIB cells can significantly improve the generalization performance of the basic model. Finally, in a more complex representation learning task, face recognition, our network structure has also achieved very competitive results.

In Multi-label Learning (MLL), kernel methods and deep neural networks (DNNs) are two typical families of approaches. Recent theory discovers an interesting connection between infinitely wide DNNs and neural tangent kernel (NTK) based methods. Further, recent work has shown the promising performance of NTK-based methods in \emph{small-data single-labeled tasks}. Then, a natural question arises: can infinitely wide DNNs help small-data multi-label learning? To answer this question, in this paper, we present to utilize infinitely wide DNNs for the MLL task. Specifically, we propose an NTK-based kernel method for MLL, which aims to minimize Hamming and ranking loss simultaneously. Moreover, to efficiently train the model, we use the Nystr{ö}m method, which has rarely been used in MLL. Further, we give rigorous theoretical analyses on learning guarantees of the proposed algorithm w.r.t. these two measures. Finally, empirical results on small-scale datasets illustrate its superior performance along with efficiency over several related baselines.

The loss landscape of neural networks is a valuable perspective for studying the trainability, generalization, and robustness of networks, and hence its visualization has been extensively studied. Essentially, visualization methods project the parameter space into a low-dimensional subspace, resulting in a substantial loss of network parameter information. The key is to identify the direction of loss reduction in the visualized loss landscape. However, the existing methods generally focus on one simple point, make it challenging to properly capture the main properties of the landscape. An obvious and important problem is that regardless of whether the center point is the convergence or not, the current methods may depict it a local optimal point in the visualization. To address this issue, we propose a visualization method that relies on the whole training process not a single solution, better reflecting the actual training loss.

This paper introduces BarlowRL, a data-efficient reinforcement learning agent that combines the Barlow Twins self-supervised learning framework with DER (Data-Efficient Rainbow) algorithm. BarlowRL outperforms both DER and its contrastive counterpart CURL on the Atari 100k benchmark. BarlowRL avoids dimensional collapse by enforcing information spread to the whole space. This helps RL algorithms to utilize uniformly spread state representation that eventually results in a remarkable performance. The integration of Barlow Twins with DER enhances data efficiency and achieves superior performance in the RL tasks. BarlowRL demonstrates the potential of incorporating self-supervised learning techniques to improve RL algorithms.

Hypertrophic obstructive cardiomyopathy (HOCM) is a leading cause of sudden cardiac death in young people. Septal myectomy surgery has been recognized as the gold standard for non-pharmacological therapy of HOCM, in which aortic and mitral valves are critical regions for surgical planning.Currently, manual segmentation of aortic and mitral valves is widely performed in clinical practice to construct 3D models used for HOCM surgical planning. Such a process, however, is time-consuming and costly. In this paper, we integrate anatomical prior knowledge into deep learning for automatic segmentation of aortic and mitral valves.In particular, a two-stage method is proposed: we first obtain the region of interest (RoI) from a CT image, where heart segmentation is then performed. The spatial relationship between heart substructures is utilized to identify a valve region that contains the aortic and mitral valves. Unlike typical two-stage methods, we feed the refined segmentation of the left ventricle, left atrium, and aorta as additional input for the valve segmentation. By incorporating this anatomical prior knowledge, deep neural networks (DNNs) can leverage the surrounding anatomical structures to improve valve segmentation. We collected a dataset of 27 CT images from patients with a medical history of septal myectomy surgery.Experimental results show that our method achieves an average Dice score of 71.2% and an improvement of 4.2% over existing methods.

Unsupervised learning allows us to leverage unlabelled data, which has become abundantly available, and to create embeddings that are usable on a variety of downstream tasks. However, the typical lack of interpretability of unsupervised representation learning has become a limiting factor with regard to recent transparent-AI regulations. In this paper, we study graph representation learning and we show that data augmentation that preserves semantics can be learned and used to produce interpretations. Our framework, which we named INGENIOUS, creates inherently interpretable embeddings and eliminates the need for costly additional post-hoc analysis. We also introduce additional metrics addressing the lack of formalism and metrics in the understudied area of unsupervised-representation-learning interpretability. Our results are supported by an experimental study applied to both graph-level and node-level tasks and show that interpretable embeddings provide state-of-the-art performance on subsequent downstream tasks.

Attributed graph clustering groups nodes into disjoint categories with graph convolutional networks and has exhibited promising performance in various applications. However, there are two issues preventing the performance from being improved further. First, the relationships between distant nodes are generally overlooked due to the sparsity of graphs. Second, the graph convolutional networks with few layers are sensitive to noises. To address these issues, we propose Attributed Graph Subspace clustering with Graph-Boosting (AGSGB). Specifically, to deal with the first issue, an auxiliary graph is built from the feature matrix to establish the distant relationships. And to address the second issue, a subspace clustering module, famous for its robustness to noise, is introduced. Based on the auxiliary graph and the subspace clustering module, a graph enhance module and a graph refine module are constructed, together with the graph autoencoder constituting the final clustering model. By using the given graph and the refined graph built by the graph refine module, a dual guidance supervisor is designed to train the clustering model. Finally, the clustering result can be obtained by the subspace clustering module. Extensive experimental results on five public benchmark datasets validate the effectiveness of the proposed method.