ACML 2023 Papers

In response to the challenge of representing data from multiple sources, researchers have proposed the use of multiplex graphs as a solution. Multiplex graphs are particularly useful for representing multi-view data, where each layer represents a specific type of interaction. Pillar community detection of multiplex graphs is a clustering application that computes groups of vertices across all layers. Modularity maximization is a popular technique for graph clustering, which has been generalized to multiplex graphs. However, this generalization did not consider the importance of each layer in pillar clustering. This paper presents a new technique called Self Weighted Multiplex Modularity (SWMM), which optimizes the weights associated with each layer and the partition that maximizes the multiplex modularity. The paper proposes two optimization methods, iterative and direct, and demonstrates the effectiveness and robustness of the technique in accurately retrieving clusters even when data is highly missing.

Prediction with the possibility of abstention (or selective prediction) is an important problem for error-critical machine learning applications. While well-studied in the classification setup, selective approaches to regression are much less developed. In this work, we consider the nonparametric heteroskedastic regression problem and develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor. We prove non-asymptotic bounds on the risk of the resulting estimator and show the existence of several different convergence regimes. Theoretical analysis is illustrated with a series of experiments on simulated and real-world data.

CT scans significantly improve analytical competencies but uses X-Rays which will produce ionizing radiation that bring higher radiation to the living tissues. Thus, optimization of CT radiation dose has a significant concern to lower the health risks. Many manufacturers have done a greater contribution by developing technologies to reduce dosage by maintaining image quality by adding noise reduction filters, automatic exposure control, and using many iterative reconstruction algorithms. Image reconstruction algorithms play a vital role in maintaining or improving image quality in reduced-dose CT. The present research work combines the state-of-the-art reconstruction technique Simultaneous Algebraic Reconstruction Technique (SART) with a Residual U-Net network to generate images from limited number of sinograms. The proposed model is trained using sinograms corresponding head and neck and head CT images of 10 patients. The proposed model predicted superior diagnostic quality images with max PSNR of 70.23 and Structural Similarity Index Measure (SSIM) of 0.99. Thus the proposed model, SART-Res-Unet, ensures a very low radiation exposure to a patient during the repeated CT imaging sequence, which is an inevitable part of radiotherapy.

The topic of synthetic graph generators (SGGs) has recently received much attention due to the wave of the latest breakthroughs in generative modelling. However, many state-of-the-art SGGs do not scale well with the graph size. Indeed, in the generation process, all the possible edges for a fixed number of nodes must often be considered, which scales in $\mathcal{O}(N^2)$, with $N$ being the number of nodes in the graph. For this reason, many state-of-the-art SGGs are not applicable to large graphs. In this paper, we present SANGEA, a sizeable synthetic graph generation framework which extends the applicability of any SGG to large graphs. By first splitting the large graph into communities, SANGEA trains one SGG per community, then links the community graphs back together to create a synthetic large graph. Our experiments show that the graphs generated by SANGEA have high similarity to the original graph, in terms of both topology and node feature distribution. Additionally, these generated graphs achieve high utility on downstream tasks such as link prediction. Finally, we provide a privacy assessment of the generated graphs to show that, even though they have excellent utility, they also achieve reasonable privacy scores.

Many research works focus on the robustness of convolutional neural networks (CNNs) on image classification. Diversity loss has been demonstrated to be an effective method to boost robustness. However, the existing diversity losses did not fully consider the strong correlation between regional features when filters are locally activated. They focused on improving filter responses constraint with classification loss. However, diversity loss has deeper optimization space. We explore the combinations of different filter diversity losses and feature diversity losses. We enhance the orthogonality between pair-wise filters to make them more diverse and penalize irrelevance between regional response mappings. We make multiple combinations and propose several methods on improving orthogonality, which have different adaptations for different datasets and network models. We evaluate their effectiveness in experiment. Our combinations could improve the efficiency of robust image recognition.

As an essential means of copyright protection, the deep learning-based robust watermarking method is being studied extensively. Its framework consists of three main parts: the encoder, the noise layer and the decoder. But practically all of the schemes are directed at the encoder rather than the decoder. And the whole network is structured by shallow Convolutional Neural Networks (CNNs) for primary feature extraction, while CNNs capture local information and do not model non-local information in watermarked images well. To solve this problem, we consider the use of Transformer networks with a spatially self-attention mechanism. We propose to construct a novel decoder network by combining Transformer and CNNs, which can not only enriches local feature information but also enhances the ability to explore global representations. Meanwhile, to embed secret messages more perfectly, we design a multi-scale attentional feature fusion module to achieve an efficient aggregation of cover image features and secret message features, resulting in the encoded images with rich hybrid features. In addition, perceptual loss is introduced to better evaluate the visual quality of the watermarked images. Extensive experimental results show that our proposed method achieves better results in terms of imperceptibility and robustness compared with existing State-Of-The-Art (SOTA) methods.

Large neural networks are often overparameterised and prone to overfitting, Dropout is a widely used regularization technique to combat overfitting and improve model generalization. However, unstructured Dropout is not always effective for specific network architectures and this has led to the formation of multiple structured Dropout approaches to improve model performance and, sometimes, reduce the computational resources required for inference. In this work, we revisit structured Dropout comparing different Dropout approaches on natural language processing and computer vision tasks for multiple state-of-the-art networks. Additionally, we devise an approach to structured Dropout we call \textbf{\emph{ProbDropBlock}} which drops contiguous blocks from feature maps with a probability given by the normalized feature salience values. We find that, with a simple scheduling strategy, the proposed approach to structured Dropout consistently improves model performance compared to baselines and other Dropout approaches on a diverse range of tasks and models. In particular, we show \textbf{\emph{ProbDropBlock}} improves RoBERTa finetuning on MNLI by $0.22%$, and training of ResNet50 on ImageNet by $0.28%$.

Wildfires pose a significant and recurring challenge in North America, impacting both human and natural environments. The size and severity of wildfires in the region have been increasing in recent years, making it a pressing concern for communities, ecosystems, and the economy. The accurate and timely detection of active wildfires in remote areas is crucial for effective wildfire management and mitigation efforts. In this research paper, we propose a robust approach for detecting active wildfires using multispectral satellite imagery by leveraging vision transformers and a vast repository of landsat-$8$ satellite data with a $30$m spatial resolution in North America. Our methodology involves experimenting with vision transformers and deep convolutional neural networks for wildfire detection in multispectral satellite images. We compare the capabilities of these two architecture families in detecting wildfires within the multispectral satellite imagery. Furthermore, we propose a novel u-shape vision transformer that effectively captures spatial dependencies and learns meaningful representations from multispectral images, enabling precise discrimination between wildfire and non-wildfire regions. To evaluate the performance of our approach, we conducted experiments on a comprehensive dataset of wildfire incidents. The results demonstrate the effectiveness of the proposed method in accurately detecting active wildfires with an \textit{Dice Score or F$1$} of $%90.05$ and \textit{Recall} of $%89.61$ . Overall, our research presents a promising approach for leveraging vision transformers for multispectral satellite imagery to detect remote wildfires.

This study addresses a gap in the utilization of Reinforcement Learning (RL) and Machine Learning (ML) techniques in solving the Stochastic Vehicle Routing Problem (SVRP) that involves the challenging task of optimizing vehicle routes under uncertain conditions. We propose a novel end-to-end framework that comprehensively addresses the key sources of stochasticity in SVRP and utilizes an RL agent with a simple yet effective architecture and a tailored training method. Through comparative analysis, our proposed model demonstrates superior performance compared to a widely adopted state-of-the-art metaheuristic, achieving a significant 3.43% reduction in travel costs. Furthermore, the model exhibits robustness across diverse SVRP settings, highlighting its adaptability and ability to learn optimal routing strategies in varying environments. The publicly available implementation of our framework serves as a valuable resource for future research endeavors aimed at advancing RL-based solutions for SVRP.

In semi-supervised learning, student-teacher distribution matching has been successful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We introduce our algorithm, Q-Match, and show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We focus our study on tabular datasets and show that Q-Match outperforms previous self-supervised learning techniques when measuring downstream classification performance. Furthermore, we show that our method is sample efficient–in terms of both the labels required for downstream training and the amount of unlabeled data required for pre-training–and scales well to the sizes of both the labeled and unlabeled data.

Counterfactual Explanations (CEs) have received increasing interest as a major methodology for explaining neural network classifiers. Usually, CEs for an input-output pair are defined as data points with minimum distance to the input that are classified with a different label than the output. To tackle the established problem that CEs are easily invalidated when model parameters are updated (e.g. retrained), studies have proposed ways to certify the robustness of CEs under model parameter changes bounded by a norm ball. However, existing methods targeting this form of robustness are not sound or complete, and they may generate implausible CEs, i.e., outliers wrt the training dataset. In fact, no existing method simultaneously optimises for closeness and plausibility while preserving robustness guarantees. In this work, we propose Provably RObust and PLAusible Counterfactual Explanations (PROPLACE), a method leveraging on robust optimisation techniques to address the aforementioned limitations in the literature. We formulate an iterative algorithm to compute provably robust CEs and prove its convergence, soundness and completeness. Through a comparative experiment involving six baselines, five of which target robustness, we show that PROPLACE achieves state-of-the-art performances against metrics on three evaluation aspects.

Generalized zero-shot learning (GZSL) is still a technical challenge of deep learning. To preserve the semantic relation between source and target classes when only trained with data from source classes, we address the quantification of the knowledge transfer from an information-theoretic viewpoint. We use the prototypical model and format the variables of concern as a probability vector. Taking advantage of the probability vector representation, information measurements can be effectively evaluated with simple closed forms. We propose two information-theoretic loss functions: a mutual information loss to bridge seen data and target classes; an uncertainty-aware entropy constraint loss to prevent overfitting when using seen data to learn the embedding of target classes. Simulation shows that, as a deterministic model, our proposed method obtains state-of-the-art results on GZSL benchmark datasets. We achieve 21% − 64% improvements over the baseline model – deep calibration network (DCN) and demonstrate that a deterministic model can perform as well as generative ones. Furthermore, the proposed method is compatible with generative models and can noticeably improve their performance.

Diffusion models are generative models that have shown significant advantages compared to other generative models in terms of higher generation quality and more stable training. However, the computational need for training diffusion models is considerably increased. In this work, we incorporate prototype learning into diffusion models to achieve high generation quality faster than the original diffusion model. Instead of randomly initialized class embeddings, we use separately learned class prototypes as the conditioning information to guide the diffusion process. We observe that our method, called ProtoDiffusion, achieves better performance in the early stages of training compared to the baseline method, signifying that using the learned prototypes shortens the training time. We demonstrate the performance of ProtoDiffusion using various datasets and experimental settings, achieving the best performance in shorter times across all settings.

Traffic forecasting plays an extremely important role in many applications such as intelligent transportation and smart cities. However, due to the hidden and complex dynamic spatio-temporal correlations and heterogeneity, achieving high-precision traffic prediction is a challenging task. This paper proposes a new spatio-temporal aware learning graph neural network (STALGNN) for traffic prediction. First, a temporal-aware graph generation module is designed to exploit the spatial-temporal features that the spatial graph may not be able to present. Then, a spatio-temporal joint module is designed to more effectively capture local spatio-temporal correlations. Next, a multi-scale gated convolutions module is proposed to capture gloable dynamic spatio-temporal correlations. Furthermore, STALGNN further learns explicit spatio-temporal correlations through integrated attention mechanisms and stacked graph convolutional networks to handle long-term prediction. Extensive experiments on several real traffic datasets show that the proposed method can achieve the superior performance compared with other baselines.

Forecasting the crossing intention of pedestrians is an essential task for the safe driving of Autonomous Vehicles (AVs) in the real world. Pedestrians’ behaviors are usually influenced by their surroundings in traffic scenes. Recent works based on vision-based neural networks extract key information from images to perform prediction. However, in the driving environment, there exists much critical information, such as the social and scene interaction in the driving area, the location and distance between the ego car and target pedestrian, and the motion of all targets. How properly exploring and utilizing the above implicit interactions will promote the development of Autonomous Vehicles. In this chapter, two novel attributes, the pedestrian’s location on the road or sidewalk, and the relative distance from the target pedestrian to the ego-car, which are derived from the semantic map and depth map combined with bounding boxes, are introduced. A hybrid prediction network based on multi-modal is proposed to capture the interactions between all the features and predict pedestrian crossing intention. Evaluated by two public pedestrian crossing datasets, PIE and JAAD, the proposed hybrid framework outperforms the state-of-the-art by about an accuracy of 3%.

Regularization plays a crucial role in machine learning models, especially for deep neural networks. The existing regularization techniques mainly rely on the i.i.d. assumption and only consider the knowledge from the current sample, without the leverage of the neighboring relationship between samples. In this work, we propose a general regularizer called \textbf{Patch-level Neighborhood Interpolation (Pani)} that conducts a non-local representation in the computation of networks. Our proposal explicitly constructs patch-level graphs in different layers and then linearly interpolates neighborhood patch features, serving as a general and effective regularization strategy. Further, we customize our approach into two kinds of popular regularization methods, namely Virtual Adversarial Training (VAT) and MixUp as well as its variants. The first derived \textbf{Pani VAT} presents a novel way to construct non-local adversarial smoothness by employing patch-level interpolated perturbations. The second derived \textbf{Pani MixUp} method extends the MixUp, and achieves superiority over MixUp and competitive performance over state-of-the-art variants of MixUp method with a significant advantage in computational efficiency. Extensive experiments have verified the effectiveness of our Pani approach in both supervised and semi-supervised settings.

It is essential for models to gradually adapt to the world’s increasing complexity, and we can use models more effectively if they keep up with the times. However, the technique of continuous learning (CL) has an issue with catastrophic forgetting, and effective continuous learning methods can only be attained by effectively limiting forgetting and learning new tasks. In this study, we offer the Classifier Expander(CE) method, which combines the regularization-based and replay-based approaches. By undergoing two stages of training, it fulfills the aforementioned standards. The training content for the new task is limited to the portion of the network relevant to that task in the first stage, which uses the replay approach to reduce the forgetting problem. This strategy minimizes disruption to the old task while facilitating efficient learning of the new one. Utilizing all of the data available, the second stage retrains the network and sufficiently trains the classifier to balance the learning performance of the old and new tasks. Our method regularly outperforms previous CL methods on the CIFAR-100 and CUB-200 datasets, obtaining an average improvement of 2.94% on the class-incremental learning and 1.16% on the task-incremental learning compared to the best method currently available. Our code is available at https://github.com/EmbraceTomorrow/CE.

Supervised learning models are challenged by the intrinsic complexities of training data such as outliers and minority subpopulations and intentional attacks at inference time with adversarial samples. While traditional robust learning methods and the recent adversarial training approaches are designed to handle each of the two challenges, to date, no work has been done to develop models that are robust with regard to the low-quality training data and the potential adversarial attack at inference time simultaneously. It is for this reason that we introduce \underline{O}utlier \underline{R}obust \underline{A}dversarial \underline{T}raining (ORAT) in this work. ORAT is based on a bi-level optimization formulation of adversarial training with a robust rank-based loss function. Theoretically, we show that the learning objective of ORAT satisfies the $\mathcal{H}$-consistency in binary classification, which establishes it as a proper surrogate to adversarial 0/1 loss. Furthermore, we analyze its generalization ability and provide uniform convergence rates in high probability. ORAT can be optimized with a simple algorithm. Experimental evaluations on three benchmark datasets demonstrate the effectiveness and robustness of ORAT in handling outliers and adversarial attacks. Our code is available at \url{https://github.com/discovershu/ORAT}.

Random feature model with a nonlinear activation function has been shown to perform asymptotically equivalent to a Gaussian model in terms of training and generalization errors. Analysis of the equivalent model reveals an important yet not fully understood role played by the activation function. To address this issue, we study the "parameters" of the equivalent model to achieve improved generalization performance for a given supervised learning problem. We show that acquired parameters from the Gaussian model enable us to define a set of optimal nonlinearities. We provide two example classes from this set, e.g., second-order polynomial and piecewise linear functions. These functions are optimized to improve generalization performance regardless of the actual form. We experiment with regression and classification problems, including synthetic and real (e.g., CIFAR10) data. Our numerical results validate that the optimized nonlinearities achieve better generalization performance than widely-used nonlinear functions such as ReLU. Furthermore, we illustrate that the proposed nonlinearities also mitigate the so-called double descent phenomenon, which is known as the non-monotonic generalization performance regarding the sample size and the model size.