ACML 2023 Papers

As the most commonly used quantization techniques for deep neural networks, the int-only quantization methods use scale factor to linearly approximate the weights or activation of each layer. However, when passing activation data between layers, such int-only quantization methods require extra Scale Factor Conversion (SFC) operations, resulting in computational overhead. In this paper, we propose a Group-Wise Quantization framework, called GWQ, to reduce computational consumption during the activation data pass process by allowing multiple layers share one scale factor in SFC operations. Specifically, in the GWQ framework, we propose two algorithms for network layers grouping and model training. For the grouping of network layers, we propose a grouping algorithm based on the similarity of data numerical distribution. Then, the network layers divided into the same group will be quantified using the same common scale factor to reduce the computational consumption. Considering the additional performance loss caused by sharing scale factors among multiple layers, we propose a training algorithm to optimize these shared scale factors and model parameters, by designing a learnable power-of-two scaling parameter for each layer. Extensive experiments demonstrate that the proposed GWQ framework is able to effectively reduce the computational burden during inference, while maintaining model performance with negligible impact.

In recent years, U-Net and its variants have been widely used in pathology image segmentation tasks. One of the key designs of U-Net is the use of skip connections between the encoder and decoder, which helps to recover detailed information after upsampling. While most variations of U-Net adopt the original skip connection design, there is semantic gap between the encoder and decoder that can negatively impact model performance. Therefore, it is important to reduce this semantic gap before conducting skip connection. To address this issue, we propose a new segmentation network called FusionU-Net, which is based on U-Net structure and incorporates a fusion module to exchange information between different skip connections to reduce semantic gaps. Unlike the other fusion modules in existing networks, ours is based on a two-round fusion design that fully considers the local relevance between adjacent encoder layer outputs and the need for bi-directional information exchange across multiple layers. We conducted extensive experiments on multiple pathology image datasets to evaluate our model and found that FusionU-Net achieves better performance compared to other competing methods. We argue our fusion module is more effective than the designs of existing networks, and it could be easily embedded into other networks to further enhance the model performance. Our code is available at: https://github.com/Zongyi-Lee/FusionU-Net

Micro defects, such as casting pores in industrial products, have been detected by human visual inspection using X-ray CT images and image processing tools. Automatic detection of micro defects is challenging for anomaly detection methods using image reconstruction errors and nearest neighbor distances because these metrics are dominated by low-frequency information and are insensitive to minor defects. Although recent methods achieve high anomaly detection performances, their detection abilities are insufficient for micro defects. To overcome these problems, we propose to extend a state-of-the-art anomaly detection method by introducing frequency-dependent losses to capture reconstruction errors appearing around micro defects and frequency-dependent data augmentation to improve the sensitivity against the errors. We demonstrate the effectiveness of the proposed method through experiments with MVTec AD dataset especially on the detection of micro defects.

Since the success of Residual Network(ResNet), many of architectures of Convolutional Neural Networks(CNNs) have adopted skip connection. While the generalization performance of CNN with skip connection has been explained within the framework of Ensemble Learning, the dependency on the number of parameters has not been revealed. In this paper, we show that Bayesian free energy of Convolutional Neural Network both with and without skip connection in Bayesian learning. Bayesian Free Energy is the negative log marginal likelihood which is equivalent to Stochastic Complexity or Minimum Description Length (MDL) used for evaluating model complexity. The upper bound of free energy of Bayesian CNN with skip connection does not depend on the oveparametrization and, the generalization error of Bayesian CNN has similar property.

Probabilistic modelling on Generative Adversarial Networks (GANs) within the Bayesian framework has shown success in estimating the complex distribution in literature. In this paper, we develop a Bayesian formulation for unsupervised and semi-supervised GAN learning. Specifically, we propose Folded Hamiltonian Monte Carlo (F-HMC) methods within this framework to learn the distributions over the parameters of the generators and discriminators. We show that the F-HMC efficiently approximates multi-modal and high dimensional data when combined with Bayesian GANs. Its composition improves run time and test error in generating diverse samples. Experimental results with high-dimensional synthetic multi-modal data and natural image benchmarks, including CIFAR-10, SVHN and ImageNet, show that F-HMC outperforms the state-of-the-art methods in terms of test error, run times per epoch, inception score and Frechet Inception Distance scores.

Most existing federated learning methods are unable to estimate model/predictive uncertainty since the client models are trained using the standard loss function minimization approach which ignores such uncertainties. In many situations, however, especially in limited data settings, it is beneficial to take into account the uncertainty in the model parameters at each client as it leads to more accurate predictions and also because reliable estimates of uncertainty can be used for tasks, such as out-of-distribution (OOD) detection, and sequential decision-making tasks, such as active learning. We present a framework for federated learning with uncertainty where, in each round, each client infers the posterior distribution over its parameters as well as the posterior predictive distribution (PPD), distills the PPD into a single deep neural network, and sends this network to the server. Unlike some of the recent Bayesian approaches to federated learning, our approach does not require sending the whole posterior distribution of the parameters from each client to the server but only the PPD in the distilled form as a deep neural network. In addition, when making predictions at test time, it does not require computationally expensive Monte-Carlo averaging over the posterior distribution because our approach always maintains the PPD in form a single deep neural network. Moreover, our approach does not make any restrictive assumptions, such as the form of the clients’ posterior distributions, or of their PPDs. We evaluate our approach on classification in federated setting, as well as active learning and OOD detection in federated settings, on which our approach outperforms various existing federated learning baselines.

In terms of multi-person multi-view 3D pose estimation, voxel-based methods gain promising accuracy by directly manipulating features in 3D space. Since their high computational cost prevents them from practical applications, Faster VoxelPose was proposed to address this complication by re-projecting the 3D feature volume onto coordinate planes, which greatly improved the efficiency of the model. However, it suffers from an obvious performance drop, especially when there are fewer cameras. In this paper, we propose a more accurate real-time 3D pose estimation method, FasterVoxelPose+, to address the above problem. We have made two improvements to the previous methods. First, we propose a novel method for constructing voxel feature volume called Depth-wise Projection Decay (DPD). It introduces extra depth information to the projection to alleviate depth ambiguity. Second, we design an Encoder-Decoder Network for processing the re-projected voxel features to further push up the performance of the model. Our method obtains 17.42mm MPJPE on Panoptic with real-time speed and can be easily used in other voxel-based models.

Multi-agent encirclement refers to controlling multiple agents to restrict the movement of a target and surround it with a specific formation. However, two challenges remain: encirclement in obstacle scenarios and encirclement of a faster target. In obstacle scenarios, we propose the utilization of obstacles for facilitating encirclement and introduce the concept of contributing angle to quantify the contribution of agents and obstacles, which enables agents to effectively utilize obstacles while mitigating the credit assignment problem. To address the challenge of encircling a faster target, we propose a two-stage encirclement method inspired by lions’ hunting strategy, effectively preventing target escape. We design the reward function based on the contributing angle and the lion encirclement method, integrating it with the Multi-Agent Deep Deterministic Policy Gradient (MADDPG). The simulation results demonstrate that our method can utilize obstacles to complete encirclement and has a higher success rate. In some conditions with insufficient numbers of agents, our methods can still accomplish the task. Ablation experiments are conducted to verify the effectiveness of the contributing angle and the lion encirclement method respectively.

Convolutional neural networks (CNNs) are becoming deeper and wider to achieve higher accuracy and lower loss, significantly expanding the computational resources. Especially, training CNN models extensively consumes memory mainly due to storing intermediate feature maps generated in the forward-propagation for calculating the gradient in the backpropagation. The memory usage of the CNN model training escalates with the increase in batch size and the complexity of the model. Therefore, a lightweight training method is essential, especially when the computational resources are limited. In this paper, we propose a CNN training mechanism called Facto-CNN, leveraging low-rank tensor factorization and lossy tensor compression to reduce the memory usage required in training the CNN models. Facto-CNN factorizes the weight tensors of convolutional and fully-connected layers and then only updates one of the factorized tensors for each layer, dramatically reducing the feature map size stored in the memory. To further reduce memory consumption, Facto-CNN compresses the feature maps with a simple lossy compression technique that exploits the value similarity in the feature maps. Our experimental evaluation demonstrates that Facto-CNN reduces the memory usage for storing the feature maps by 68-93% with a trivial accuracy degradation when training the CNN models.

Multivariate time series forecasting is of great importance in a diverse range of domains. In recent years, a variety of spatial-temporal graph neural networks (STGNNs) have been proposed to address this task and achieved promising results. However, these networks are typically handcrafted and require extensive human expertise. Additionally, the temporal and spatial dependencies hidden within different scenarios vary, making it difficult for them to adapt to different scenarios. In this paper, we propose an evolutionary neural architecture search framework, entitled EMTSF, for automated STGNN design. Specifically, we employ fine-grained neural architecture search into both the spatial convolution module and the temporal convolution module. For the spatial convolution search space, various feature filtering and neighbor aggregation operations are employed to find the most suitable message-passing mechanism for modeling the spatial dependencies. For the temporal convolution search space, gated temporal convolutions with different kernel sizes are utilized to best capture temporal dependencies with various ranges. The spatial convolution module and temporal convolution module are jointly optimized with the proposed evolutionary search algorithm to heuristically identify the optimal STGNN architecture. Extensive experiments on four commonly used benchmark datasets show EMTSF achieves promising performance compared with the state-of-the-art methods, which demonstrates the effectiveness of the proposed framework.

Counterfactual analysis is intuitively performed by humans on a daily basis eg. ”What should I have done differently to get the loan approved?”. Such counterfactual questions also steer the formulation of scientific hypotheses. More formally it provides insights about potential improvements of a system by inferring the effects of hypothetical interventions into a past observation of the system’s behaviour which plays a prominent role in a variety of industrial applications. Due to the hypothetical nature of such analysis, counterfactual distributions are inherently ambiguous. This ambiguity is particularly challenging in continuous settings in which a continuum of explanations exist for the same observation. In this paper, we address this problem by following a hierarchical Bayesian approach which explicitly models such uncertainty. In particular, we derive counterfactual distributions for a Bayesian Warped Gaussian Process thereby allowing for non-Gaussian distributions and non-additive noise. We illustrate the properties of our approach on a synthetic and on a semi-synthetic example and show its performance when used within an algorithmic recourse downstream task.

We present a novel approach, Equilibrium Point Learning (EPL), for training the deep equilibrium model (DEQ). In this method, the equilibrium point of the DEQ serves as the learnable parameters. Notably, the DEQ parameters encapsulate the learning algorithm itself and remain fixed. Consequently, by exploring the parameter space, we can discover a more efficient learning algorithm without relying on conventional techniques such as backpropagation or Q-learning. In this paper, we adopt an evolutionary approach inspired by biological neurons to evolve the DEQ model parameters. Initially, we examine the physical dynamics of neurons at the molecular level and translate them into a dynamical system representation. Subsequently, we formulate a deep implicit layer that is mathematically proven to possess an equilibrium point. The energy function of the implicit layer is defined using a quadratic form augmented with entropy and momentum terms. Given the resemblance between the dynamics of the deep implicit layer and the principles of physics and chemistry, it can effectively capture the biomodel of systems biology and the neural model of spiking neural networks (SNNs). This equivalence enables us to define the implicit layer of the DEQ, allowing for seamless integration with existing artificial neural networks (ANNs). Finally, we employ HyperNEAT to evolve the parameters of the dynamical system. Through our experiments, we observe a consistent improvement in learning efficiency, with each successive generation exhibiting a 0.2% increase in learning speed per generation.

Cervical cancer is the most common gynecologic malignancy, and in clinical practice, cervical cancer is best treated if it is detected at an early stage. Thinprep Cytologic Test (TCT) is the best early detection method for cervical cancer as determined by the WHO. As the coverage of early detection of cervical cancer increases, the number of samples in hospitals increases annually, and the pressure on the pathologists to read the cytological images increases, which easily leads to an increase in the rate of misdiagnosis and missed diagnosis. Therefore, automatic detection of abnormal cells in cervical cytology images of cervical fluid using deep learning techniques has become a hot research topic today. However, existing deep learning models for cell detection often collect a single data source from a medical institution for construction. Different medical institutions have different equipment and staining methods, and the accuracy, magnification, and staining results of the images obtained will be different. As a result, the application performance of the model in different medical institution data is not good, and there is a problem of domain shift. To address these problems, this paper proposes a method for cervical fluid-based cell detection based on causal feature extraction. The method is based on the one-stage detection model RetinaNet, and incorporates causal autoencoder to learn the invariant causal feature representation from data. It reduces the impact of task-irrelevant feature representations, reduces the variability of feature distributions in different datasets, and effectively solves the domain shift problem. The addition of deformable convolution and attention mechanism enhances the feature extraction capability for foreground categories with variable shapes in cervical fluid-based pathology images. This reduces the impact of possible strong correlation between background features and goal cells, and reduces the interference of the foreground categories by fading and lack of brightness in the staining. The generalization ability of the model is improved, which makes the model better applicable to different medical institutions. The experimental results show that the method in this paper not only improves the accuracy of the model detection, but also verifies its good generalization effect on different datasets.

Modern DNN-based recommendation systems rely on training-derived embeddings of sparse features. Input sparsity makes obtaining high-quality embeddings for rarely-occurring categories harder as their representations are updated infrequently. We demonstrate a training-time technique to produce superior embeddings via effective cross-category learning and theoretically explain its surprising effectiveness. The scheme, termed the multi-layer embeddings training (MLET), trains embeddings using factorization of the embedding layer, with an inner dimension higher than the target embedding dimension. For inference efficiency, MLET converts the trained two-layer embedding into a single-layer one thus keeping inference-time model size unchanged. Empirical superiority of MLET is puzzling as its search space is not larger than that of the single-layer embedding. The strong dependence of MLET on the inner dimension is even more surprising. We develop a theory that explains both of these behaviors by showing that MLET creates an adaptive update mechanism modulated by the singular vectors of embeddings. When tested on multiple state-of-the-art recommendation models for click-through rate (CTR) prediction tasks, MLET consistently produces better models, especially for rare items. At constant model quality, MLET allows embedding dimension, and model size, reduction by up to 16x, and 5.8x on average, across the models.

The right to be forgotten (RTBF) is a concept that pertains to an individual’s right to request the removal or deletion of their personal information when it is no longer necessary, relevant, or accurate for the purposes for which it was initially collected. Machine Learning (ML) models often rely on large, diverse datasets for optimal performance. Hence, when an individual exercises the RTBF, it can impact the ML model’s performance and accuracy. In the context of Federated Learning (FL), where a server trains a model across multiple decentralized devices without moving data away from clients, implementing the RTBF in FL presents some unique challenges compared to traditional ML approaches. For instance, the decentralized nature makes it challenging to identify and remove specific user data from the model. Although various unlearning methods have been proposed in the literature, they have not been well investigated from the efficiency perspective. To fill this gap, this paper presents an empirical study to investigate the impacts of various unlearning methods. Our experiments are designed in diverse scenarios involving multiple communication and unlearning rounds using three datasets, MNIST, CIFAR-10, and CIFAR-100. We utilize backdoor attack and Cosine Similarity to assess the effectiveness of each unlearning method. The findings and insights from this research can be integrated into FL systems to enhance their overall performance and effectiveness. Our research codes are available on GitHub at \url{https://github.com/sail-research/fed-unlearn}.

Text detection in medical images is a critical task, essential for automating the extraction of valuable information from diverse healthcare documents. Conventional text detection methods, predominantly based on segmentation, encounter substantial challenges when confronted with text-rich images, extreme aspect ratios, and multi-oriented text. In response to these complexities, this paper introduces an innovative text detection system aimed at enhancing its efficacy. Our proposed system comprises two fundamental components: the Efficient Feature Enhancement Module (EFEM) and the Multi-Scale Feature Fusion Module (MSFM), both serving as integral elements of the segmentation head. The EFEM incorporates a spatial attention mechanism to improve segmentation performance by introducing multi-level information. The MSFM merges features from the EFEM at different depths and scales to generate final segmentation features. In conjunction with our segmentation methodology, our post-processing module employs a differentiable binarization technique, facilitating adaptive threshold adjustment to enhance text detection precision. To further bolster accuracy and robustness, we introduce the integration of a vision-language pre-training model. Through extensive pretraining on large-scale visual language understanding tasks, this model amasses a wealth of rich visual and semantic representations. When seamlessly integrated with the segmentation module, the pretraining model effectively leverages its potent representation capabilities. Our proposed model undergoes rigorous evaluation on medical text image datasets, consistently demonstrating exceptional performance. Benchmark experiments reaffirm its efficacy.

With advancements in text-to-image (TTI) models, text-to-video (TTV) models have recently been introduced. Motivated by approaches on TTV models adapting from diffusion-based TTI models, we suggest the text-guided video editing framework given only a pretrained TTI model and a single

Alzheimer disease is one of the most common neuro-degenerative diseases, with an estimated 6.2 million cases in the United States. This research article investigates the potential of Transformer-based deep learning techniques to accelerate the processing of diffusion tensor imaging (DTI) measures and improve the early diagnosis of Alzheimer disease (AD) using sparse data. Diffusion Weighted Imaging (DWI) is a time-consuming process, with each diffusion direction taking between 2-5 minutes, and at least 40 diffusion directions are needed for routine clinical diagnosis, which needs scanning duration exceeding 3 hours for each patient. By leveraging the attention mechanism, our proposed model generates quantitative measures of fractional anisotropy (FA), axial diffusivity (AxD), and mean diffusivity (MD) using 5 and 21 diffusion directions, making it useful for clinical diagnosis through reduced scanning time of more than half. Our experimental results on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset demonstrate that our proposed model outperforms the traditional linear least square method, achieving accurate quantitative measurement of FA, AxD, and MD scores for early diagnosis of AD patients from healthy controls using sparse diffusion directions. Our analysis highlights the potential of Swin-Transformer attention-based deep learning framework to improve the early diagnosis and treatment of Alzheimer’s disease.

The cold-start problem poses a significant challenge in recommendation systems, particularly when interaction data is scarce. While meta-learning has shown promise in few-shot classification, its application to cold-start recommendations has mostly involved simple transplantations of generic approaches. The effectiveness of metric learning, a powerful meta-learning method, is hindered by differences in problem definition when applied to rating prediction. Heterogeneous information networks (HINs), as high-order graph structures, can capture valuable semantic information even in data-starved conditions. Efficient utilization of HINs can alleviate the cold-start dilemma. However, in the cold-start domain, there is a lack of dynamic node-level and semantic-level feature fusion schemes, resulting in the underutilization of complex information. This study addresses these issues by combining metric learning and HINs, proposing OMHIN (Dynamic Offset Metric approach to Heterogeneous Information Networks). Our approach transforms a direct similarity metric into an indirect metric to enhance model robustness. By flexibly applying one-dimensional convolution, OMHIN effectively integrates rich information from HINs while minimizing noise introduction. Experimental results on two datasets demonstrate that OMHIN achieves state-of-the-art performance in various scenarios, particularly in complex and challenging situations. It is especially suitable for sequence cold-start recommendations.