Privacy Adhering Machine Un-learning in NLP

Abstract

tems to remove such sensitive information before being used for retraining purposes. Industry apRegulations introduced General Data Protecplications also adopt rigorous approval processes tion Regulation (GDPR) in the EU or Caliwith opt-in or opt-out capabilities to ensure that fornia Consumer Privacy Act (CCPA) in the user generated content is used in a safe manner US have included provisions on the right to be while raising awareness among users on how their forgotten that mandates industry applications to remove data related to an individual from generated content will be used. their systems. In several real world industry apMore recently, the GDPR and CCPA provisioned plications that use Machine Learning to build the right to be forgotten, introducing new regulamodels on user data, such mandates require sigtions that mandates industry applications to support nificant effort both in terms of data cleansing deletion of user generated content when requested as well as model retraining while ensuring the by users. Although privacy concerns have gained models do not deteriorate in prediction quality significant importance now, the scale at which these due to removal of data. As a result, continuous removal of data and model retraining steps do models are built make it extremely expensive and not scale if these applications receive such retime-consuming to remove data efficiently while quests at a very high frequency. Recently, a few providing complete removal guarantees. researchers proposed the idea of Machine UnUnlearning assures the users that their data has learning to tackle this challenge. Despite the been completely removed when they prefer to have significant importance of this task, the area of their data erased (Bourtoule et al., 2019). This Machine Unlearning is under-explored in Natural Language Processing (NLP) tasks. In this task is different from Differential Privacy that fopaper, we explore the Unlearning framework on cuses on models that ensure that the users’ inforvarious GLUE tasks (Wang et al., 2018), such mation in the training data cannot be inferred and as, QQP, SST and MNLI. We propose compuprovides guarantees on the effect of an individual tationally efficient approaches (SISA-FC and examples. Such a task is challenging as removal SISA-A) to perform guaranteed Unlearning of data points can have significant deterioration on that provides significant reduction in terms of the performance of the models aka catastrophic both memory (90-95%), time (100x) and space consumption (99%) in comparison to the baseunlearning (Du et al., 2019; Golatkar et al., 2019; lines while having minimal impact on model Nguyen et al., 2020, 2022a). performance.1 . Approaches that guarantee complete removal of users’ data points have recently gained momentum