A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings

摘要

We propose a Neighbourhood-Aware Differential Privacy (NADP) mechanism considering the neighbourhood of a word in a pretrained static word embedding space to determine the minimal amount of noise required to guarantee a specified privacy level. We first construct a nearest neighbour graph over the words using their embeddings, and factorise it into a set of connected components (i.e. neighbourhoods). We then separately apply different levels of Gaussian noise to the words in each neighbourhood, determined by the set of words in that neighbourhood. Experiments show that our proposed NADP mechanism consistently outperforms multiple previously proposed DP mechanisms such as Laplacian, Gaussian, and Mahalanobis in multiple downstream tasks, while guaranteeing higher levels of privacy. 1 Figure 1: Anonymizing a target word (shown in red) in a dense (left) vs. a sparse (right) neighbourhoods of words (shown in blue). In the sparse neighbourhood, NADP adds a higher level of perturbation noise z to the target word embedding x in order to protect its privacy by disguising it among its neighbours, while in a dense neighbourhood it adds less noise. the vertices of the nearest neighbour graph) computed using their word embeddings. Second, we compute the connected components in the nearest neighbour graph to find the neighbourhoods of words. Third, we apply Gaussian noise to all words in each neighbourhood, such that the variance of the noise is determined by the words in that neighbourhood.