Knowledge Graph-Driven Memory Editing with Directional Interventions

Abstract

Large Language Models (LLMs) have revolutionized language processing and understanding, yet their performance is hampered by inaccuracies and outdated information. Model editing techniques offer a solution but face two key challenges: (I) Most methods inject knowledge by constructing rigid loss, which leads to poor compatibility when dealing with higher-order multi-hop problems. (II) Locate-then-edit vein, by altering pre-trained parameters, inevitably affect normal knowledge and even face the catastrophic forgetting. In this paper, we introduce KGMET, a framework that constructs knowledge graphs using available information to guide the direction of knowledge editing, enabling consistent, aligned, and stable information during large-scale editing scenario. Furthermore, KGMET goes beyond this by employing orthogonal constraints to block the interference of irrelevant information, ensuring the updates are both controllable and generalizable. Experiments on Multi-Conterfact, ZsRE, and MQuAKE datasets using Llama-3-8B, GPT-J-6B, and GPT-2-XL models showcase improvements over state-of-the-art methods, with \uparrow 5\%-17\% in multi-hop tasks while remaining generalizable (at least \uparrow 20\% in fluency). Our code is available on Github.