A Closer Look at Machine Unlearning for Large Language Models

This repository is the official implementation for the paper: A Closer Look at Machine Unlearning for Large Language Models

Installation

We follow the TOFU Benchmark to install the required dependencies, please run the following commands:

conda create -n tofu python=3.10
conda activate tofu
conda install pytorch pytorch-cuda=11.8 -c pytorch -c nvidia
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
pip install -r requirements.txt
pip install flash-attn --no-build-isolation

All experiments are conducted on two NVIDIA A100 GPUs with 40GB of memory.

Fictitious unlearning scenario

(1) ME+GD

bash scripts/tofu/me_gd.sh

(2) IDK+AP

bash scripts/tofu/idk_ap.sh

Continual unlearning scenario

(1) ME+GD

bash scripts/continual_tofu/me_gd.sh

(2) IDK+AP

bash scripts/continual_tofu/idk_ap.sh

Real-world unlearning scenario

To evaluate LLMs on general tasks in real-world unlearning scenarios, we follow lm-evaluation-harness to install the lm-eval package:

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

(1) ME+GD

bash scripts/real_world/me_gd.sh

(2) IDK+AP

bash scripts/real_world/idk_ap.sh

Acknowledgments

This repository is based on the codebase of the TOFU Benchmark. Thanks for their impressive works!