Release 2025.02

# 🎉 Major Updates
- 🚀 1.25x speed improvements (1.5x with `use_kv_cache=True`)
- 📉 Introduced `autoGCG` - automatic GCG tuning using Bayesian
optimization
- 💼 Data subsystem refactor to enable arbitrary dataset
support
- 🧠 Add a tutorial on how to use **LLM**art as a standalone
library.

# 🎈 Minor Updates
- Support for uv
- More intuitive dataset splitting parameters
- Disable early stopping via `early_stop=False`
- Run test only via `attack=None` or `steps=0`
- Option to enable/disable batch splitting via
`data.split_batches=True/False`
- Reusable closure creation

# 🚧 Bug Fixes
- Remove `world_size` from optimizer
- Fix `_local_swap_count` being on wrong device in optimizer

---------

Co-authored-by: Marius Arvinte <[email protected]>

Author: dxoigmn

README.md

@@ -10,17 +10,22 @@
 
 </div>
 
-## :pushpin: What is **LLM**art?
+## 🆕 Latest updates
+❗Release 2025.02 brings significant speed-ups to the core library, with zero user involvement.\
+We additionally recommend using the command line argument `per_device_bs` with a value as large as possible on GPUs with at least 48GB to take the most advantage of further speed-ups.
+
+❗We now offer command-line support for jailbreaking thoughts and responses for DeepSeek-R1 on multi-GPU:
+```bash
+accelerate launch -m llmart model=deepseek-r1-distill-llama-8b data=basic per_device_bs=64 "response.replace_with=`echo -e '\"<think>\nOkay, so I need to tell someone about Saturn.\n</think>\n\nNO WAY JOSE\"'`"
+```
+
+❗Check out our new [notebook](examples/basic/basic_dev_workflow.ipynb) containing a detailed step-by-step developer overview of all `llmart` components and how to customize them.
 
 **LLM**art is a toolkit for evaluating LLM robustness through adversarial testing. Built with PyTorch and Hugging Face integrations, **LLM**art enables scalable red teaming attacks with parallelized optimization across multiple devices.
 **LLM**art has configurable attack patterns, support for soft prompt optimization, detailed logging, and is intended both for high-level users that want red team evaluation with off-the-shelf algorithms, as well as research power users that intend to experiment with the implementation details of input-space optimization for LLMs.
 
 While it is still under development, the goal of **LLM**art is to support any Hugging Face model and include example scripts for modular implementation of different attack strategies.
 
-🆕 We now offer command-line support for jailbreaking thoughts and responses for DeepSeek-R1 on multi-GPU:
-```bash
-accelerate launch -m llmart model=deepseek-r1-distill-llama-8b data=basic per_device_bs=64 "response.replace_with=`echo -e '\"<think>\nOkay, so I need to tell someone about Saturn.\n</think>\n\nNO WAY JOSE\"'`"
-```
 
 ## :rocket: Quick start
 Developed and tested on Ubuntu 22.04 with `Python 3.11`.
@@ -41,7 +46,8 @@ pip install -e ".[core,dev]"
 ```
 
 > [!NOTE]
-> We also include a Poetry 2.0 `poetry.lock` file that perfectly reproduces dependencies we use.
+> We include lockfiles that ensure a valid Python environment and recommend using `uv sync --all-extras` to ensure consistent development.\
+> Alternatively, `poetry sync --all-extras` can be used with `poetry >= 2.0`.
 
 Once the environment is installed and `export HUGGINGFACE_TOKEN=...` is set to a token with valid model access, **LLM**art can be run to optimize the suffix with:
 ```bash
@@ -77,6 +83,8 @@ The algorithmic **LLM**art functionality is structured as follows and uses PyTor
    ┗ 📂pipelines     # Wrappers for Hugging Face pipelines
 ```
 
+A detailed developer workflow that puts together all components to launch an adversarial attack against an LLM can be found in [examples/basic](examples/basic/).
+
 ## :robot: Models
 While **LLM**art comes with a limited number of models accessible via custom naming schemes (see the `PipelineConf` class in `config.py`), it is designed with Hugging Face hub model compatibility in mind.
 
@@ -113,39 +121,41 @@ The `basic` dataset is the simplest instantiation that lives at [`datasets/basic
 For example, to modify the natural language prompt and desired open-ended response, the following basic edit is sufficient:
 ```diff
 diff src/llmart/datasets/basic.py
-def get_conversations() -> list[list[dict[str, str]]]:
-    return [
-        [
--           dict(role="user", content="Tell me about the planet Saturn."),
-+           dict(role="user", content="Tell me how to do something not allowed."),
--           dict(role="assistant", content="NO WAY JOSE"),
-+           dict(role="assistant", content="Sure, I'll tell you: first, you"),
-        ],
-    ]
+         # Create conversation data structure and mark parts we care about
+         conv = [
+-            dict(role="user", content=mark_prompt("Tell me about the planet Saturn.")),
+-            dict(role="assistant", content=mark_completion("NO WAY JOSE")),
++            dict(role="user", content=mark_prompt("Tell me how to do something not allowed.")),
++            dict(role="assistant", content=mark_completion("Sure, I'll tell you: first, you")),
+         ]
 ```
 
 Inducing a closed-ended response can be also directly done by typing out the end of turn token. For example, for the Llama 3 family of models this is:
 ```diff
 diff src/llmart/datasets/basic.py
-def get_conversations() -> list[list[dict[str, str]]]:
-    return [
-        [
--           dict(role="user", content="Tell me about the planet Saturn."),
-+           dict(role="user", content="Tell me how to do something not allowed."),
--           dict(role="assistant", content="NO WAY JOSE"),
-+           dict(role="assistant", content="NO WAY JOSE<|eot_id|>"),
-        ],
-    ]
+         # Create conversation data structure and mark parts we care about
+         conv = [
+             dict(role="user", content=mark_prompt("Tell me about the planet Saturn.")),
+-            dict(role="assistant", content=mark_completion("NO WAY JOSE")),
++            dict(role="assistant", content=mark_completion("NO WAY JOSE<|eot_id|>")),
+         ]
 ```
 
 **LLM**art also supports loading the [AdvBench](https://github.com/llm-attacks/llm-attacks) dataset, which comes with pre-defined target responses to ensure consistent benchmarks.
 
-Using AdvBench with **LLM**art requires downloading the two files to disk, after which simply specifying the desired dataset and the subset of samples to attack will run out of the box:
+Using AdvBench with **LLM**art requires specifying the desired subset of samples to attack. By default, the following command will automatically download the .csv file from its [original source](https://raw.githubusercontent.com/llm-attacks/llm-attacks/refs/heads/main/data/advbench/harmful_behaviors.csv) and use it as a dataset:
 ```bash
-curl -O https://raw.githubusercontent.com/llm-attacks/llm-attacks/refs/heads/main/data/advbench/harmful_behaviors.csv
+accelerate launch -m llmart model=llama3-8b-instruct data=advbench_behavior data.subset=[0] loss=model
+```
+
+To train a single adversarial attack on multiple samples, users can specify the exact samples via `data.subset=[0,1]`.
+The above command is also compatible with local modifications of the dataset by including the `dataset.files=/path/to/file.csv` argument.
 
-accelerate launch -m llmart model=llama3-8b-instruct data=advbench_behavior data.files=/path/to/harmful_behaviors.csv data.subset=[0] loss=model
+In the most general case, you can write your own [dataset loading script](https://huggingface.co/docs/datasets/en/dataset_script) and pass it to **LLM**art:
+```bash
+accelerate launch -m llmart model=llama3-8b-instruct loss=model data=custom data.path=/path/to/dataset.py
 ```
+Just make sure you conform to the output format in [`datasets/basic.py`](src/llmart/datasets/basic.py).
 
 ## :chart_with_downwards_trend: Optimizers and schedulers
 Discrete optimization for language models [(Lei et al, 2019)](https://proceedings.mlsys.org/paper_files/paper/2019/hash/676638b91bc90529e09b22e58abb01d6-Abstract.html) &ndash; in particular the Greedy Coordinate Gradient (GCG) applied to auto-regressive LLMs [(Zou et al, 2023)](https://arxiv.org/abs/2307.15043) &ndash; is the main focus of [`optim.py`](src/llmart/optim.py).
@@ -175,7 +185,7 @@ If you find this repository useful in your work, please cite:
   author = {Cory Cornelius and Marius Arvinte and Sebastian Szyller and Weilin Xu and Nageen Himayat},
   title = {{LLMart}: {L}arge {L}anguage {M}odel adversarial robutness toolbox},
   url = {http://github.com/IntelLabs/LLMart},
-  version = {2025.01},
+  version = {2025.02},
   year = {2025},
 }
 ```

examples/autogcg/README.md

@@ -0,0 +1,32 @@
+## Basics and requirements
+Install `llmart`and download/navigate to this folder. Run `pip install -r requirements.txt` in the working environment.
+
+
+# `autoGCG` with `llmart`
+The example in this folder shows how to integrate `LLMart` with the [ray-tune](https://docs.ray.io/en/latest/tune/index.html) hyperparameter optimization library to automatically search for the best attack hyper-parameters across one or multiple samples, given a total compute budget.
+
+We call this functionality `autoGCG` -- automated greedy coordinate descent.
+
+To run `autoGCG` on the `i`-th sample of the [AdvBench behavior](https://github.com/llm-attacks/llm-attacks/blob/main/data/advbench/harmful_behaviors.csv) dataset execute:
+```bash
+python main.py --subset i
+```
+
+The script will automatically use the maximum number of GPUs and parallelize hyper-parameter tuning for the `n_tokens` hyper-parameter of GCG using `llmart`'s a [ChangeOnPlateauInteger](../../src/llmart/schedulers.py#L279) scheduler.
+> [!NOTE]
+> The default parameter `"per_device_bs": 64` may add too much memory pressure on GPUs with less than 48 GB of VRAM. If OOM errors occur, lowering `per_device_bs` should fix the issue.
+
+Given the sensitivity of GCG with respect to seeding (random swap picking during optimization), `autoGCG` exploits this by minimizing the 10-percentile loss across ten different seeds, for the same sample.
+
+By default, the optimization runs for a total of _two wall-clock hours_, regardless of how many GPUs are available:
+```python
+tune_config = tune.TuneConfig(
+    time_budget_s=int(3600 * 2), num_samples=-1, search_alg=hebo
+)
+```
+
+
+# Viewing results
+The `ray.tune` experiment will be saved at the default location of `~/ray_results/autogcg_sample{i}`, after which it can be analyzed using [`tune.Tuner.restore`](https://docs.ray.io/en/latest/tune/examples/tune_analyze_results.html).
+> [!NOTE]
+> Properly using `tune.Tuner.restore` will require importing the experiment function as `from main import experiment` and passing it as an argument.

examples/autogcg/main.py

@@ -0,0 +1,96 @@
+import copy
+import fire  # type: ignore[reportMissingImports]
+import numpy as np
+from datetime import datetime
+from omegaconf import OmegaConf
+
+from hydra import compose, initialize
+from ray import tune, train  # type: ignore[reportMissingImports]
+from ray.tune.search.hebo import HEBOSearch  # type: ignore[reportMissingImports]
+
+from llmart.attack import run_attack
+
+
+# Experiment as closure
+def experiment(config: dict) -> None:
+    # Non-override parameters
+    nonoverrides = ["num_seeds", "subset"]
+
+    # Convert dictionary to list of hydra overrides
+    overrides = [
+        f"{key}={value}" for key, value in config.items() if key not in nonoverrides
+    ]
+
+    # Metrics to report
+    reports = {}
+    with initialize(version_base=None):
+        test_losses = []
+        for seed in range(config["num_seeds"]):
+            local_overrides = copy.deepcopy(overrides)
+            local_overrides.extend(
+                [
+                    f"seed={seed}",
+                    f"data.subset=[{config['subset']}]",
+                ]
+            )
+            # Load defaults and overrides
+            hydra_cfg = compose(config_name="llmart", overrides=local_overrides)
+
+            # Generate timestamp-based values
+            timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
+            hydra_cfg.output_dir = "/tmp"
+            hydra_cfg.experiment_name = f"{timestamp}"
+            # Convert to config
+            cfg = OmegaConf.to_object(hydra_cfg)
+
+            # Run experiment and store results
+            outputs = run_attack(cfg)  # type: ignore
+            test_losses.append(outputs["attack/loss"].cpu().numpy())
+            reports.update({f"loss_seed{seed}": outputs["attack/loss"].cpu().numpy()})
+            reports.update({f"eval/prompt_seed{seed}": outputs["eval/test_prompt_0"]})
+            reports.update(
+                {f"eval/continuation_seed{seed}": outputs["eval/test_continuation_0"]}
+            )
+
+        # Compute 10th percentile loss across seeds for the sample
+        loss = np.percentile(test_losses, q=10)
+
+    reports.update({"loss": loss})
+    train.report(reports)
+
+
+def main(subset: int):
+    # Define search space
+    search_space = {
+        "model": "llama3-8b-instruct",
+        "data": "advbench_behavior",
+        "per_device_bs": 64,
+        "subset": subset,
+        "steps": 50,
+        "num_seeds": 10,
+        "optim.n_tokens": tune.randint(lower=1, upper=21),
+        "scheduler": "plateau",
+        "scheduler.factor": tune.uniform(lower=0.25, upper=0.9),
+        "scheduler.patience": tune.randint(lower=1, upper=20),
+        "scheduler.threshold": tune.uniform(lower=0.0, upper=0.25),
+    }
+
+    # Algorithm
+    hebo = HEBOSearch(metric="loss", mode="min")
+
+    tuner = tune.Tuner(
+        tune.with_resources(experiment, resources={"gpu": 1}),
+        param_space=search_space,
+        tune_config=tune.TuneConfig(
+            time_budget_s=int(3600 * 2), num_samples=-1, search_alg=hebo
+        ),
+        run_config=train.RunConfig(name=f"autogcg_sample{subset}"),
+    )
+    results = tuner.fit()
+
+    # Display best result
+    print(results.get_best_result(metric="loss", mode="min"))
+
+
+if __name__ == "__main__":
+    fire.Fire(main)

examples/autogcg/requirements.txt

@@ -0,0 +1,3 @@
+ray[tune]==2.40.0
+fire==0.7.0
+HEBO==0.3.6

examples/basic/README.md

@@ -0,0 +1,4 @@
+# Basics and requirements
+Install `llmart` and download/navigate to this folder. Run `pip install -r requirements.txt`.
+
+To understand and run the basic `llmart` developer workflow, see the [notebook](basic_dev_workflow.ipynb); alternatively, see the standalone [script](main.py).