Towards Adversarially Robust Dataset Distillation by Curvature Regularization

Eric Xue1, Yijiang Li2, Haoyang Liu3, Peiran Wang3, Yifan Shen3, Haohan Wang3,
1University of Toronto, 2UC San Diego, 3University of Illinois Urbana-Champaign
Accepted by AAAI 2025
Website Paper Code

Abstract

Dataset distillation (DD) allows datasets to be distilled to fractions of their original size while preserving the rich distributional information so that models trained on the distilled datasets can achieve a comparable accuracy while saving significant computational loads. Recent research in this area has been focusing on improving the accuracy of models trained on distilled datasets. In this paper, we aim to explore a new perspective of DD. We study how to embed adversarial robustness in distilled datasets, so that models trained on these datasets maintain the high accuracy and meanwhile acquire better adversarial robustness. We propose a new method that achieves this goal by incorporating curvature regularization into the distillation process with much less computational overhead than standard adversarial training. Extensive empirical experiments suggest that our method not only outperforms standard adversarial training on both accuracy and robustness with less computation overhead but is also capable of generating robust distilled datasets that can withstand various adversarial attacks.

Motivation

How can we embed adversarial robustness into the dataset distillation process, thereby generating datasets that inherently lead to more robust models? While embedding adversarial training directly within the dataset distillation process may seem like an intuitive and straightforward approach, our comprehensive analysis reveals its limitations across various distillation methods.

Motivation for robust dataset distillation

Accuracy of ResNet18 on ImageNette trained on distilled datasets from GUARD, SRe2L, and SRe2L with adversarial training

Robust Dataset Distillation

We formulate robust dataset distillation as a tri-level optimization problem, aiming to find distilled data set $\mathcal{S}^*$ that minimizes the expected worst-case adversarial loss. Specifically, the objective is to ensure that models trained on the distilled data not only achieve high accuracy but also exhibit strong adversarial robustness.

By leveraging a quadratic Taylor expansion of the loss function and assuming convexity, we derive an upper bound for the adversarial loss on the distilled data:

$$ \tilde{\ell}_\rho^{adv}(\mathbf{x}^\prime) \le \mathbb{E}_{\mathbf{x} \sim D_c} \ell(\mathbf{x}) + \frac{1}{2}\rho^2\, \mathbb{E}_{\mathbf{x} \sim D_c} \lambda_1(\mathbf{x}) + L\sigma, $$

where $\lambda_1$ denotes the largest eigenvalue of the Hessian (curvature) of the loss, and $L\sigma$ captures the feature discrepancy between the distilled and real data distributions.

Implication: The bound highlights that minimizing the curvature term $\lambda_1$ via geometric regularization is crucial for adversarial robustness in distilled datasets. By explicitly regularizing the curvature during distillation, we can generate synthetic datasets that inherently lead to more robust models.

Experiments

We conduct extensive experiments to evaluate the effectiveness of GUARD on standard benchmarks, including ImageNette and TinyImageNet and ImageNet-1K. We take the following two metrics into consideration:

  • Clean accuracy: The performance of models trained on distilled datasets when evaluated on unperturbed test data.
  • Adversarial robustness: The performance under various adversarial attacks, such as PGD, Square, AutoAttack, CW, and MIM.
Figure 1

Qualitative:

Distilled Images

Examples of distilled images from GUARD (top) and SRe2L (bottom) on ImageNet-1K.

Ablation Study:

Ablation on Curvature Regularization

Effect of different $\lambda_g$. ccuracy on ImageNette.

Ablation on Feature Discrepancy

Effect of GUARD regularizer, on DC, SRe2L, and CDA. Performance of GUARD regularizer ismarked $\dagger$.

Computation Overhead:

Computation Overhead

Computation overhead of GUARD compared with embedded adversarial training

Conclusion: GUARD provides a simple yet effective way to endow distilled datasets with adversarial robustness, outperforming existing methods in both robustness and efficiency. Our code and distilled datasets are available at https://github.com/yumozi/GUARD.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{xue2025towards,
    author = {Eric Xue and Yijiang Li and Haoyang Liu and Peiran Wang and Yifan Shen and Haohan Wang},
    title = {Towards Adversarially Robust Dataset Distillation by Curvature Regularization},
    booktitle = {Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25)},
    year = {2025},
}