論文 / arXiv:2611.08457
Adversarial Training with Large Step Sizes: Implicit Bias, Optimization, and Sharpness
PLAIN SUMMARY / やさしい要約
この研究は、学習の「一歩の大きさ」を大きく固定したまま、モデルを鍛える方法がどう振る舞うかを調べています。たとえるなら、細かく刻んで歩くのではなく、大股で進むときに、どんなクセが出るかを見るイメージです。その結果、ふつうの学習と比べて、たどり着きやすい解の傾向や、うまく近づく速さのルール、学習の不安定さの出方が違い得ることを示しています。対象は、まず単純なモデルでの解析、次にニューラルネットでの観察という流れです。
AIが専門用語を使わずに書いた解説です。内容の正確さは、下の原文の要旨で確認してください。
ABSTRACT / 要旨(原文)
Adversarial Training (AT) is an important method for improving model robustness. We study the dynamics of deterministic Gradient Descent (GD) in AT. Most existing works in this direction focus on the small or vanishing step sizes regime, where GD resembles a continuous-time flow. However, such step sizes can make training inefficient and are rarely used in practice. In this paper, we study the AT dynamics of GD with large constant step sizes. We first study GD in adversarial linear logistic regression, focusing on two aspects: where the model parameters converge (implicit bias) and how quickly the losses decrease (optimization). We then turn to neural networks training and study how the sharpness of losses evolve during AT. Our results advance the understanding of the behaviors of GD with large step sizes in AT and highlight three key differences from its behavior in Standard Training (ST): (i) the implicit bias gap between AT and ST, previously established only for small step sizes, persists for arbitrarily large constant step sizes, (ii) different from ST, there is no uniform rate of convergence for the optimization error in AT, and (iii) sharpness can exhibit two-stage dynamics that differ from the Edge-of-Stability behavior observed in ST.
ここに表示しているのは原論文の要旨です。AIによる要約や解釈は含みません。
