論文 / arXiv:2610.20588
Your Transformer Is Secretly Elastic
PLAIN SUMMARY / やさしい要約
大きいモデルは速さやコストの都合で、いつも同じサイズのまま使うのが難しいことがあります。この研究は、学習し直さなくても、その場で「軽量版」に切り替えられるようにする方法を提案します。部品ごとの重要度を細かく並べ替えて、必要に応じて間引けるようにします。例えるなら、工具箱の中身を全部持ち歩かず、状況に応じて必要な工具だけ選んで持ち出せるようにする発想です。
AIが専門用語を使わずに書いた解説です。内容の正確さは、下の原文の要旨で確認してください。
ABSTRACT / 要旨(原文)
Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once. Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining. At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to $1.80\times$ higher throughput and $1.26\times$ lower single-query latency on an A100 GPU. Elastic models are produced in minutes, with construction time scaling sub-linearly with parameter count.
ここに表示しているのは原論文の要旨です。AIによる要約や解釈は含みません。
