論文 / arXiv:2610.06804
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution
PLAIN SUMMARY / やさしい要約
言語モデルは、正しい答えが「1つずつの間違い」よりは起こりやすくても、たくさんの間違いが合計すると間違いを引きやすいことがあります。そこで、いちばんありそうな答えに確率を寄せる方法（power distribution）がありますが、たくさんの候補を出して点を付ける手間が必要でした。この研究は、その「候補をたくさん試す」効果を、なるべく1回の出力で出せるようにモデルを学習させる方法を提案します。たとえるなら、何度も試行して当たりを引くのではなく、当たりを引きやすい引き方を練習して身につける発想です。
AIが専門用語を使わずに書いた解説です。内容の正確さは、下の原文の要旨で確認してください。
ABSTRACT / 要旨（原文）
A language model can give a correct answer more probability than any single incorrect answer and still usually sample an incorrect one, because the incorrect answers together hold more probability. The power distribution raises each complete answer's probability to a power above one and renormalizes, shifting probability toward answers the model finds most likely (sharpening). Sampling from it improves reasoning without changing parameters, but needs many scored candidates per query. We show that a model can instead be trained to produce such answers in one generation. On-policy power distillation (OPPD) runs a sequential Monte Carlo sampler in which the model being trained generates candidates and a frozen teacher's power distribution weights them; the same probabilities weight each answer in a maximum-likelihood update. Training raises single-generation accuracy by up to 23.0 points on MATH500 and 27.3 on GSM8K over the untrained model at the same temperature, and one generation scores 2.4 and 3.5 points above published power sampling with 64 candidates, recovering 94 percent of the gain that 16 candidates give the untrained model. For context, against GRPO trained with verified rewards from the same checkpoint and budget, OPPD scores 3.8, 4.0 and 5.4 points higher on MATH500, GSM8K and AIME using no reference answers; the two are complementary, and OPPD applied after GRPO adds up to 9.3 points. Trained only on mathematics, OPPD raises HumanEval accuracy by up to 5.3 points. One loss coefficient moves the sharpening exponent the model absorbs between 1.19 and 2.02, against 1.14 for ordinary on-policy distillation, and it rises mostly on the model's own answers. Gains hold across model families and sizes, including a model already trained with verified rewards, where lowering the temperature gives nothing and OPPD adds 4.4 points on MATH500. Code: https://github.com/ArminAzizi98/OPPD.
ここに表示しているのは原論文の要旨です。AIによる要約や解釈は含みません。