論文 / arXiv:2610.08789
QF3: Fast Flow RL with Filtered Q-Gradients
PLAIN SUMMARY / やさしい要約
ロボットの動きを学ぶ方法の一つに、見本データから学んだモデルを、試行錯誤でさらに良くするやり方があります。QF3は、この「試行錯誤で良くする」部分を、過去の経験データも使いながら進めるための手法です。学習の手がかりとして、評価役の情報を使いますが、使う範囲を「過去の行動の近く」に限定します。料理でいえば、味見の結果を反映する際に、レシピから大きく外れた変更は避けて、近い調整だけ行うイメージです。
AIが専門用語を使わずに書いた解説です。内容の正確さは、下の原文の要旨で確認してください。
ABSTRACT / 要旨(原文)
Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. We introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates where the critic and this prediction are reliable, QF3 applies the critic gradient only to action dimensions that stay near the replay action. To our knowledge, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware. Paired with a high-throughput off-policy training recipe, it trains humanoid locomotion and motion-tracking policies with a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method. We further apply QF3 to fine-tune pretrained flow-based manipulation policies on both ABC-Sim and Robomimic tasks. These results suggest that QF3 can both learn robot policies from scratch and refine those acquired from demonstrations. Website: https://qf3-rl.github.io/
ここに表示しているのは原論文の要旨です。AIによる要約や解釈は含みません。
