論文 / arXiv:2610.22363
Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
PLAIN SUMMARY / やさしい要約
この研究は、手元を映した動画から「手がどう動いたか」を読み取り、次に何が起きるかを予測し、さらに目的に向けた動きまで考える仕組みを、1つのモデルでまとめようとするものです。状況に応じて、動画や動きの情報を「手がかり」にも「当てるべき答え」にもする学び方を使います。実験では、別々に学ぶよりまとめて学ぶ方が、複数の能力が良くなったと述べています。たとえるなら、見る・先を読む・段取りを組むのを、別々の訓練ではなく同じ教室で一緒に鍛えるような発想です。
AIが専門用語を使わずに書いた解説です。内容の正確さは、下の原文の要旨で確認してください。
ABSTRACT / 要旨(原文)
We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/
ここに表示しているのは原論文の要旨です。AIによる要約や解釈は含みません。
