CoRL 2020, Spotlight Talk 35: Learning 3D Dynamic Scene Representations for Robot Manipulation

CoRL 2020

CoRL 2020, Spotlight Talk 35: Learning 3D Dynamic Scene Representations for Robot Manipulation

Dec 17, 2020
|
35 views
|
|
Code
Details
"**Learning 3D Dynamic Scene Representations for Robot Manipulation** Zhenjia Xu (Columbia University)*; Zhanpeng He (Columbia University); Jiajun Wu (Stanford University); Shuran Song (Columbia University) Publication: http://corlconf.github.io/paper_35/ **Abstract** 3D scene representation for robot manipulation should capture three key object properties: permanency โ€“ objects that become occluded over time continue to exist; amodal completeness โ€“ objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity โ€“ the movement of each object is continuous over space and time. In this paper, we introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics while capturing all three properties. We further propose DSR-Net, which learns to aggregate visual observations over multiple interactions to gradually build and refine DSR. Our model achieves state-of-the-art performance in modeling 3D scene dynamics with DSR on both simulated and real data. Combined with model predictive control, DSR-Net enables accurate planning in downstream robotic manipulation tasks such as planar pushing. Code and data are available at dsr-net.cs.columbia.edu.

Comments
loading...