When told to understand the shape of a new object, the most instinctual approach is to pick it up and inspect it with your hand and eyes in tandem. Here, touch provides high fidelity localized information while vision provides complementary global context. However, in 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. In this paper, we study this problem and present an effective chart-based approach to fusing vision and touch, which leverages advances in graph convolutional networks. To do so, we introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects. Our results show that (1) leveraging both vision and touch signals consistently improves single-modality baselines, especially when the object is occluded by the hand touching it; (2) our approach outperforms alternative modality fusion methods and strongly benefits from the proposed chart-based structure; (3) reconstruction quality boosts with the number of grasps provided; and (4) the touch information not only enhances the reconstruction at the touch site but also extrapolates to its local neighborhood.
  |
|||
| Main Code: 10,496 LOC (18 files) = OBJ (86%) + PY (12%) + CU (<1%) + CPP (<1%) Secondary code: Test: 0 LOC (0); Generated: 0 LOC (0); Build & Deploy: 10 LOC (2); Other: 480 LOC (4); |
|||
| Duplication: 1% | |||
| File Size: 85% long (>1000 LOC), 12% short (<= 200 LOC) | |||
| Unit Size: 0% long (>100 LOC), 54% short (<= 10 LOC) | |||
| Conditional Complexity: 0% complex (McCabe index > 50), 89% simple (McCabe index <= 5) | |||
|
|
Logical Component Decomposition: primary (5 components) | ||
|
|
1 year, 1 month old
|
||
|
|
0% of code updated more than 50 times Also see temporal dependencies for files frequently changed in same commits. |
||
| Goals: Keep the system simple and easy to change (4) | |||
Latest commit date: 2021-09-21
|
0
commits
(30 days)
0
contributors
(30 days) |
|