Monocular visual odometry (VO) suffers severely from error accumulation during frame-to-frame pose estimation. In this paper, we present a self-supervised learning method for VO with special consideration for consistency over longer sequences. To this end, we model the long-term dependency in pose prediction using a pose network that features a two-layer convolutional LSTM module. We train the networks with purely self-supervised losses, including a cycle consistency loss that mimics the loop closure module in geometric VO. Inspired by prior geometric systems, we allow the networks to see beyond a small temporal window during training, through a novel a loss that incorporates temporally distant (e.g., O(100)) frames. Given GPU memory constraints, we propose a stage-wise training mechanism, where the first stage operates in a local time window and the second stage refines the poses with a "global" loss given the first stage features. We demonstrate competitive results on several standard VO datasets, including KITTI and TUM RGB-D.

@inproceedings{zou2020learning,
    author    = {Zou, Yuliang and Ji, Pan and and Tran, Quoc-Huy and Huang, Jia-Bin and Chandraker, Manmohan}, 
    title     = {Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling}, 
    booktitle = {European Conference on Computer Vision},
    year      = {2020}
}

• • Zhou et al., “Unsupervised Learning of Depth and Ego-Motion from Video,” CVPR 2017.
• • Yin and Shi, “GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose,” CVPR 2018.
• • Zou et al., “DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency,” ECCV 2018.
• • Ranjan et al., “Adversarial Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation,” CVPR 2019.
• • Xue et al., “Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry,” CVPR 2019.
• • Godard et al., “Digging Into Self-Supervised Monocular Depth Estimation,” ICCV 2019.
• • Teed and Deng, “DeepV2D: Video to Depth with Differentiable Structure from Motion,” ICLR 2020.