Learning Representational Invariances for
Data-Efficient Action Recognition

* indicates equal contribution
Motivation: Humans recognize actions in videos effortlessly, even in the presence of large photometric, geometric, temporal, and scene variations. Injecting these task-specific video invariances through data augmentations helps improve the data efficiency of video action recognition models.

Data augmentation is a ubiquitous technique for improving image classification when labeled data is scarce. Constraining the model predictions to be invariant to diverse data augmentations effectively injects the desired representational invariances to the model (e.g., invariance to photometric variations), leading to improved accuracy. Compared to image data, the appearance variations in videos are far more complex due to the additional temporal dimension. Yet, data augmentation methods for videos remain under-explored. In this paper, we investigate various data augmentation strategies that capture different video invariances, including photometric, geometric, temporal, and actor/scene augmentations. When integrated with existing consistency-based semi-supervised learning frameworks, we show that our data augmentation strategy leads to promising performance on the Kinetics-100, UCF-101, and HMDB-51 datasets in the low-label regime. We also validate our data augmentation strategy in the fully supervised setting and demonstrate improved performance.


Semi-Supervised Learning (3D ResNet-18)
Supervised Learning (R(2+1)D ResNet-34)
Cross-Dataset Semi-Supervised Learning (R(2+1)D ResNet-34)

  title={Learning Representational Invariances for Data-Efficient Action Recognition},
  author={Zou, Yuliang and Choi, Jinwoo and Wang, Qitong and Huang, Jia-Bin},
  journal={arXiv preprint arXiv:2103.16565},