Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video - INRIA 2 Access content directly
Conference Papers Year : 2024

Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video

Abstract

Self-supervised learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hourslong, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning. Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt imagebased pretraining approaches to incorporate more frames. Instead, we advocate a "tracking to learn to recognize" approach. Our method called DORA, leads to attention maps that Discover and tRAck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks.
Fichier principal
Vignette du fichier
dora_iclr.pdf (15.29 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04407117 , version 1 (20-01-2024)
hal-04407117 , version 2 (22-01-2024)

Licence

Attribution

Identifiers

  • HAL Id : hal-04407117 , version 2

Cite

Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki Asano, Yannis Avrithis. Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video. ICLR 2024 - Twelfth International Conference on Learning Representations, May 2024, Vienna, Austria. pp.1-21. ⟨hal-04407117v2⟩
74 View
29 Download

Share

Gmail Facebook X LinkedIn More