STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (ECCV'20)
Aug 18, 202052 views
European Conference on Computer Vision 2020 Paper: https://arxiv.org/abs/2003.08429 Code: https://github.com/sabarim/STEm-Seg Project Page: https://www.vision.rwth-aachen.de/publication/00202/ Ali Athar*, Sabarinath Mahadevan*, Aljoša Ošep, Laura Leal-Taixé, Bastian Leibe ABSTRACT: Existing methods for instance segmentation in videos typically involve multi-stage pipelines that follow the tracking-by-detection paradigm and model a video clip as a sequence of images. Multiple networks are used to detect objects in individual frames, and then associate these detections over time. Multiple networks are used to detect objects in individual frames, and subsequently associate them over time. Hence, these methods are often non-end-to-end trainable and highly tailored to specific tasks. In this paper, we propose a different approach that is well-suited to a variety of tasks involving instance segmentation in videos. In particular, we model a video clip as a single 3D spatio-temporal volume, and propose a novel approach that segments and tracks instances across space and time in a single stage. Our problem formulation is centered around the idea of spatio-temporal embeddings which are trained to cluster pixels belonging to a specific object instance over an entire video clip. To this end, we introduce (i) novel mixing functions that enhance the feature representation of spatio-temporal embeddings, and (ii) a single-stage, proposal-free network that can reason about temporal context. Our network is trained end-to-end to learn spatio-temporal embeddings as well as parameters required to cluster these embeddings, thus simplifying inference. Our method achieves state-of-the-art results across multiple datasets and tasks.