Addressing Multiple Object Tracking with Segmentation Masks

Abstract

Multiple Object Tracking (MOT) aims to locate all the objects from a video, assigning them the same identities across all frames. Traditionally, this problem was addressed following the Tracking by Detection (TbD) paradigm, using detections represented by bounding boxes. However, bounding boxes can contain information from several objects, something that does not happen with segmentation masks. This work takes the ByteTrack MOT system as a starting point. Our proposal, ByteTrackMask, integrates a class-agnostic segmentation method and a segmentation-based tracker in ByteTrack in order to rescue tracks that would have been lost. Results over validation sets of MOT challenge datasets provide improvements in MOT metrics of interest like MOTA, IDF1 and false negatives.