Exploring Open-Vocabulary Models for Category-Free Detection

Abstract

Object detection models typically rely on a predefined setof categories, limiting their applicability in real-world scenarios whereobject classes may be unknown. In this paper, we propose a novel,training-free framework that enables off-the-shelf open-vocabulary ob-ject detectors (OvOD) to perform category-free detection —localizingand classifying objects without any prior category knowledge. Our ap-proach leverages image captioning to dynamically generate descriptiveterms directly from the image content, followed by a WordNet-based fil-tering process to extract semantically meaningful category names. Thesediscovered categories are then embedded and matched with visual regionfeatures using a frozen OvOD model to perform detection. We evaluateour method on the COCO dataset in a fully zero-shot setting and demon-strate that it significantly outperforms strong multimodal large languagemodel baselines, achieving an improvement of over 30 AP points. Thishighlights our method as a promising direction for more adaptive solu-tions to real-world detection challenges.