Learning to Detect Label Errors by Making Them: A Method
for Segmentation and Object Detection Datasets
Sarina Penquitt1 • Tobias Riedlinger2 • Timo Heller1 • Markus Reischl3 • Matthias Rottmann4
1 Department of Mathematics, University of Wuppertal, Wuppertal, Germany
2 Department of Mathematics, Technical University of Berlin, Berlin, Germany
3 Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
4 Institute of Computer Science, Osnabrück University, Osnabrück, Germany
Abstract
We present a unified method for detecting label errors in object detection, semantic segmentation, and instance segmentation datasets. Our approach - learning to detect label errors by making them – works as follows:
- We inject different kinds of label errors into the ground truth.
- Then, the detection of label errors, across all mentioned primary tasks, is framed as an instance segmentation problem based on a composite input. In our experiments, we compare the label error detection performance of our method with various baselines and state-of-the-art approaches of each task’s domain on simulated label errors across multiple tasks, datasets, and base models.
- This is complemented by a generalization study on realworld label errors.
- Additionally, we release 459 real label errors identified in the Cityscapes dataset and provide a benchmark for real label error detection in Cityscapes.
Method Overview
We propose the first unified method to detect label errors across object detection, semantic segmentation and instance segmentation by learning the detection of label errors. We simulate three different types of label errors by randomly perturbing the ground truth of a given datasets. Then, the detection of these simulated label errors are learned by an instance segmentation model, our label error detector, based on composite input. The predictions of the instance segmenter serve as label error proposals. This method detects simulated and real label errors.
Visualization of our label error detection method
Performance Results on Perturbed Segmentation Datasets
Metrics at a Glance
Performance Results on Perturbed Segmentation Datasets
Performance Results on Perturbed Object Detection Datasets
Qualitative Results
Examples of Found Real Label Errors in Cityscapes, ADE20K, LIVECell, PascalVOC and COCO
Resources & Links
Acknowledgments
S.P. and M.R. acknowledge support by the German Federal Ministry of Research, Technology and Space (BMFTR) within the junior research group project “UnrEAL” (grant no. 01IS22069).
Citation
@article{penquitt2025learninglabelerrors,
title={Learning to Detect Label Errors by Making Them: A Method for Segmentation and Object Detection Datasets},
author={Penquitt, S. and Riedlinger, T. and Heller, T. and Reischl, M. and Rottmann, M.},
journal={arXiv:2508.17930},
year={2025}
}
Contact
Have questions or want to collaborate? Reach out:
- Email: penquitt@uni-wuppertal.de
- GitHub Issues: open a ticket ↗