A good audio-visual model should understand what it hears and what it sees equally well. Often it doesn't, many models quietly favour one sense over the other. And standard benchmarks like VGGSound [1] can't catch this, because they never record whether a label can be heard or seen.
VGGSounder labels every class as heard, seen, or both, so you can check exactly what a model sees and hears, and quantify which modality it prefers.
Same videos, richer ground truth: every co-occurring label, tagged as audible, visible, or both.
What this makes measurable: modality profiling modality confusion controlled, confounder-free evaluation
This is the task our Mechanical Turk annotators faced. Watch the clip and decide, for each proposed label, whether it is audible, visible, both, or neither, then check your answers against VGGSounder's ground-truth modality annotations.
Scroll to explore more samples →
A model can answer correctly from audio alone, then lose that answer once it also sees the video. We call this modality confusion. Toggle the input below to see it happen.
VGGSounder's per-modality annotations let us profile every model, and even surface biases in the data they were trained on.
The emergence of audio-visual foundation models underscores the importance of reliably assessing their multimodal understanding. The VGGSound dataset is commonly used as a benchmark for evaluating audio-visual classification. However, our analysis identifies several limitations of VGGSound, including incomplete labelling, partially overlapping classes, and misaligned modalities. These lead to distorted evaluations of auditory and visual capabilities. To address these limitations, we introduce VGGSounder, a comprehensively re-annotated, multi-label test set that extends VGGSound and is specifically designed to evaluate audio-visual foundation models. VGGSounder features detailed modality annotations, enabling precise analyses of modality-specific performance. Furthermore, we reveal model limitations by analysing performance degradation when adding another input modality with our new modality confusion metric.
@inproceedings{zverevwiedemer2025vggsounder,
author = {Daniil Zverev and Thaddäus Wiedemer and Ameya Prabhu and Matthias Bethge and Wieland Brendel and A. Sophia Koepke},
title = {VGGSounder: Audio-Visual Evaluations for Foundation Models},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year = {2025}
}