Ph.D. Candidate: Fatih Çağatay Akyön
Program: Multimedia Informatics
Date: 01.09.2026 / 11:00
Place: A-212

Abstract: Grounded scene understanding identifies what is present in an image, where objects are located, and how they relate to one another. It is particularly important for systems that must justify their decisions when the concepts of interest are rare, domain-specific, and costly to miss. Sensitive-content moderation is a representative example. Current moderation systems typically return only a safe/unsafe label, without explaining what sensitive behavior was detected, who is involved, or where it appears.

This thesis addresses these limitations through four contributions: (1) SenBen, a benchmark for grounded and explainable sensitive-content understanding, (2) SenBen-Score, a recall-oriented evaluation metric, (3) a vocabulary-aware distillation framework for scene graph generation, and (4) a compact, deployable moderation model derived from a state-of-the-art vision-language model. SenBen comprises 13,999 movie frames from 157 films annotated with Visual Genome-style scene graphs (25 object classes, 28 attributes, and 14 predicates) and 16 sensitivity tags spanning 5 categories. Annotations were corrected by a domain expert and validated on a stratified subset by a second reviewer, yielding a Cohen's κ of 0.85 for sensitivity tags. SenBen-Score jointly evaluates objects, attributes, predicates, and sensitivity tags through a recall-weighted composite measure. A coverage analysis further shows that 75% of the sensitive vocabulary is absent from general lexical resources, motivating the release of a domain-specific synonym map.

To facilitate deployment, we distill Gemini into a 241M-parameter Florence-2 student and address vocabulary imbalance in autoregressive scene graph generation through suffix-based object identity prediction, a Vocabulary-Aware Recall (VAR) loss, and a decoupled Query2Label sensitivity-tag head. Averaged over six random seeds, the proposed training recipe improves SenBen Recall by 5.2 percentage points over standard cross-entropy training. Scaling to Florence-2-large (794M parameters) achieves a SenBen recall score of 0.441 while trading recall for precision. On the composite benchmark, the base student outperforms the evaluated vision-language models except Gemini, including GPT-5.2, Claude Opus, and all tested commercial safety APIs. It also achieves the highest object-detection and image-captioning scores while running 7.6× faster and requiring 16× less GPU memory than the strongest competing local model.