Spatio‐Temporal Video Grounding to Extract Features Needed to Characterize YouTube Video Genres

Efrei Paris, école d’ingénieurs, composante de l’Université Paris-Panthéon-Assas, est un établissement privé d’enseignement supérieur technique, reconnu par l’Etat, EESPIG, dont dépend le laboratoire Efrei Research Lab, dirigé par Etienne PERNOT.

L’Efrei Research Lab est le laboratoire de recherche de l’Efrei. Il se compose d’une cinquantaine d’enseignants-chercheurs en informatique et électronique ainsi que d’autant de doctorants. Depuis janvier 2022, en intégrant l’université Paris-Panthéon-Assas, Efrei Research Lab est reconnu comme le laboratoire numérique de l’Université, unité de recherche 202224306D, rattaché à l’école doctorale ED 455 EGIC, délivrant le doctorat en informatique.

Ses domaines de recherche se concentrent sur les domaines du numérique à travers quatre axes :

• données et Intelligence Artificielle ;
• sécurité, résilience et confiance numérique ;
• réseaux de communication ;
• systèmes embarqués intelligents.

Le Laboratoire se concentre sur de la recherche appliquée avec deux domaines d’applications majeurs : les sciences du vivant (santé, agriculture et biodiversité, sport, éducation) et les territoires intelligents (entreprises, habitations, réseaux).

L’Efrei Research Lab s’est engagé dans la mise en œuvre de sa responsabilité sociétale vis-à-vis des enjeux environnementaux à travers l’ensemble des activités du laboratoire. Le chercheur de l’Efrei Research Lab s’engage à prendre en compte la transition écologique pour un développement soutenable dans ses activités de recherche menées au sein de l’Efrei Research Lab.

Context & Motivation

YouTube video genres are often defined not only by their content but also by their structural patterns—who or what appears on screen, when, and for how long. Recent advances in grounding models (e.g., VideoMolmo [1]), combined with lightweight audio/text alignment, enable interpretable detection and tracking of faces, hands, props, overlays, and scene changes. These grounded cues provide reliable, explainable signals for downstream genre characterization.

Examples of Video Features

• Ads / Sponsor Segments: Detection of sponsor bumpers, logo overlays, abrupt shot-template changes, and call-to-action screens.
• Discussion vs. Monologue: Dominant-face alternation, active-speaker overlap (interruptions), reaction-shot cadence, and split-screen dwell time.
• Demonstration / Tutorial: Sustained prop or tool focus, hand-object manipulation sequences, top-down camera views, and screen recordings.
• News Explainer: Presenter with graphic boards or lower-thirds, B-roll interleaving patterns, and infographic persistence.
• Podcast Panel: Multi-face persistence, stable seating layout, minimal B-roll, long speaking turns, and limited use of props.

Objectives

This internship focuses on studying and implementing grounding‐based video models to extract explainable, spatio‐temporally coherent cues that characterize the structure of YouTube videos across genres. The goal is to identify structural signatures of specific video types—such as ads or sponsor insertions, discussions versus monologues, panel interactions, tutorial demonstrations, news explainers, and podcast panels. The outcome will be a compact, reproducible pipeline that transforms raw videos into grounded timelines and genre‐discriminative feature vectors.

Main Tasks

• Model selection (SOTA): survey video grounding approaches; select one primary model (e.g., VideoMolmo‐style pointing/grounding or an alternative)
• Grounded feature design : define a small set of interpretable spatio‐temporal features capturing the characteristics above.
• Multimodality : Use automatic speech recognition to transcribe the audio and speaker identification to determine who is talking and whether people speak at the same time ; Mini‐benchmark : evaluate grounding quality, feature reliability, and genre discrimination on a curated YouTube subset.

Data :

• Public dataset : AVA‐ActiveSpeaker, TVQA+ [2] for pretraining/validation convenience.
• Mini‐corpus : ~10–12 hours across 3–4 genres chosen at kickoff (e.g., debate panel, interview/monologue, tutorial, news explainer, podcast panel) with Efrei Research Lab Partners.

References

1. Ahmad, G.S., Heakl, A., Gani, H., Shaker, A.M., Shen, Z., Krishna, R., Khan, F.S., & Khan, S.H. (2025). VideoMolmo: Spatio-Temporal Grounding Meets Pointing.ArXiv, abs/2506.05336. https://github.com/mbzuai-oryx/VideoMolmo

2. Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. 2020. TVQA+: Spatio-Temporal Grounding for Video Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8211–8225, Online. Association for Computational Linguistics.

Artificial Intellgience Enginneer or Master (Bac+5) with strong theoritical background on AI methods, Python, PyTorch, Computer Vision (tracking/transformers), basics of ASR/diarization