Optimized Transmission of Spherical Video Streams for Immersive Teleoperation of Autonomous Robots

Context and Motivation 

This PhD project is part of a new collaboration between the DRIVE Laboratory at Université Bourgogne Europe on the Nevers campus in France (https://drive.ube.fr/) and the CNRS-AIST JRL Laboratory, IRL 3218 in Tsukuba, Japan (http://jrl.cnrs.fr).

Robot teleoperation is a key challenge in many fields, from exploration in hazardous environments (disaster zones, dangerous industrial sites, underwater or space missions) to the maintenance of critical infrastructure and human assistance. The use of 360° cameras enhances the operator’s perception and immersion, thereby facilitating remote control and interaction with the environment.

However, real-time transmission of such high-definition and non-conventional geometry video streams presents several major challenges:

1. Bandwidth optimization: How can we efficiently transmit spherical video while ensuring low latency and high quality where it matters most?
2. Remote 3D reconstruction and SLAM: How can visual analysis and simultaneous mapping be combined for better perception and navigation?
3. Context-aware and network-constrained adaptation: How can transmission be dynamically adjusted based on operator needs and available resources?

This PhD position aims to explore advanced strategies such as adaptive tiling, intelligent multicast, and spherical mesh optimization to enhance video streaming in a shared-control robot teleoperation context.

Scientific and Technical Objectives 

The main goal is to design an efficient transmission framework enabling smooth interaction between a remote operator and a robot equipped with 360° cameras, while minimizing network and computational resource usage.

1. Optimized Transmission of Spherical Video Streams

• Development of an adaptive spherical tiling strategy, transmitting only the most relevant regions in high definition. Leveraging contextual information:
  • Operator's gaze to prioritize visible tiles.
  • Scene dynamics to detect the most informative regions.
• Integration of an intelligent multicast mechanism, enabling differentiated transmission depending on corresponding recipients (operator, vision algorithms, archiving, etc.).

2. Distributed 3D Reconstruction and SLAM for Enhanced Teleoperation

• Development of a method combining distributed SLAM and selective tile transmission based on visual information density.
• Optimizing the tradeoff between visual quality (for the operator) and 3D reconstruction accuracy (for robot navigation).
• Integration of dynamic network and computing resource management to adapt to latency and bandwidth constraints.

3. Validation and Experiments on Various Robot Platforms

• Experiments will be conducted on humanoid robots at the CNRSAIST JRL lab in Japan, and on drone fleets at the DRIVE lab in Nevers, to evaluate the effectiveness of the proposed approach in diverse, real-world scenarios.
• Performance analysis will include latency, robustness, and perceived teleoperation quality.

Scientific and Technical Challenges

• How to ensure realtime, low-latency transmission of immersive video streams under constrained networks?
• What is the optimal spherical tiling and prioritization strategy to balance immersion and 3D reconstruction performance?
• How to design an adaptive and intelligent multicast system optimized for heterogeneous recipients with varying requirements?
• What is the impact of video stream latency and jitter on teleoperated driving of humanoids or drones?

Applications and Impact

The outcomes of this thesis can be applied in multiple domains:

• Exploration and intervention in extreme environments (e.g., rescue robots in disaster zones, space missions).
• Inspection and maintenance of critical infrastructure (e.g., industrial or underwater inspection robots).
• Surveillance and security (e.g., autonomous patrols with humanin-the-loop).
• Humanrobot assistance and interaction (e.g., teleoperated humanoid assistance robots).

Expected profile:

Applicants must hold a Master’s or Engineering degree in Computer Science. Solid knowledge in Artificial Intelligence, including machine learning and deep learning, as well as practical skills in programming and software tools (e.g., Python, C++) are required. Fluent English (written and spoken) is also essential. Candidates must be highly motivated, quick learners, and able to work effectively on challenging research problems.

Funding:

This position is supported by dual funding:

• The first phase will take place in Nevers, France (funded by a European project).
• The second phase will be conducted in Tsukuba, Japan (funded by AIST Japan).

Important dates:

Application deadline: Early May 2025

Expected start date: September / October 2025