Tailoring Input Devices for Digital Twins: A Software Engineering Approach to Operator-Digital-Twin Interaction

Context

A digital twin is a virtual representation of a target entity with data connections that enable convergence between the physical and digital states at an appropriate rate of synchronisation, enabling simulation, integration, testing, monitoring, training and maintenance. A digital twin can provide an integrated view throughout the life cycle of the target entity. Digital Twins also play a pivotal role in optimising complex systems across a wide range of domains, from industrial manufacturing and energy to environmental monitoring and healthcare.

The Engineering Digital Twin EDT program, funded by the France 2030 investment plan, is a national initiative to advance the foundations of digital twin engineering in France and across Europe. By bringing together leading academic and industrial partners, EDT seeks to strengthen the bases for the design, use, and deployment of digital twins, addressing key open challenges in model hybridisation, composability, development methodologies, digital coupling, and human-twin interaction.

A digital twin fundamentally depends on data visualisation [1], allowing all stakeholders throughout its lifecycle to interact with it via their chosen input and output devices. Output devices can take multiple forms, from fully digital virtual reality immersive systems to hybrid augmented reality systems [2] or a mix of both in asymmetric scenarios, as well as classical 2D graphical user interfaces or interactive video wall systems. In practice, considering the diversity of users and the relatively long life cycle of systems accurately replicated by digital twins, it is highly probable that most of these devices will be used with digital twins. Input devices are also diverse, ranging from direct hand control (e.g., (multi-)touch interaction) to indirect hand control (e.g., mouse, keyboard), but also voice- or gaze-based input devices. There also exist specialised devices such as the cubic mouse [3] or the sphere designed for 3D interactions.

Users interact with a digital twin to perform various tasks. Some tasks are generic to digital twins, such as sending (resp. receiving) commands to update the real (resp. virtual) system, or navigating across time and heterogeneous, multi-scale visual representations. Others are specific to a particular digital twin, such as sketching a Bezier curve with a HoloLens to capture the geometric modifications to a real system needed to update the virtual replica.

These considerations highlight the importance of ensuring that all users interacting with a digital twin throughout its life cycle can utilise input devices best suited to their needs, preferences, tasks, and output devices.

Thesis Objectives

Instead of developing (self-)adaptive capabilities to ensure the virtual environment accommodates a range of input and output devices, this PhD focuses on developing new software engineering methods and tools that enable input devices to be reliably tailored to the task performed using the digital twin, the output device, and the type(s) of interactive visual representations that make up the digital twin, and that align with users’ preferences. The software engineering methods and tools proposed to support the tailoring of input devices will, in the short term, aim to facilitate the reconfiguration of existing input devices to adapt to a new task, output device, or user profile. However, we anticipate that this contribution will pave the way for developing new capabilities to predict performance for the a priori evaluation of input devices – either to reconfigure existing ones or to develop new ones.

As a starting point, we will limit the work to configurations that involve the most common input and output devices, including: 1) a mouse and keyboard used with a PC; 2) multi-touch for an interactive video wall system; and 3) XR HMDs, if possible. Based on progress and case studies, we might modify the input and output devices considered. The choice of input and output devices, the tasks that require adapting input devices, and user profiles will primarily depend on the use cases used to validate the claimed contribution.

Several use cases from the EDT project are of interest and are under validation, but we will very likely work on the KOPR digital twin  developed by the datathings company. Kopr is a DT designed for the supervision and operational analysis of electricity distribution networks. Kopr provides a map view for visualizing the state of the electrical grid, displaying all the physical elements of the high-, medium-, and low-voltage grid, are represented by dedicated icons and enabling simulation of changes such as installing new cables or connecting new consumers. Kopr illustrative video.

Using KOPR, various stakeholders interact with KOPR digital twin to perform various tasks, including:

• grid planners exploiting the map view with added visual information (a heatmap display) for identifying risk areas on lowvoltage,
• maintenance managers relying on prioritized equipment faults to deploy corrective actions,
• longterm planners exploiting the clustered view of power consumption to anticipate future electricity demand
• …

Close cooperation with the datathings company will be setup, which will share its access to the underlying system and its end-users. The PhD candidate will be able to investigate the generalisability of developed approaches and tools by applying them to other use cases from the EDT project

In this PhD, the digital twin is not just the artefact users interact with but also a tool to adapt the chosen input devices. Indeed, digital twins help simulate and manage input and output devices [4], making them suitable candidates for designing and developing usable and reliable input devices for interacting with digital twins. Thus, to overcome the issue presented above, the PhD candidate will develop a digital twin of the interactive system, including relevant aspects of the user, the input device, and the interactions between the user and the digital twin of the system of interest through the input and output devices.

The global approach, illustrated in Figure 2, relies on the digitalisation of the user and the entire interactive system (hardware, interaction technique, and interactive application) to build and study interactions with the digital twin (system digital twin).

The position is based at Université de Toulouse in Toulouse, within the ICS team at IRIT lab.

The PhD candidate will be co-supervised by Camille Fayollas from Université Toulouse Capitole, Philippe Palanque from Université de Toulouse, and Romain Pinquié from G-SCOP (Grenoble INP-UGA).

The PhD candidate will benefit from a stimulating scientific and industrial environment of the highest level, with access to a national network of leading research institutions and industry partners, regular interactions with the broader EDT community through workshops, seminars, and joint demonstrators, and the opportunity to contribute to Artemis, the program's open software platform.

Qualifications

Required

• Master degree in computer science with a focus on software engineering and/or humancomputer interaction
• Proficiency in programming languages (e.g., Java)
• Excellent written and oral communication skills in English and French
• Strong analytical and problemsolving abilities
• Initiative, curiosity, inventiveness

Preferred

• Knowledge of modelling, formal modelling (e.g., Simulink, Petri nets)
• Familiarity with realtime systems
• Previous research experience in HCI or 2D/3D applications or related fields