Vision-Based Automatic Calibration and Synchronization of Digital Twins for Cyber-Physical Production Systems Using Neuromorphic Sensors

SUMMARY

The emergence of Digital Twin (DT) technologies has profoundly transformed industrial production by enabling high-fidelity virtual replicas of physical systems. According to the recent report of the Alliance Industrie du Futur (AIF), a Digital Twin is defined as an organized ensemble of digital models, continuously updated with respect to reality and equipped with advanced computational and operational tools. In the context of production equipment, achieving and maintaining calibration and synchronization between the Digital Twin and its Physical Twin remains a significant scientific and industrial challenge, particularly during commissioning and throughout operational use.

The current industrial strategy to improve observability relies heavily on industrial IoT technologies. This generally leads to an “over-instrumentation” of the physical system, increasing costs, complexity, and potential reliability issues. In contrast, industrial vision technologies—now widely deployed for product quality inspection and robotic guidance—have recently benefited from major breakthroughs, including 3D imaging and embedded machine-learning capabilities. Despite this progress, their use remains largely restricted to product-centric tasks rather than system-centric observation.

The proposed research internship addresses this gap by exploring the use of neuromorphic vision as an alternative high-performance sensing modality for the automatic calibration and synchronization of Digital Twins of production equipment. Neuromorphic cameras, based on event-driven asynchronous sensing, provide high temporal resolution and low latency, making them particularly suited for observing dynamic behaviors in automated systems. Early work conducted in 2024 has demonstrated the feasibility of characterizing the mechanical behaviour of a pneumatic cylinder using event-based vision. The next steps, which constitute the core objectives of this internship, include:

• Simultaneous observation and behavioural characterization of multiple actuators using a neuromorphic camera.
• Identification of sensor and actuator behaviour (including distributors and end-effectors) through correlation between camera-detected events and PLC I/O transitions.
• Integration of energy consumption data (electrical and pneumatic) to enrich multi-modal behavioural modelling.

Ultimately, the aggregation of these component-level behavioural models will support the automatic construction and calibration of a Digital Twin of production equipment, grounded in discrete event system (DES) theory.

DETAILED SUBJECT

The concept of Cyber-Physical Systems (CPS), formalized in early NSF reports (notably [1], [2]), has become central to contemporary research in systems engineering and intelligent manufacturing. A CPS is typically characterized by:

(i) multi-scale temporal and spatial dynamics,

(ii) high degrees of automation, connectivity, and embedded intelligence, and

(iii) the capacity for dynamic reconfiguration in response to failures or functional modifications.

This internship focuses more specifically on Cyber-Physical Production Systems (CPPS) within the class of Reconfigurable Manufacturing Systems (RMS). In this context, the definition of the Digital Twin aligns with the viewpoint of industrial systems engineers. The recent AIF report [3] describes a DT as:

• an organized collection of models representing a real-world entity for specific uses;
• updated at a frequency and fidelity adapted to its purpose;
• equipped with advanced tools enabling understanding, analysis, prediction, and optimization of the real system.

The scientific objective of this project is to investigate the dynamic updating problem, i.e., the calibration and synchronization of the Digital Twin relative to its physical counterpart. Traditional calibration strategies rely on (i) observable states from the control system and (ii) additional sensors from industrial IoT. While effective, such approaches result in substantial instrumentation requirements, with direct consequences for cost, robustness, and maintainability.

In parallel, industrial vision has become pervasive in manufacturing for tasks such as quality inspection and robotic handling. Modern vision systems (e.g., Cognex, Keyence, Sensopart) increasingly integrate 3D capabilities and machine learning algorithms. However, their usage remains highly product-centric. The proposed research aims to shift this paradigm by exploiting neuromorphic vision as a holistic observation tool for system-level behaviour characterization. Earlier research [6] demonstrated that event-based vision from a neuromorphic camera can identify characteristic mechanical states of a pneumatic actuator and reconstruct its behaviour using discrete event models. Building upon these results, the internship will address three research challenges:

1. Multi-Actuator Behaviour Observation

Develop methods to observe, discriminate, and model the behaviour of multiple actuators simultaneously from event-based visual streams.

2. Cross-Modal Correlation between Camera Events and PLC I/O Signals

Characterize additional system components—sensors, distributors, logic-controlled modules—by correlating event streams with transitions recorded by the programmable logic controller.

3. Integration of Energy-Related Observables

Incorporate electrical and pneumatic energy consumption signals into behaviour modelling for richer, multi-modal Digital Twin construction.

The resulting behavioural models will be aggregated through discrete event system (DES) theory—potentially using chronicles, timed automata, or other temporal formalisms—to generate an executable, dynamically updated Digital Twin of the production equipment.

EXPECTED DELIVERABLES

• A comprehensive state of the art on neuromorphic sensing in industrial environments, temporal modelling techniques for discrete event systems (including chronicles), and behaviour identification methods.
• A methodology for automatic behavioural identification of the physical twin from observed multimodal events, and automatic generation of DES-based models.
• A functional demonstrator using a Prophesee event-based camera to detect, segment, and model behaviours on an industrial production machine (e.g., bottle-packaging line).
• A scientific publication (conference or journal submission) presenting the methodological contributions and experimental results.

Bibliographie

[1]      E. A. Lee, “Cyber-Physical Systems - Are Computing Foundations Adequate ?,” in Position Paper for NSF Workshop On Cyber-Physical Systems, 2006, vol. 2, pp. 1–9.

[2]      L. Monostori, “Cyber-physical production systems: Roots, expectations and R\&D challenges,” Procedia CIRP, vol. 17, pp. 9–13, 2014, doi: 10.1016/j.procir.2014.03.115.

[3]      http://www.industrie-dufutur.org/content/uploads/2023/05/AIF_JumeauNumerique_FR-version-Web.pdf

[4]      Guyet, Thomas, et Philippe Besnard. Chronicles: Formalization of a Temporal Model. SpringerBriefs in Computer Science. Cham: Springer International Publishing, 2023. https://doi.org/10.1007/978-3-031-33693-5.

[5]      Alexandre Sahuguède, Euriell Le Corronc, Marie-Véronique V Le Lann. Chronicle Discovery for Diagnosis from Raw Data: A Clustering Approach. 10th IFAC Symposium on Fault Detection, Supervision and Safety for Technical Processes, SAFEPROCESS 2018, Aug 2018, Varsaw, Poland. 8p. ⟨hal-01817529⟩

[6]      Nithish SALLUSTRE, Digital Twin Calibration Using An Industrial Vision System, master’s thesis report, Université Lyon 1, August 2024