From Tactile Estimation to Control: Towards Vision/Tactile Shape Manipulation of Flexible Objects under Uncertainty

Complete offer: https://seafile.lirmm.fr/f/bf84cc46ea394d658375/

Manipulating deformable objects requires reasoning about shape, contact forces, local compliance, friction, and uncertainty. Vision-based shape servoing can reconstruct and control the visible deformation of soft objects, and online FEM can extend this estimate to occluded regions. Nevertheless, vision-based perception alone does not observe the local contact mechanics at the grasp or along the robot body. These hidden conditions often determine the task outcome: a similar visible shape may arise from different stiffness distributions, contact conditions, incipient slip, or force distributions. This thesis develops a tactile-informed, uncertainty-aware framework for dual-arm manipulation of deformable objects, in which a physical belief state provides the interface between model-based estimation, safe control, and generative decision-making. A central scientific question is which physical quantities are identifiable from tactile interactions and how this information can be used for control and policy learning. The thesis therefore targets task-relevant physical beliefs, such as local normal and tangential compliance, effective bending stiffness, damping, friction or slip state, grasp boundary conditions, and uncertainty over the predicted deformation. This belief state is designed to retain the information required for robust deformable manipulation while keeping the estimation problem well-posed and directly exploitable by both control algorithms and learned policies.

The first objective extends a SOFA-based deformable-object digital twin with tactile, force, and proprioceptive feedback. SOFA serves primarily as a real-time forward simulator and structured prior: predicted deformations, forces, and contact states are compared against RGB-D, wrist force/torque, joint torques, and tactile signals. The thesis investigates a hierarchy of estimation methods, including reduced-order sensitivity models, local interaction matrices, derivative-free updates for low-dimensional parameters, learned residual models, and, when appropriate, implicit differentiation through selected reduced solver components. The output is a calibrated physical belief state over object shape, contact state, local compliance, slip risk, and prediction uncertainty.

The second objective characterizes tactile identifiability and active probing. The thesis studies how active tactile exploratory motions, contact location, grasp configuration, object geometry, and sensor modality affect the observability of compliance, and friction, distinguishing beliefs recoverable from passive contact from those requiring active excitation. Because tactile measurements are local, informative estimation requires purposeful motion: the robot will actively select small, safe probing actions that reduce belief uncertainty while respecting force and stress limits, turning the grasp into an estimation instrument.

The third objective uses the resulting belief state and its calibrated uncertainty for tactile/force- aware control. A model-predictive or hybrid force-position controller regulates both deformation and contact forces while accounting for uncertainty: it favors information-gathering actions when the belief is highly uncertain, enforces stress or force constraints with margins that widen as observability degrades, and uses tactile feedback to detect contact state changes (slip or unexpected compliance) and trigger re-estimation. This provides an interpretable and safe control layer for contact-state estimation and manipulation under object deformation.

In collaboration with ClearLab/NUS, the aforementioned aspects will be investigated as a conditioning representation for uncertainty-aware generative policies, such as diffusion policies, for longer-horizon routing, fixture insertion, and collaborative tasks. The policy conditions its action generation on the estimated shape, contact state, compliance, slip margins, and uncertainty provided by the tactile-physical estimation layer. Learning is therefore guided by physical predictions and tactile evidence, while the policy contributes long-horizon decision-making and data-efficient action generation under uncertainty. This provides a natural division of work with ClearLab/NUS. The France-based work will focus on tactile/FEM belief estimation and robust control, while the NUS-side work will focus on belief-conditioned generative policies for deformable-object manipulation.

The expected outcome is a framework for manipulation of deformable objects, combining physical estimation, active probing, robust force/tactile control, and belief-conditioned generative policies. Experiments will progress from controlled deformable beams and foam objects to deformable linear objects such as cables, hoses, and flexible pipes, using the LIRMM (Montpellier, France) dual-arm platform equipped with force/torque sensing, accurate joint torque measurements, and capacitive tactile/proximity skin. Evaluation will measure shape error, force regulation, slip detection, stress-limit violations, uncertainty calibration, data efficiency, and generalization across objects with different stiffness and contact properties.

LIRMM

Required skills:

- Master's degree or Engineering diploma in robotics, mechatronics, or a closely related field.

- Strong background in robotics modeling, dynamics, and control.

- Proficiency in Python and C/C++ programming.

- Basic knowledge of continuum mechanics or FEM is a plus.

- Excellent written and oral communication skills in English.