LLM-Multimodal based model for diagnosis and prognosis in nephropathology

Renal pathology has a central role in the diagnosis and evaluation of the prognosis of many kidney diseases and in transplantation. Numerous studies have shown the decisive importance of quantifying the histological structures of biopsy sections. This sub-discipline of conventional histology is called morphometry. These structures can be very numerous and calculating the area, perimeter and/or radius they represent requires cutting out each structure, which is impossible to achieve in a reasonable time. In addition, the relatively high inter-pathologist variability makes this method less interesting. Thus, these markers are rarely used in practice. Artificial Intelligence (AI)-based automations have been developed to automate the recognition of normal and pathological renal histological structures, in particular through the collaboration of our Skinet team. However, these studies are unimodal, in the sense that they only use biopsy images. The latest research in Artificial Intelligence (AI) and in particular in deep learning has led to the emergence of Transformers-type networks whose performance has revolutionized the field. These networks are the basis of next-generation image processing models (such as VisionTransformer), but also of natural language processing (Large Language Model - LLM, extended language models). These recent advances in AI make it possible to envisage multimodal studies by combining biopsy images and biological and clinical textual data. Recent research, from a more general point of view, as well as for nephrology in particular, have shown the interest, but also the challenges posed by such studies. In the context of the anatomical renal pathology mentioned above, the objective of the thesis is to combine these two types of models by first using the capabilities of LLMs to process biological and clinical data and produce representations that will serve as a context for the processing of biopsy images. Indeed, the provision of a context could make it possible to refine the predictions in an unprecedented way and get even closer to the approach of a specialized pathologist. The ultimate goal will be to move forward the model with a view to generalizing the methodology and use it "live" on microscopes using cameras.

Position and Working Environment

The PhD studentship is a three years position starting in October/ November 2026. It does not include teaching obligation, but it is possible to engage if desired. The PhD candidate will work at Université Bourgogne Europe, in Dijon city. He/She will integrate the Laboratoire d’Etude de l’Apprentissage et du Développement (LEAD CNRS UMR 5022), Bât. I3M, 64B, rue de Sully, 21000 Dijon).

General information:
– Supervisor: Fan YANG-SONG and Patrick Bard
– Collaboration as part of the PhD thesis: CHU Dijon

– Start date: October / November 2026

Required Profile:

Master degree in computer science or applied mathematics, Engineering school. Background and experience in machine learning. Good technical skills in programming.