EN FR
EN FR

2024Activity reportProject-TeamTANGRAM

RNSR: 202023684L
  • Research center Inria Centre at Université de Lorraine
  • In partnership with:Université de Lorraine, CNRS
  • Team name: Visual Registration with Physically Coherent Models
  • In collaboration with:Laboratoire lorrain de recherche en informatique et ses applications (LORIA)
  • Domain:Perception, Cognition and Interaction
  • Theme:Vision, perception and multimedia interpretation

Keywords

Computer Science and Digital Science

  • A5.3. Image processing and analysis
  • A5.4. Computer vision
  • A5.4.1. Object recognition
  • A5.4.5. Object tracking and motion analysis
  • A5.4.6. Object localization
  • A5.6. Virtual reality, augmented reality
  • A5.10.2. Perception

Other Research Topics and Application Domains

  • B2.6. Biological and medical imaging
  • B5.9. Industrial maintenance
  • B9.5.3. Physics

1 Team members, visitors, external collaborators

Research Scientists

  • Marie-Odile Berger [Team leader, INRIA, Senior Researcher]
  • Erwan Kerrien [INRIA, Researcher]

Faculty Members

  • Vincent Gaudilliere [UL, Associate Professor, from Sep 2024]
  • Fabien Pierre [UL, Associate Professor]
  • Gilles Simon [UL, Associate Professor]
  • Frédéric Sur [UL, Professor]
  • Pierre-Frédéric Villard [UL, Associate Professor]

PhD Students

  • Youssef Assis [UL, until Apr 2024]
  • Nathan Boulangeot [UL]
  • Abdelkarim Elassam [Inria, until Aug 2024]
  • Radhouane Jilani [INRIA]
  • Vaishnavi Kanagasabapathi [UNIV BOURGOGNE, from Oct 2024]
  • Nariman Khaledian [Inria, until Aug 2024]
  • Hugo Leblond [UL]
  • Liang Liao [CHRU NANCY]
  • Nicolas Maignan [UL]
  • Insaf Mellakh [UL, from Oct 2024]

Interns and Apprentices

  • Pauline Bonnet [INRIA, Intern, from Apr 2024 until Aug 2024]
  • Charly Feltre [UL, Intern, from Apr 2024 until Sep 2024]

Administrative Assistants

  • Nathalie Bethus [CNRS, from Mar 2024]
  • Véronique Constant [INRIA]

Visiting Scientists

  • Gao Hao [Glasgow University, from Sep 2024 until Oct 2024]
  • François Rameau [UNIV SUNY, from Jun 2024 until Jul 2024]

External Collaborators

  • Cédric Demonceaux [UNIV BOURGOGNE]
  • Renato Martins [UNIV BOURGOGNE]

2 Overall objectives

Visual registration is a research topic with a rich history in computer vision. Though a plethora of methods have been developed and can be used for general situations, there are still many open problems which originate in the nature of the scene (poorly textured or specular environments), in the type of motion undergone by the object (tiny motions which hardly emerge from the noise floor, or in contrast, highly deformable objects) and in dissimilarities which may occur in the scene between the time the modeling stage occurs and the application time.

Registration is in practice tightly linked to the choice of the model which represents the scene and the desirable physical properties of the objects. Handling complex —possibly dynamic— scenes thus requires a tradeoff between physical realism of the model, convergence issues and robustness of the registration or tracking tasks.

Recent years have seen a surge in research at the intersection of image and deep learning which has impacted many topics of computer vision. Besides our continued exploration of modeling and registration with traditional approaches derived from signal processing, geometry, and robust estimation, one of the team's aims is to integrate machine learning methods, either as end-to-end methods or as components, into these 2D or 3D geometric tasks.

Targeted trans-disciplinary applications are mixed and augmented reality, computational photomechanics and minimally invasive medical interventions.

3 Research program

3.1 Localization and geometric reasoning with high level features

Our goal is to push forward vision-based scene understanding and localization through the joint use of learning-based methods with geometrical reasoning. Our hypothesis is that the use of intermediate representations instead or in addition to the classical point feature will lead to increased capacity in terms of scale and robustness to changing conditions. These intermediate representations can be concrete objects which are recognized and used directly in the global pose computation, in the continuity of our works on ellipsoid modeling of objects, or conceptual objects such as vanishing points (VP) or horizon lines that are of specific interest both for localization and modeling of urban or industrial scenes.

A first goal is to improve our method for localization from sets of ellipse/ellipsoid correspondences 3, 9. Besides the need to have more accurate prediction of ellipses, another objective is to elaborate robust strategies and associated numerical schemes for refining the initial pose from a set of objects. This requires us to develop appropriate metrics for characterizing good reprojection of 3D objects onto 2D ones and study their impact on minimization issues in localization. Another goal is to define strategies to integrate into the localization procedure various features such as points, objects and VPs, which each bring information at different levels. We especially want to investigate how predictive uncertainty and explainability mechanisms can be used to select and weight these various features in the estimation process.

3.2 Building dedicated models

In this line of research, our goal is to build physically coherent models with a good accuracy vs. efficiency compromise despite the interactive time constraint set in some targeted applications. Though general purpose solutions exist for building models, such techniques are still greatly challenged in more complex cases when specific constraints on the shape or its deformation must be met. This is especially the case in medical imaging of thin deformable organs, such as the diaphragm, the mitral valve or blood vessels, but also for classical scene modeling where constraints, such as ellipsoidal abstraction of objects, must be introduced. The use of mechanical models has become increasingly important in the team's activities in medical imaging, especially for handling organs with large deformations. We want to push forward the development of such models with image-guided procedures or predictive simulation in view.

Facing difficulties of meshing complex geometries, especially thin ones, we want to promote mesh free methods such as implicit models. In the continuity of past works 4, automatic adaptation of node locations and sizes to the image will be investigated to improve compactness, and computational efficiency of implicit models. As the fidelity of a mechanical model is often impaired by approximations required to solve its dynamical system equations at interactive frame rates, a second objective is to take advantage of our implicit models to improve contact and deformation resolution.

Another topic of interest is the investigation of shape-aware methods either for shape segmentation or shape recognition, in order to be able to enforce global shape constraints or geometric shape priors on the output of CNNs.

3.3 Estimation and inverse problems

Most aforementioned tasks lead to image-based inverse, possibly ill-posed, problems. While some of them can be solved with well-established estimation techniques, others necessitate the design of new strategies. In this perspective, we consider in this research axis several fundamental aspects of estimation, common to our problems, such as sampling methods, traditional optimization methods, or end-to-end learning methods for pose estimation.

3.3.1 Optimization, variational calculus and numerical schemes

We are interested in non-convex optimization problems, especially those raised by variational calculus. While the convergence of numerical schemes is well established for convex problems, this is not always the case for non-convex functionals. Our aim is to continue the work already carried out in the biconvex framework 7, and extend it to primal-dual algorithms. We especially want to address energy minimization problems where the energy is convex with respect to each variable, but non-convex with respect to the pair of variables.

Another research topic is to investigate new neural architectures adapted to non-Euclidean data, and also to plug variational methods into deep learning approaches to regularize the results. The obtained theoretical results will be applied to image colorization, with the idea to reduce artefacts caused both by a lack of regularization and by the non-Euclidean structure of color information as perceived by the human visual system.

3.3.2 Machine learning for physical problems

We aim at continuing our efforts towards supervised and unsupervised learning for estimation problems. Concerning supervised learning, we intend to investigate further the opportunities offered by neural network estimation of displacement and strain fields in experimental mechanics that we have recently introduced with colleagues in mechanics and signal processing 1. Besides, we also aim at developing unsupervised learning in problems where a quantity has to be estimated over a spatio-temporal domain, which is a recent trend in several application domains. Neural networks are indeed universal approximators whose derivative can be exactly computed with the backpropagation algorithm, which is supposed to make them robust to acquisition noise.

4 Application domains

Applications on which our program is expected to have an impact are mixed reality, computational photomechanics and minimally invasive medical interventions. These fields correspond to areas where we have established trans-disciplinary collaborations with academic or industrial experts of the applicative fields. Common to these applications are the need for finely characterizing the acquisition context of vision-based applications and the need for accurate registration procedures. Another common point is the availability of a limited amount of data for characterizing the variability of the observed phenomena.

Mixed reality

Being able to perform reliable and accurate registration under large viewpoint variations, seasonal or lighting changes opens the way towards challenging mixed reality applications. Urban AR and industrial maintenance in large and cluttered environments are examples of application fields that would successfully capitalize on more robust localization solutions. Improved robustness of camera localization is especially expected for poorly textured, specular environments and in the presence of repeated patterns that are common in industrial contexts

Photomechanics

Photomechanics is the field of experimental mechanics which is dedicated to mechanical measurement from images. In particular, we are interested in contactless image-based methods for extensometry, that is the estimation of displacement and strain fields on the surface of materials subjected to different types of mechanical loads. Full-field extensometry is a challenging task since strains often have tiny values and result in gray level changes at the limit of the sensor noise floor. The economic stakes are high and concern for example the automotive and aeronautics industries, or civil engineering. In order for these methods to be adopted by industry, it is, however, necessary to quantify their metrological performance, which is limited by the registration process or by the image acquisition chain, and especially by sensor noise. This topic is the subject of a long-term trans-disciplinary collaboration with Institut Pascal (Clermont-Ferrand Université).

Minimally invasive medical interventions

The trend towards the design and performance of minimally invasive procedures will increase in the near future. But the benefit for the patient is at the expense of the surgeon who can only sense the surgical scene through intra-operative imaging. Commercial solutions now exist to teach this increasingly difficult surgical gesture with interactive simulation technologies. However, challenges remain to fill the gap between the learning environment, where qualitative correctness of the setup is sufficient, and the surgical theater, where accuracy and predictability are required. In this context, we aim at addressing the key problem of modeling the geometry and dynamics of deformable organs and surgical devices, in order to make progress towards a faithful 3D rendition of the surgical scene. To circumscribe practical and experimental difficulties, three specific applications will be addressed with our clinical partners: intra-operative guidance in interventional neuroradiology, augmented reality for laparoscopic liver surgery, and simulation of the mitral valve behaviour.

5 New software, platforms, open data

5.1 New software

5.1.1 DeepAneDet

  • Name:
    Brain aneurysm detection using deep learning
  • Keywords:
    Deep learning, Anomaly detection, Medical imaging, Brain aneurysm, Brain MRI, Object detection
  • Functional Description:
    DeepAneDet is a deep convolution network for the detection of intracranial aneurysms from 3D TOF-MRI images. It is a YOLOv3-inspired anchor-free detection model in 3D, coupled with an original strategy for small patch generation that combines data augmentation and data synthesis. Each aneurysm detection output is given by a sphere (3D center, and radius) associated with a confidence score.
  • URL:
  • Contact:
    Erwan Kerrien
  • Partner:
    Loria

6 New results

6.1 Visual localization

6.1.1 Gaussian splatting and Visual localization

Participants: Marie-Odile Berger, Cédric Demonceaux, Hugo Leblond, Renato Martins, Gilles Simon.

Although originally designed for realistic rendering of novel views, the Gaussian Splatting (GS) method introduces an anisotropic representation using 3D Gaussians to model radiance fields, making it particularly promising for visual localization and SLAM applications. Recent work has explored methods for loop closure detection through Gaussian registration, improving the accuracy and consistency of mapped environments. However, challenges remain in achieving simultaneous large-scale mapping and loop closure detection, particularly in dynamic and large-scale environments. Within the context of H. Leblond's PhD thesis, we are interested in solving the loop closure problem by generating bird-eye views (BEV) of the optimized Gaussian models. Unlike 3D registration methods, which can suffer from inefficiencies as the number of Gaussians increases, our method relies on 2D renders, which allow for constant-time performance. In addition, subject only to knowledge of the gravity vector—which can be obtained using an IMU sensor or by detecting the zenith vanishing point in images—it reduces the degrees of freedom of the pose computation from six to three, which allows us to make registration more robust to matching errors. Experiments carried out on two public datasets show that our method is more robust and more accurate than LoopClosure, a previous approach based on BEV maps of point densities, and faster, with comparable performance, than GaussReg, a recent method for 3D registration of GS models.

6.1.2 Vanishing point computation and applications

Participants: Marie-Odile Berger, Cedric Demonceaux, Abdelkarim Elassam, Vaishnavi Kanagasabapathi, Renato Martins, Gilles Simon.

For several years now, the Tangram team has been working on the automatic detection of vanishing points in an image, a preliminary step in the rectification of planar structures present in a picture, which is also useful for determining the intrinsic and orientation parameters of a camera. Traditionally considered as a problem of grouping line segments by meeting points, this task has been reconsidered in recent years in the light of deep learning techniques. Within the context of A. Elassam's PhD thesis 23, we proposed last year CollabVP, a deep learning based method for extracting structural vanishing points. This year we investigated the use of structural vanishing points for wide baseline registration. We introduced an original method for VP matching, a previously unexplored problem. Building on our previous work, CollabVP, which provides both VPs and masks of associated vertical structures, matching VPs becomes a task of matching these masks. The masks are orthorectified using their corresponding horizontal and zenith VPs and matched using ConvNet descriptors.

We subsequently introduced VP-based image matching. Wide-baseline registration poses significant challenges due to the substantial projective distortions caused by distant camera positions. To address this, our approach leverages metric rectification of masks associated with the VPs, facilitating more effective keypoint matching. Although rectification may be imperfect in cases of inaccurate VP detection, it still enhances the similarity of keypoints between the images, enabling additional keypoint correspondence hypotheses under conditions of pronounced projective distortion. Our framework integrates seamlessly with any existing keypoint detection and matching algorithms, demonstrating enhanced camera pose estimations, particularly for tasks involving large-baseline matching

V. Kanagabapathi's PhD thesis started in october 2024 in the continuity of these works. Her thesis addresses the problem of visual feature learning on videos for scene understanding. Our goal is to design strategies that are capable of leveraging temporal consistency and physical constraints when a sequence of images is available. We are interested notably in designing strategies for vanishing point estimation from a sequence of images.

6.1.3 Multispectral information fusion

Participant: Vincent Gaudillière.

Visual localization may need to be performed under adversarial illumination conditions. To address such challenges, the combination of thermal and visible images has demonstrated its advantages. However, existing fusion methods rely on the critical assumption that the RGB-Thermal (RGB-T) image pairs are fully overlapping. These assumptions often do not hold in real-world applications, where only partial overlap between images can occur due to sensors configuration. Moreover, sensor failure can cause loss of information in one modality. In 14, we proposed a novel module called the Hybrid Attention (HA) mechanism as our main contribution to mitigate performance degradation caused by partial overlap and sensor failure, i.e. when at least part of the scene is acquired by only one sensor. We proposed an improved RGB-T fusion algorithm, robust against partial overlap and sensor failure encountered during inference in real-world applications. We also leveraged a mobile-friendly backbone to cope with resource constraints in embedded systems. We conducted experiments on the pedestrian detection problem, by simulating various partial overlap and sensor failure scenarios to evaluate the performance of our proposed method. The results demonstrate that our approach outperforms state-of-the-art methods, showcasing its superiority in handling real-world challenges.

6.2 Handling non rigid deformation

6.2.1 Individual mitral valve modeling

Participants: Marie-Odile Berger, Nariman Khaledian, Pierre-Frédéric Villard.

We continued the work started last year on simulating mitral valve closure by incorporating the interaction between the leaflets and blood, while accounting for an anisotropic constitutive law and patient-specific data. This year, our focus shifted to scaling up the approach by testing it on multiple patient datasets 13, 24. To achieve this, we have adapted the biomechanical model to mitigate numerical issues arising from less smooth geometries. It also includes the influence of modeling the left ventricle with a cylinder fitted to the valve dimension or a large one as well as the influence of the fiber orientation in the anisotropic modeling of the leaflet.

6.2.2 Image-based biomechanical simulation of the diaphragm during mechanical ventilation

Participant: Pierre-Frédéric Villard.

The ultimate goal of this project is to perform high-fidelity, real-time simulations of the respiratory function of a critical care patient. The focus is on the respiratory muscles, of which the diaphragm is the main muscle. The first step is to create a realistic tissue model that allows us to model the passive deformation of the muscle.

Following last year's work, we have used a test problem with manufactured solution to evaluate the error and convergence behaviour of an elastic solver with the method of the radial basis function using the principle of unity. The solver shows convergence for the manufactured solution, which was not the case last year. The next step is to make it work from non manufactured problems. Next year, we should apply it in the context of the diaphragm with boundary conditions coming from non-rigid deformation information extracted from CT scans at different times steps.

6.3 Interventional radiology

6.3.1 Detection of brain aneurysms using deep learning

Participants: Youssef Assis, Erwan Kerrien, Liang Liao, Fabien Pierre.

Youssef Assis defended his PhD thesis where he proposed an anchor-free deep neural network for the detection of intracranial aneurysms from Magnetic Resonance Angiography (MRA) images 22. Many concurrent works consider a segmentation approach, which demands a tedious voxel by voxel annotation of the inherently small pathologies. Our model relies on a weak representation of aneurysms as spheres. First, this makes annotation easier and faster. But also, it allows for an object detection approach, where a confidence score can be attached to each aneurysm detection. The model was extensively evaluated through an ablation study, and compared favorably with state of the art methods 10. The model, code and annotation data were distributed as the DeepAneDet software 5.1.1.

We are also involved in the co-supervision of Liang Liao's PhD thesis, a neurointerventionalist, with Prof. René Anxionnat from CHRU Nancy and Université de Lorraine. The goal of this PhD is to continue improving the performances of our model above, and develop methods to clinically evaluate such aneurysm detection methods. A team of 5 annotators with variying experience in neuroimaging was put together to assess the impact of automatic detection models on their raw performance as experts. Double blind annotation of the full database of 270 patients was performed by each annotator, based a prior automatic detection, the resulting 2700 annotations were processed. Our findings are that detection performance are obtained when human observers merely remove obvious false positives after automatic detection, especially for medical doctors with little to average experience.

6.3.2 Predictive simulation of catheter navigation

Participants: Radhouane Jilani, Erwan Kerrien, Pierre-Frédéric Villard.

Our main contribution to the PreSPIN ANR project consists in achieving smooth, interactive and predictive simulation of a catheter navigating in the brain vasculature.

In the course of his PhD Radhouane Jilani investigates the Cosserat model for flexible rods, in order to model catheters. Such models can be solved using continuous methods such as the shooting method or a collocation method. Classical computer graphics methods can be quite straightforwardly used to incorporate frictional contact in discretized rod models. The problem is much more complex to tackle in the continuous case. A solution based on the shooting method has been proposed only on the static case. A collocation method has been described in the dynamic case, but it relies on a linear approximation for contact management. In our work presented at IROS 18, we describe an exact solution for the dynamic case under frictional contact. Our method relies on an implicit representation of the contact surface, leveraging our previous work on blood vessel surface modeling. Furthermore, using continuous representations for both the catheter and the vessel surface, was proven to dramatically speed up simulation time.

Another line of methods to solve the Cosserat model are reduction-based approaches. These can be solved using strain parameterization. We have proposed a new angular strain parameterization for the static case, that dramatically reduces the computation time associated with this type of approach, without compromising accuracy 21.

Furthermore, the dynamic equations involved by the continuous methods were recently shown to be fundamentally unstable for soft rods like catheters. We are currently investigating strain parameterization methods to solve the Cosserat model under frictional contact in the static and quasi-static cases.

This year, we also published a study validating the Cosserat model through a physical experiment 25. The simulation was performed using a discrete representation, in contrast to our previous work.

6.4 Image and signal processing

6.4.1 Computational photomechanics

Participant: Frédéric Sur.

The work of this year concerns three aspects of displacement and strain field estimation over a material submitted to a mechanical load. In order to process the surface of the material under deformation, a periodic pattern is marked on it.

  • The checkerboard constitutes the best pattern for full-field strain measurement because it maximizes image gradient and metrological performance in turn. In the experimental mechanics community, employing this pattern is currently strongly limited because depositing it on the surface of specimens raises practical difficulties. A recent study shows that it is technically possible by using a laser engraver. In  17, the authors aim to push this solution forward by printing the same pattern on a thin polymeric film, and then gluing the resulting laser-engraved film on the specimen surface. The underlying idea is to separate the manufacturing process of this optical strain gauge on the one hand, and its use on the other hand, to help spread this strain measuring tool in the experimental mechanics community. The polymeric film employed here is the same as that used in the manufacturing process of classic electric gauges, so one can rely on the know-how of classic strain gauge bonding to glue this optical strain gage on the specimen surface.
  • Sampling Moiré is a spatial fringe pattern analysis method which can be used to retrieve displacement fields on the surface of deformed specimens marked by a periodic pattern. In 16, we prove that, under mild assumptions, Sampling Moiré (SM) is equivalent to another full-field method, namely the Localized Spectrum Analysis (LSA). The prospects opened up by this result concern two metrological indicators, namely the measurement bias (the retrieved displacement being affected by a systematic error) and the measurement resolution (limited by the propagation of sensor noise to displacement maps). Previous studies on LSA yield predictive formulas for measurement bias and displacement resolution. These formulas are adapted to the case of SM.
  • Finally, 11 presents an open-source LSA software written in Python and illustrates its use by two application cases in experimental mechanics.

6.4.2 Variational methods for image processing

Participants: Nicolas Maignan, Fabien Pierre, Frédéric Sur.

The colorization of videos is mostly based on color transfer between frames. The analysis and the comparisons of the colorization results are based on various metrics but there is no consensus about one of them to be able to judge the quality of the final results. In the context of N. Maignan PhD thesis, a survey about these metrics and in particular their ability to measure temporal consistency is being written.

6.5 Application of machine learning

6.5.1 Accurate Potential Energy Surfaces at Low Computational Cost by Machine Learning and DFT-Based Data

Participants: Nathan Boulangeot, Frédéric Sur.

We have developed a collaboration in machine learning for material sciences with Émilie Gaudry at Institut Jean Lamour (Nancy). Intermetallic compounds are promising materials in numerous fields, especially those involving surface interactions, such as catalysis. A key factor to investigate their surface properties lies in adsorption energy maps. However, exploring the adsorption energy landscapes of intermetallic compounds can be cumbersome, usually requiring huge computational resources. In this work, we propose an efficient method to predict adsorption energies, based on a Machine Learning (ML) scheme fed by a few Density Functional Theory (DFT) estimates performed on a limited number of carefully selected sites. We have studied its application on the Al13Co4(100) quasicrystalline approximant surface for several atomic adsorbates (H, O, and Pb). On this specific example, our approach is shown to outperform both simple interpolation strategies and the recent ML force field MACE, especially when the number of sites selected for training is small, i.e., below 36 sites 12.

6.5.2 Inversion of downhole resistivity properties through infrared spectroscopy and whole‐rock geochemistry using machine‐learning

Participants: Mehdi Serdoun, Frédéric Sur.

We are engaged in the co-supervision (together with Julien Mercadier, GéoRessources, Université de Lorraine) of the PhD thesis of Mehdi Serdoun, which is part of the GeoMin3D project funded by ANR and Orano Mining, since January 2022. The goal is to develop statistical learning models to analyze the large amount of data of diverse nature provided during the exploratory drillings in Athabasca basin, the largest known source of uranium 19. The ultimate goal is to develop new analysis tools to accelerate exploration and reduce its cost, in cooperation with the industrial actors.

Publication 15 concerns electrical properties of rocks, which are widely used in the geophysical exploration of natural resources, such as minerals, hydrocarbons and groundwater. In mining exploration, the primary goal is to map electrically anomalous geological features associated with different mineralization styles, such as clay alteration haloes, metal oxides and sulphides, weathered crystalline rocks or fractured zones. As such, the reconciliation of geophysical data with geological information (geochemistry, mineralogy, texture and lithology) is a critical step and can be performed based on petrophysical properties collected either on core samples or as downhole measurements. Based on data from 189 diamond drill cores collected for uranium exploration in the Athabasca Basin (Saskatchewan, Canada),  15 presents a case study of reconciliation of downhole resistivity probing with core sample geochemistry and short‐wave infrared spectroscopy (350–2500 nm).

6.5.3 Neural network architectures dedicated to crystalline orientations and Electron BackScattered Diffraction (EBSD)

Participants: Pengru Zhao, Frédéric Sur.

We are engaged in the co-supervision of the PhD thesis of Pengru Zhao with Lionel Germain (LEM3, Université de Lorraine). The goal is to develop machine learning models to process crystal orientation maps obtained by Electron Backscattered Diffraction (EBSD). The beginning of Pengru's thesis concerns the design of a neural network architecture to detect grain boundary on the surface of crystalline structures, a simpler problem than EBSD data processing 20.

7 Partnerships and cooperations

7.1 International initiatives

7.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

CURATIVE

 

Participants: Marie-Odile Berger, Nariman Khaledian, Pierre-Fredéric Villard.

  • Title:
    CompUteR-based simulAtion Tool for mItral Valve rEpair
  • Duration:
    2020 -> 2024
  • Coordinator:
    Robert Howe (howe@seas.harvard.edu)
  • Partners:
    • Harvard University Cambridge (États-Unis)
  • Inria contact:
    Pierre-Fredéric Villard
  • Summary:
    The mitral valve of the heart ensures one-way flow of oxygenated blood from the left atrium to the left ventricle. However, many pathologies damage the valve anatomy producing undesired backflow, or regurgitation, decreasing cardiac efficiency and potentially leading to heart failure if left untreated. Such cases could be treated by surgical repair of the valve. However, it is technically difficult and outcomes are highly dependent upon the experience of the surgeon. One of the main difficulties of valve repair is that valve tissues must be surgically altered during open heart surgery such that the valve opens and closes effectively after the heart is closed and blood flow is restored. In order to do this successfully, the surgeon must essentially mentally predict the displacement and deformation of anatomically and biomechanically complex valve leaflets and supporting structures. Even if patient-based mitral valve models have been recently used for scientific understanding of its complex physiology, the patient geometry is manually segmented on medical images. This task is long and cumbersome except if the valve has been artificially isolated in-vitro. There is a lack in the literature about the variety of metrics in both anatomy and biomechanics of the valve. In order to study mitral valve behavior or to prepare models for planning, it is necessary to have methods to extract the valve components i) on real clinical data ii) with minor user input and iii) that is mechanically valid.

7.2 International research visitors

7.2.1 Visits of international scientists

Other international visits to the team
Hao Gao
  • Status
    Senior Lecturer
  • Institution of origin:
    University of Glasgow
  • Country:
    United Kingdom
  • Dates:
    20/09/2024 - 20/10/2024
  • Context of the visit:
    The research stay was in the context of our work on the mitral valve simulation. We worked on preparing the simulation framework to replicate a phantom valve deformation in a control environment and we start working on the parametrization of a valve for our first join publication.
  • Mobility program/type of mobility:
    Invited professor funded by the university of Lorraine
Peter Hammer
  • Status
    Senior researcher
  • Institution of origin:
    Harvard Medical School
  • Country:
    USA
  • Dates:
    20/09/2024 - 27/09/2024
  • Context of the visit:
    The research stay was in the context of our work on the mitral valve simulation. We work on preparing the experiments of phantom valve deformation in a control environment.
  • Mobility program/type of mobility:
    Visiting researcher funded by the associate team CURATIVE
François Rameau
  • Status
    Assistant Professor
  • Institution of origin:
    State University of New York (SUNY)
  • Country:
    Korea
  • Dates:
    15/06/2024 - 12/07/2024
  • Context of the visit:
    In the context of pose computation, the goal was to design new approches to decompose minimal computer vision problems into smaller sub-problems where consensus maximization techniques can be applied on a smaller subset of points.
  • Mobility program/type of mobility:
    Visiting researcher funded by INRIA

7.2.2 Visits to international teams

Research stays abroad
Pierre-Frédéric Villard
  • Visited institution:
    Harvard University
  • Country:
    USA
  • Dates:
    31/10/2024 - 02/12/2024
  • Context of the visit:
    The research stay was in the context of our work on the mitral valve simulation. We work on the experiments of phantom valve deformation in a control environment. We tested how to gather ultrasound and stereoscopy data that can be used to validate our simulation model. Finally we performed the experiment with different parameter conditions.
  • Mobility program/type of mobility:
    Visiting research stay funded by the associate team CURATIVE
Pierre-Frédéric Villard
  • Visited institution:
    Uppsala University
  • Country:
    Sweden
  • Dates:
    01/04/2024 - 31/04/2024
  • Context of the visit:
    The research stay was in the context of our work on the diaphragm simulation. We worked on testing a linear deformation problem with a manufactured solution to evaluate the errors and convergence behaviour of an elastic solver with the method of the radial basis function using the principle of unity.
  • Mobility program/type of mobility:
    Visiting research stay funded by the Swedish scientific council

7.3 National initiatives

PEPR ICCARE
  • Title
    : Cultural and Creative Industries: Action, Research, Experimentation
  • Coordinator:
    Gilles Simon
  • Participants:
    G. Simon, M.-O. Berger
  • Duration:
    2025-2030
  • Additionnal info/keywords:
    With a budget of €25 million for a six-year period, ICCARE's aim is to bring together research communities (human and social sciences/computer sciences) in a process of co-construction, co-realization and co-valorization, in order to help the ICC industry transform and adapt to digital, economic and social challenges. Gilles Simon is joint coordinator of the “Museum and Heritage” sector, along with Lise Renaud (SHS, univ. Avignon) and Thomas Sagory (Musée d'Archéologie Nationale).
ANR Arcé
  • Title
    : Colorisation automatique de vidéos
  • Coordinator:
    Fabien Pierre
  • Participants:
    F. Pierre, N. Maignan, F. Sur
  • Duration:
    2022-2026
  • Additionnal info/keywords:
    The Arcé project aims at proposing new methods for automatic, fast and perceptually satisfying video colorization. Image colorization methods based on deep learning based have encountered a great success in recent years. These techniques are fully automatic and very fast, but they have not been adopted by colorization industry. The reason is that they do not ensure the temporal coherence of the colorization, which is particularly disturbing for the viewer. The ultimate goal is the use of our work in audiovisual production studios.
ANR PRC PreSPIN
  • Title:
    Predictive Simulation for Planning Interventional Neuroradiology procedures
  • Partners:
    CReSTIC (Reims), Creatis (Lyon) and CIC-IT/CHRU Nancy
  • Coordinateur:
    Erwan Kerrien
  • Participants:
    Y. Assis, R. Jilani, E. Kerrien, P.-F Villard.
  • Duration:
    2020-2025
  • Additionnal info/keywords:
    This project is coordinated by E. Kerrien. It aims at improving the planning phase in the therapeutic management of cerebral ischemic strokes thanks to predictive simulation of both the therapeutic interventional gesture and post-interventional perfusion images. The consortium is set to address the challenges of geometrical and topological modeling of the full brain vasculature; physics-based simulation of interventional devices; simulation of MRI perfusion images; and clinical validation.

8 Dissemination

8.1 Promoting scientific activities

8.1.1 Scientific events: organisation

Member of the organizing committees
  • Erwan Kerrien is a member of the scientific committee of the Digital Science and Health community at Université de Lorraine (digital science labs, health and biology labs and University Hospital). This committee organized a half-day workshop dedicated to Digital sciences and Neuro.

8.1.2 Scientific events: selection

Member of the conference program committees
  • Frédéric Sur was a member of the program committee of VISAPP 2024.
Reviewer
  • Marie-Odile Berger was a reviewer for IROS (International Conference on Intelligent Robots and Systems), ICRA (International Conference on Robotics and Automation) and for the French conference RFIAP.
  • Vincent Gaudillière was a reviewer for ACCV 2024 (Asian Conference on Computer Vision) and CVPR 2025 (Conference on Computer Vision and Pattern Recognition).
  • Erwan Kerrien was a reviewer for MIDL (Medical Imaging with Deep Learning), MICCAI (Medical Image Computing and Computer Assisted Intervention), IROS, and ICRA.
  • Gilles Simon was a reviewer for IROS 2024 (Conference on Intelligent Robots and Systems).
  • Pierre-Frédéric Villard was a reviewer for the Eurographics Workshop on Visual Computing for Biology and Medicine 2024 and the International Conference on Computer Graphics, Visualization, Computer Vision And Image Processing 2024 and the national conference Intelligence Artificielle en Imagerie Biomédicale 2024.

8.1.3 Journal

Reviewer - reviewing activities
  • Marie-Odile Berger was a reviewer for Robotic and automation letter, IEEE Geoscience and Remote Sensing Letters and Pattern Recognition Letter.
  • Erwan Kerrien was a reviewer for Medical Image Analysis, IEEE Transactions on Medical Imaging, IEEE Transactions on Biomedical Imaging, and Computers in Biology and Medicine.
  • Fabien Pierre was a reviewer for IEEE Transactions on Image Processing.
  • Gilles Simon was a reviewer for IEEE Robotics and Automation Letters (RA-L) and Multimedia Tools and Applications.
  • Frédéric Sur was a reviewer for Pattern Recognition Letters, Mechanical Systems and Signal Processing, Optics and Lasers in Engineering, Experimental Mechanics.

8.1.4 Invited talks

  • Marie-Odile Berger gave an invited talk at the INRIA-DFKI IDESSAI European summer school on Artifical Intelligence in Sarrebruck on september 12th. Title "AI for computer vision: Using high level features for visual localization".
  • Pierre-Frederic Villard gave a seminar at the department of information technology of Uppsala University. Title: "Patient-based Simulation of Mitral Closing" on May 2024.
  • Pierre-Frédéric Villard did a presentation at Harvard University on November 21st. Title: "Catheter Modeling using the Cosserat Rod".
  • Pierre-Frédéric Villard did a presentation at the MIT on November 25th. Title: "Modeling the Mitral Valve: From Medical Image Analysis to Fluid-Structure Interaction".
  • Pierre-Frédéric Villard gave a talk at the GDR Mecabio Santé on December 5th in Metz. Title: "Modélisation du comportement de la valve mitrale".

8.1.5 Scientific expertise

  • Marie-Odile Berger was a member of the recruitment committees for a professeur and for a junior professor chair in robotics at Université de Lorraine. She was a member of the committee that awards the annual AFRIF thesis prize.
  • Pierre-Frédéric Villard was a member of the recruitment committees for an associate professor at Université de Strasbourg and for an associate professor at the engineering school INSA.

8.1.6 Research administration

  • Marie-Odile Berger is the head of the INRIA COMIPERS PhD and postdoctoral recruitment committee.
  • Pierre-Frédéric Villard is an elected member of the Scientific Council of the Université de Lorraine
  • Gilles Simon is coordinator of the "Museum and Heritage" sector of the Priority Research Programmes and Equipments (PEPR) ICCARE.
  • Erwan Kerrien is the head of the Digital Health transversal axis of the Loria lab.
  • Erwan Kerrien is a member of the steering committee of the France Live Imaging (FLI) Hub Grand Est.

8.2 Teaching - Supervision - Juries

8.2.1 Teaching

The professors and assistant professors of the TANGRAM team actively teach at Université de Lorraine with an annual number of around 200 teaching hours in computer sciences, some of them being accomplished in the field of image processing. INRIA researchers have punctual teaching activities in computer vision and shape recognition mainly in the computer science Master of Nancy and in several Engineering Schools near of Lorraine (ENSMN Nancy, SUPELEC Metz, ENSG, TELECOM Nancy). Our goal is to attract Master students with good skills in applied mathematics towards the field of computer vision.

The list of courses given by staff members is detailed below:

  • M.-O. Berger
    • Master : Shape recognition, 24 h, Université de Lorraine.
    • Master : Introduction to image processing, 12 h, ENSMN Nancy.
  • E. Kerrien
    • Master : Introduction to image processing, 15 h, ENSMN Nancy.
    • Master : Shape recognition, 12 h, Université de Lorraine.
    • Licence : Initiation to software developement, 70h, IUT St Dié-des-Vosges.
  • Fabien Pierre
    • Master: Introduction à l’apprentissage automatique, 14h, Mines Nancy.
    • Master: Vision artificielle et traitement des images, 12h, Polytech Nancy.
    • Licence: Introduction au traitement d’image, 30h, IUT Saint-Dié des Vosges.
    • Licence: Algorithmique et programmation, 87h, IUT Saint-Dié des Vosges
    • Licence: Culture scientifique et traitement de l’information, 69h, IUT Saint-Dié des Vosges
    • Licence: Programmation objet et évènementielle, 35h, IUT Saint-Dié des Vosges
    • Licence: Initiation à l’intelligence artificielle, 18h, IUT Saint-Dié des Vosges
  • G. Simon
    • Master: Augmented reality, 9 h, Télécom-Nancy.
    • Master: Augmented reality, 24h, M2 Informatique FST
    • Master: Visual data modeling, 12h, M1 Informatique FST
    • Master: Computer Vision, 12h, M1 Informatique FST
    • Licence pro: 3D modeling and augmented reality, 50h FST - CESS d’Epinal
    • Licence: Programming methodology, L1 informatique, 48h FST
  • F. Sur
    • Master: Introduction to machine learning, 40 h, Université de Lorraine (ENSMN Nancy).
    • Licence: Javascript programming, 100h, IUT Charlemagne
  • P.-F Villard
    • Master : Augmented and Virtual Reality, 16h, M2 Cognitive Sciences and Applications, Institut des Sciences du Digital, Université de Lorraine
    • Licence: Computer Graphics with webGL, 30h, IUT Saint-Dié des Vosges.
    • Licence: Virtual and Augmented Reality in Industrial Maintenance, 2h, Faculty of Science and Technology, Université de Lorraine
    • Licence: Web programming, 20h, IUT Saint-Dié des Vosges.
    • Licence: Graphical user interface programming, 30h, IUT Saint-Dié des Vosges.
    • Licence: Security and life privacy with internet, 2h, IUT Saint-Dié des Vosges.
    • Licence: Parallel programming, 18h, IUT Saint-Dié des Vosges.
    • Licence: Initiation to machine learning, 24h, IUT Saint-Dié des Vosges.
    • Licence: Initiation to cryptography, 12h, IUT Saint-Dié des Vosges.
  • V. Gaudillière
    • Licence: Synthesis project, 20h, L1 informatique, FST.
    • Licence: Basics of object-oriented programming, 42h, L2 informatique, FST.
    • Licence: Algorithms and complexity, 18h, L2 informatique, FST.

8.2.2 Supervision

  • PhD completed: Youssef Assis, Intracranial aneurysm detection using deep learning, defended in March 2024, Erwan Kerrien et René Anxionnat (CHRU Nancy).
  • PhD completed: Abdelkarim Elassam, Learning-based vanishing point detection and its application to large-baseline image registration, defended in July 2024, Gilles Simon et Marie-Odile Berger.
  • PhD completed: Nariman Khaledian, Towards Patient-Based Fluid-Structure Interaction Simulation of the Mitral Valve, defended in June 2024, Marie-Odile Berger and Pierre-Frédéric Villard.
  • PhD in progress: Radhouane Jilani, Predictive simulation for interventional neuroradiology, October 2021, Erwan Kerrien, Pierre-Frédéric Villard.
  • PhD in progress: Liang Liao, Detection of cerebral aneurysms from MRI images using deep learning: deep neural network creation and its clinical evaluation, November 2021, René Anxionnat (CHRU Nancy) and Erwan Kerrien.
  • PhD in progress: Nathan Boulangeot, Coupling machine learning and quantum chemistry methods to predict surface properties of intermetallic catalysts, October 2021, Émilie Gaudry (Institut Jean-Lamour), Frédéric Sur.
  • PhD in progress: Mehdi Serdoun, Multivariate analysis of mineralogical, geochemical and physical signatures, January 2022, Julien Mercadier (GéoRessources), Frédéric Sur.
  • PhD in progress: Nicolas Maignan, Image and video colorization, October 2022, Fabien Pierre, Frédéric Sur
  • PhD in progress: Hugo Leblond, November 2023, , Analyse de scènes dynamiques à partir d’une représentation neuronale implicite (NeRF) basée sur des données LiDAR-caméra, Gilles Simon, Renato Martins
  • PhD in progress: Pengru Zhao, February 2024, Neural network architectures dedicated to crystalline orientations and EBSD, Lionel Germain (LEM3), Frédéric Sur.
  • PhD in progress: Insaf Mellakh, Quantitative analysis of X-ray angiography images in acute ischemic stroke, October 2024, Erwan Kerrien and Julien Oster (IADI, INSERM).
  • PhD in progress: Vaishnavi Kanagabapathi, October 2024, Feature Learning with Temporal/Physical Constraints: Application to Vanishing Point Estimation on Videos, Renato Martins, Cédric Demonceaux and Gilles Simon

8.2.3 Juries

  • Marie-Odile Berger was president of the PhD committee of Léo Valque (Université de Lorraine) and Kassem Dia (Ecole des Mines de Saint-Etienne). She was external reviewer for the PhD thesis of François Lejeune (Université de Strasbourg) and examiner for the PhD of Marco Freire (Université de Lorraine).
  • Gilles Simon was president of the PhD committee of Florian Delconte (Université de Lorraine) and external reviewer for the PhD thesis of Raphael Haenel (Université de Strasbourg).
  • Erwan Kerrien was external examiner for the PhD of François Zhu (Université de Lorraine).

8.3 Popularization

8.3.1 Specific official responsibilities in science outreach structures

  • Pierre-Frédéric Villard is the official scientific godfather of the secondary school of Champigneulles (France) as a "Collège Pilote" of "La Main à la pâte" foundation.
  • Erwan Kerrien is Chargé de Mission for scientific mediation at Inria research center at Université de Lorraine, and thereby is part of the Inria scientific mediation network. As such, he is a member of the steering committee of "la Maison pour la Science de Lorraine", and member of the IREM Lorraine (Institut de Recherche sur l’Enseignement des Mathématiques - Research Institute for Teaching Mathematics) steering council.
  • Erwan Kerrien shares the local coordination of MATh.en.JEANS in the Lorraine area with Samuel Tapie from the IECL lab (mathematics).
  • Erwan Kerrien is a member of the steering committee for the "Programme National de Formation Informatique", the national continuing education program for Ministery of Education executives in charge of overseeing computer science teaching in high schools. A 2 day event was organized in Rennes this year.
  • Erwan Kerrien was involved in a partnership with the Parliament Office for the Evaluation of Scientific and Technological Choices (OPECST - Office Parlementaire d'Évaluation des Choix Scientifiques et Technologiques) as part of a trio also made of a Member of the Senate (Sen. Arnaud Bazin), a member of the National Academy of Surgery (Prof. Jacques Hubert). He organized a visit of the Loria lab by Sen. Bazin.

8.3.2 Productions (articles, videos, podcasts, serious games, ...)

  • Erwan Kerrien participated in the creation of the exposition "Video Games" at Féru des Sciences museum in Jarville.

8.3.3 Participation in Live events

  • Gilles Simon was invited to take part in the EUR Translitterae seminar at the Ecole Normale Supérieure Paris Sciences et Lettres, along with historian Ludovic Balavoine. They gave a joint talk entitled "Representing the living and the dead. The invention of photography in the 15th century".
  • Erwan Kerrien and Gilles Simon took part in a public event on digital sciences, co-organised by the "Le Repaire des Castors" toy library in Blainville-sur-l'Eau, the Inria research center at the University of Lorraine and the Loria laboratory.
  • Pierre-Frédéric Villard gave a presentation at the annual study day of the Association of Medical and Social Sciences Teachers titled "Artificial Intelligence: What Impact(s) on Health and Social Action?" on October 18. The title of the presentation was "AI and Medical Imaging".
  • Erwan Kerrien was an associate researcher to a MATh.en.JEANS workshop within Henri Loritz high school in Nancy, and Pierre Mendès France high school in Contrexéville.
  • Erwan Kerrien animated workshops during the Math and Computer Science festival organized at Rosa Parks high school in Thionville, as well as during events organized by the academic rectorate (Web3.0 day, Math seminar).
  • Erwan Kerrien gave a presentation titled "Algorithms against Stroke" during the AI and Health workshop organized by Metz City.
  • Pierre-Frédéric Villard presented deep learning techniques for automatic character recognition using at the "Fête de la Science" in St-Dié-des-Vosges.
  • Pierre-Frédéric Villard presented demonstrated deep learning techniques at five high school students during an immersion week in the Loria lab.

9 Scientific production

9.1 Major publications

9.2 Publications of the year

International journals

International peer-reviewed conferences

Conferences without proceedings

Doctoral dissertations and habilitation theses

Other scientific publications