Keywords
Computer Science and Digital Science
- A5.3. Image processing and analysis
- A5.4. Computer vision
- A5.4.1. Object recognition
- A5.4.5. Object tracking and motion analysis
- A5.4.6. Object localization
- A5.6. Virtual reality, augmented reality
- A5.10.2. Perception
Other Research Topics and Application Domains
- B2.6. Biological and medical imaging
- B5.9. Industrial maintenance
- B9.5.3. Physics
1 Team members, visitors, external collaborators
Research Scientists
- Marie-Odile Berger [Team leader, INRIA, Senior Researcher, HDR]
- Erwan Kerrien [INRIA, Researcher, HDR]
Faculty Members
- Fabien Pierre [UL, Associate Professor]
- Gilles Simon [UL, Associate Professor, HDR]
- Frédéric Sur [UL, Professor, HDR]
- Pierre-Fredéric Villard [UL, Associate Professor]
- Brigitte Wrobel-Dautcourt [UL, Associate Professor]
PhD Students
- Youssef Assis [UL]
- Nathan Boulangeot [UL]
- Abdelkarim Elassam [INRIA]
- Radhouane Jilani [INRIA]
- Nariman Khaledian [INRIA]
- Liang Liao [CHRU NANCY]
- Nicolas Maignan [UL, from Oct 2022]
- Mehdi Serdoun [CNRS, laboratoire GéoRessources]
- Matthieu Zins [INRIA]
Technical Staff
- Romain Boisseau [INRIA, Engineer]
Interns and Apprentices
- Gaetano Agazzotti [UL, from Sep 2022]
Administrative Assistant
- Isabelle Blanchard [INRIA]
2 Overall objectives
Visual registration is a research topic with a rich history in computer vision. Though a plethora of methods have been developed and can be used for general situations, there are still many open problems which originate in the nature of the scene (poorly textured or specular environments), in the type of motion undergone by the object (tiny motions which hardly emerge from the noise floor, or in contrast, highly deformable objects) and in dissimilarities which may occur in the scene between the time the modeling stage occurs and the application time.
Registration is in practice tightly linked to the choice of the model which represents the scene and the desirable physical properties of the objects. Handling complex —possibly dynamic— scenes thus requires a tradeoff between physical realism of the model, convergence issues and robustness of the registration or tracking tasks.
Recent years have seen a surge in research at the intersection of image and deep learning which has impacted many topics of computer vision. Besides our continued exploration of modeling and registration with traditional approaches derived from signal processing, geometry, and robust estimation, one of the team's aims is to integrate machine learning methods, either as end-to-end methods or as components, into these 2D or 3D geometric tasks.
Targeted trans-disciplinary applications are mixed and augmented reality, computational photomechanics and minimally invasive medical interventions.
3 Research program
3.1 Localization and geometric reasoning with high level features
Our goal is to push forward vision-based scene understanding and localization through the joint use of learning-based methods with geometrical reasoning. Our hypothesis is that the use of intermediate representations instead or in addition to the classical point feature will lead to increased capacity in terms of scale and robustness to changing conditions. These intermediate representations can be concrete objects which are recognized and used directly in the global pose computation, in the continuity of our works on ellipsoid modeling of objects, or conceptual objects such as vanishing points (VP) or horizon lines that are of specific interest both for localization and modeling of urban or industrial scenes.
A first goal is to improve our method for localization from sets of ellipse/ellipsoid correspondences 3. Besides the need to have more accurate prediction of ellipses, another objective is to elaborate robust strategies and associated numerical schemes for refining the initial pose from a set of objects. This requires to develop appropriate metrics for characterizing good reprojection of 3D objects onto 2D ones and study their impact on minimization issues in localization. Another important line of research aims at defining strategies to integrate into the localization procedure various features such as points, objects and VPs, which each bring information at different levels. We especially want to investigate how predictive uncertainty and explainability mechanisms can be used to select and weight these various features in the estimation process.
3.2 Building dedicated models
In this line of research, our goal is to build physically coherent models with a good accuracy vs. efficiency compromise despite the interactive time constraint set in some targeted applications. Though general purpose solutions exist for building models, such techniques are still greatly challenged in more complex cases when specific constraints on the shape or its deformation must be met. This is especially the case in medical imaging of thin deformable organs, such as the diaphragm, the mitral valve or blood vessels, but also for classical scene modeling where constraints, such as ellipsoidal abstraction of objects, must be introduced. The use of mechanical models have become increasingly important in the team activities in medical imaging, especially for handling organs with large deformations. Our goal is to push forward the development of such models with image-guided procedures or predictive simulation in view.
Facing difficulties of meshing complex geometries, especially thin ones, we want to promote mesh free methods such as implicit models. In the continuity of past works 4, automatic adaptation of node locations and sizes to the image will be investigated to improve compactness, and computational efficiency of implicit models. As the fidelity of a mechanical model is often impaired by approximations required to solve its dynamical system equations at interactive frame rates, a second goal is to take advantage of our implicit models to improve contact and deformation resolution.
The second axis is about investigating shape-aware methods either for shape segmentation or shape recognition, in order to be able to enforce global shape constraints or geometric shape priors in the output of CNNs. This topic is still addressed in the team in the context of localization from 3D ellipsoidal abstraction of objects 8. Two applications are especially targeted: (i) We aim at improving the detection of pathologies (e.g. brain aneurysms that are mostly located at vessel bifurcations), through adapted and guided sampling of input data during training, as well as through mechanisms inspired by visual attention modules. (ii) In the context of fluid structure simulation methods for patient-based mitral valve simulation, it often appears that the geometric segmented model leads to divergence of the numerical scheme. Our intention is to identify geometric conditions under which simulation works well with the idea to incorporate them in the segmentation process.
3.3 Estimation and inverse problems
Most aforementioned tasks lead to image-based inverse, possibly ill-posed, problems. While some of them can be solved with well-established estimation techniques, others necessitate the design of new strategies. In this perspective, we consider in this research axis several fundamental aspects of estimation, common to our problems, such as sampling methods, traditional optimization methods, or end-to-end learning methods for pose estimation.
3.3.1 Optimization, variational calculus and numerical schemes
We are interested in non-convex optimization problems, especially those raised by variational calculus. While the convergence of numerical schemes is well established for convex problems, this is not always the case for non-convex functionals. Our aim is to continue the work already carried out in the biconvex framework 7, and extend it to primal-dual algorithms. We especially want to address energy minimization problems where the energy is convex with respect to each variable, but non-convex with respect to the pair of variables.
Another research topic is to investigate new neural architectures adapted to non-Euclidean data, and also to plug variational methods into deep learning approaches to regularize the results. The obtained theoretical results will be applied to image colorization, with the idea to reduce artefacts caused both by a lack of regularization and by the non-Euclidean structure of color information as perceived by the human visual system.
3.3.2 Machine learning for physical problems
We aim at continuing our efforts towards supervised and unsupervised learning for estimation problems. Concerning supervised learning, we intend to investigate further the opportunities offered by neural network estimation of displacement and strain fields in experimental mechanics that we have recently introduced with colleagues in mechanics and signal processing 1. Besides, we also aim at developing unsupervised learning in problems where a quantity has to be estimated over a spatio-temporal domain, which is a recent trend in several application domains. Neural networks are indeed universal approximators whose derivative can be exactly computed with the backpropagation algorithm, which is supposed to make them robust to acquisition noise.
4 Application domains
Applications on which our program is expected to have an impact are mixed reality, computational photomechanics and minimally invasive medical interventions. These fields correspond to areas where we have established trans-disciplinary collaborations with academic or industrial experts of the applicative fields. Common to these applications are the need for finely characterizing the acquisition context of vision-based applications and the need for accurate registration procedures. Another common point is the availability of a limited number of data for characterizing the variability of the observed phenomena.
Mixed reality
Being able to perform reliable and accurate registration under large viewpoint variations, seasonal or lighting changes opens the way towards challenging mixed reality applications. Urban AR and industrial maintenance in large and cluttered environments are examples of application fields that would successfully capitalize on more robust localization solutions. Improved robustness of camera localization is especially expected for poorly textured, specular environments and in the presence of repeated patterns that are common in industrial contexts
Photomechanics
Photomechanics is the field of experimental mechanics which is dedicated to mechanical measurement from images. In particular, we are interested in contactless image-based methods for extensometry, that is the estimation of displacement and strain fields on the surface of materials subjected to different types of mechanical loads. Full-field extensometry is a challenging task since strains often have tiny values and result in gray level changes at the limit of the sensor noise floor. The economic stakes are high and concern for example the automotive and aeronautics industries, or civil engineering. In order for these methods to be adopted by industry, it is, however, necessary to quantify their metrological performance, which is limited by the registration process or by the image acquisition chain, and especially by sensor noise. This topic is the subject of a long-term trans-disciplinary collaboration with Institut Pascal (Clermont-Ferrand Université).
Minimally invasive medical interventions
The trend towards the design and performance of minimally invasive procedures will increase in the near future. But the benefit for the patient is at the expense of the surgeon who can only sense the surgical scene through intra-operative imaging. Commercial solutions now exist to teach this increasingly difficult surgical gesture with interactive simulation technologies. However, challenges remain to fill the gap between the learning environment, where qualitative correctness of the setup is sufficient, and the surgical theater, where accuracy and predictability are required. In this context, we aim at addressing the key problem of modeling the geometry and dynamics of deformable organs and surgical devices, in order to make progress towards a faithful 3D rendition of the surgical scene. To circumscribe practical and experimental difficulties, three specific applications will be addressed with our clinical partners: intra-operative guidance in interventional neuroradiology, augmented reality for laparoscopic liver surgery, and simulation of the mitral valve behaviour.
5 Highlights of the year
5.1 Awards
The Tangram team received the best paper award at the International Conference on Information Processing in Computer-Assisted Interventions (IPCAI 2022) for the paper “Capturing Contact in Mitral Valve Dynamic Closure with Fluid-Structure Interaction Simulation” written by Nariman Khaledian, Pierre-Frederic Villard and Marie-Odile Berger 10.
6 New software and platforms
6.1 New software
6.1.1 DeepEllPose
-
Name:
Deep Ellipses Pose
-
Keywords:
Pose estimation, Visual localization, Ellipses, Neural networks
-
Functional Description:
DeepEllPose contains the training and inference code of the object detection and ellipse prediction networks (PyTorch). This code corresponds to an extended implementation of the article: 3D-Aware Ellipse Prediction for Object-Based Camera Pose Estimation 13. The 7-Scenes dataset is used as example, but the provided tools make it easily applicable to other datasets. The library also contains the code for camera pose estimation from 3 pairs ellipse-ellipsoid, as well as tools to easily manipulate and visualize such objects.
- URL:
-
Contact:
Matthieu Zins
6.1.2 OA-SLAM
-
Name:
Object -aided SLAM
-
Keywords:
Localization, 3D reconstruction, Object detection
-
Scientific Description:
Details on the method can be found in the paper published in ISMAR 2022 15.
-
Functional Description:
OA-SLAM uses objects as landmarks to improve the relocalization capabilities of SLAM systems. OA-SLAM builds on the point-based ORB-SLAM2. It allows online reconstruction of 3D objects modeled as ellipsoids from their detections in 2D images. OA-SLAM dramatically improves the relocalization capabilities of SLAM.
- URL:
-
Contact:
Matthieu Zins
7 New results
7.1 Visual localization
Participants: Marie-Odile Berger, Romain Boisseau, Gilles Simon, Matthieu Zins.
Recent years have seen the emergence of very effective ConvNet-based object detectors that have reconfigured the computer vision landscape. As a consequence, new approaches that propose object-based reasoning to solve traditional problems, such as camera pose or SLAM estimation, have appeared. In the context of Matthieu Zins' PhD thesis 20, and following the track of research initiated in 3, we have continued to investigate how to use objects as landmarks for localization issues. Objects are here approximated by ellipsoids from their 2D detection in images as ellipses. Using generic object detectors thus allows handling any environment without accurate knowledge of its geometry.
A first contribution of the year is a 3D-Aware method for predicting more accurate elliptic approximations of objects, thus leading to improved pose estimations 13. We then addressed in 14 the problem of refining the rough pose estimate, computed from the best subset of 2 or 3 ellipse/ellipsoids correspondences, by using all the available correspondences. Contrary to point correspondences, the definition of a cost function characterizing the adequation of the projection of a 3D object onto a 2D object is not straightforward. We first studied various metrics commonly used to quantify the matching of ellipses (e.g. IoU, Bbox corners, distance to contour) and evaluated their performance in the context of camera pose refinement. We also proposed a metric based on level sets that outperforms existing metrics. Finally, we showed that using a predictive uncertainty on the detected ellipses allows a fair weighting of the contribution of the correspondences, which noticeably improves the computed pose.
Lastly, we explored in 15 the use of objects in Simultaneous Localization and Mapping in unseen worlds and proposed an object-aided system (OA-SLAM). More precisely, we showed that, compared to low-level points, the major benefit of objects lies in their higher-level semantic and discriminating power. Points, on the contrary, have a better spatial localization accuracy than the generic coarse models used to represent objects (cuboid or ellipsoid). We showed that combining points and objects is of great interest to address the problem of camera pose recovery. With OA-SLAM, we propose an automatic system, capable of identifying, tracking, and reconstructing objects with 3D ellipsoids. Our fully automatic system allows on-the-fly object mapping and enhanced pose tracking recovery thanks to objects. This paves the way towards increased robustness of augmented reality applications. Code and datasets are freely released at gitlab.inria.fr/tangram/oa-slam.
7.2 Vanishing point computation and applications
Participants: Marie-Odile Berger, Abdelkarim Elassam, Gilles Simon.
Vanishing point detection is a prerequisite for many computer vision problems, ranging from localization in human-made environments 2 to perspective analysis in paintings 6. Classical approaches, based on the grouping of line segments that meet at the same point, pose difficulties, especially regarding fortuitous convergences and the representativeness of the tiniest structures. Abdelkarim Elassam's PhD aims at taking advantage of image semantics through deep learning methods. Our first work 18 addressed the challenge of generating an arbitrary number of vanishing points, whereas the network architectures used previously could only generate a fixed number of them. The key of our method was to transform a regression problem into three classification ones concerning the offset of the horizon line, its slope, and the position of the vanishing points on the horizon line. In each case, the classification scores are considered as describing a probability distribution whose peaks correspond to the output values. This method proved competitive on three reference datasets but often failed to detect some vanishing points, especially those associated with small structures.
To tackle this issue, we recently designed a multi-task model that jointly estimates the horizon line, vanishing points and a segmentation map of the image, where each non-null class is associated with a range of possible orientations for the vanishing point generated by the underlying structure. This method allows to obtain vanishing points corresponding to a priori non meaningful peaks of the classification scores, sometimes even to non-existent peaks, in particular on small structures. In addition, it provides 2D masks of the structures associated with vanishing points, which can prove valuable in certain application contexts, particularly for SLAM. It is currently under revision for ICRA 2023.
7.3 Handling non rigid deformation
Participants: Marie-Odile Berger, Nariman Khaledian, Pierre-Frédéric Villard.
7.3.1 Individual mitral valve modeling
We previously worked on the segmentation of the valve anatomy and started last year a work on modeling mitral valve closure using fluid-structure interaction (FSI). This year, we focused first on a FSI model with a generic valve that improves the detection of blood leakage by building a map of contact 10. Our model is based on the immersed boundary method that captures a map of contact and perfect closure of the mitral valve, without the presence of orifice holes, which often appear with existing methods. We also identified important factors influencing convergence issues. Our method was demonstrated in three typical clinical situations: mitral valve with leakage, bulging, and healthy. In addition to the classical ways of evaluating mitral valve closure, such as stress distribution and flow rate, the contact map provides easy detection of leakage with identification of the sources of leakage and a quality assessment of the closure.
We then started to work on incorporating anisotropy in our model to get closer to the real valve behavior. Results on our generic model show that anisotropy leads to less bulging than using an isotropic constitutive law, which is consistent with clinician observations. We are currently investigating valve mitral closure simulation from real valve automatically segmented from porcine data.
7.3.2 Image-based biomechanical simulation of the diaphragm during mechanical ventilation
Following our previous research on modeling the diaphragm with a Radial Basis Function (RBF) formulation within the INVIVE project, we focused this year on 2D simulations 11. We used a system of partial differential equations that model linear elasticity. We computed displacements and stresses in a 2D cross section of the diaphragm in its contracted state. The boundary conditions consist of a mix of displacement and traction conditions. If these are imposed as they are, and the conditions are not compatible, this leads to reduced smoothness of the solution. Therefore, the boundary data was first smoothed using the least-squares RBF generated finite difference framework. Then the boundary conditions were reformulated as a Robin boundary condition with smooth coefficients. The same framework was also used to approximate the boundary curve of the diaphragm cross section based on data obtained from a slice of a CT scan. To solve the PDE we employed the unfitted least-squares RBF method. This made it easier to handle the geometry of the diaphragm, which is thin and non-convex. We showed numerically that our solution converges with high-order towards a finite element solution evaluated on a fine grid. Through this simplified numerical model, we also gained an insight into the challenges associated with the diaphragm geometry and the boundary conditions before approaching a more complex three-dimensional model.
7.4 Interventional radiology
Participants: Youssef Assis, Marie-Odile Berger, Radhouane Jilani, Erwan Kerrien, Pierre-Frédéric Villard.
7.4.1 Detection of brain aneurysms using deep learning
Youssef Assis's PhD thesis aims at the detection of brain aneurysms from 3D Time-of-Flight (TOF) MRI images with a deep learning approach. A preliminary clinical evaluation of last year's method 16 led us to revisit the methodology used by the ADAM challenge and most publications about the detection of brain aneurysms with deep learning techniques. In their vast majority, such methods rely on a U-net architecture, which was designed for segmentation problems. The usual criteria used to evaluate their performance (e.g. based on connected component extraction) are thereafter not suited to a detection problem. We reassessed our method against performance measures used in computer vision for detection problems (e.g. Average Precision (AP)). We also implemented an anchor-free version of YOLOv3 that processes 3D patches as input and outputs a bounding sphere associated with a level of confidence on the detection within the patch. We compared this new network against nnDetection1 on 3 databases: an in-house dataset, collected in Nancy University Hospital with our medical partners (132 patients), ADAM training database (93 subjects) and CHUV Lausanne public database. Our results demonstrate an improved performance slightly better sensitivity, fewer False Positives/case) with a much reduced training time. These results are currently being exhaustively reviewed by medical experts for consolidation with regard to the many annotations errors in the databases.
7.4.2 Clinical evaluation of registration and segmentation software
Our work in collaboration with Nancy University Hospital and GE Healthcare company was presented at ABC-Win seminar 17. It addressed the clinical evaluation of a blood vessel segmentation and brain image registration software to help planning and performing the endovascular treatment of brain arteriovenous malformations.
7.4.3 Predictive simulation of catheter navigation
Off-the-shelf catheter simulation algorithms made available by the Sofa framework were investigated through a comparison with ground-truth catheter shapes reconstructed with our experimental validation bench (real catheters reconstructed by stereovision while deployed in a vascular phantom in silicon). Despite their capacity to perform interactively, these algorithms demonstrated a low accuracy, that cannot meet the requirements for planning interventional neuroradiology procedures. Radhouane Jilani's PhD thesis aims to achieve collision with implicit, continuous, blood vessel surfaces 4. Therefore, we investigated Cosserat rod models designed for continuous robots. Such methods consider the rod a parametric function instead of a sequence of discrete elements. The derived Boundary Value Problem (BVP) can be solved using the shooting method or a collocation method. We adapted the collocation method to be used with an implicit dynamics framework and proved its superior stability over a wider range of deformations.
7.5 Image and signal processing
Participants: Gaetano Agazzotti, Nicolas Maignan, Fabien Pierre, Frédéric Sur.
7.5.1 Computational photomechanics
The work of this year concerns deep learning networks able to estimate the displacement and strain fields over a material submitted to a mechanical load. Displacement field estimation is related to optical flow estimation in computer vision 19. In order to process the surface of the material, a contrasted marking is projected on it, giving a speckle pattern. The first goal was to customize a state-of-the-art convolutional neural network dedicated to optical flow estimation (processing natural images) to reach better performance when processing speckle images. The second goal was to simplify the CNN by reducing as much as possible the number of filters while keeping equivalent metrological performance to the original version (and almost equivalent to state-of-the-art digital image correlation estimation), in order to accelerate image processing on a power-efficient compact Graphics Processing Unit (GPU). The ultimate goal is to develop smart-cameras dedicated to displacement and strain measurement. A paper journal has been accepted at the end of year 2022 9.
7.5.2 Variational methods for image processing
During the internship of Gaetano Agazzotti, the combination of deep image priors and variational method has been explored for single image colorization. The deep image prior model is able to produce realistic image while suffering of overfitting of the data that produces noisy images and blurring of the colorization results. With a regularization of the result with total variation, these effects can be avoided and controlled.
The colorization of videos is mostly based on color transfer between frames. The state-of-the-art methods suffer from occlusions and dis-occlusions during the matching phase. Since such problems are generally solved through 3D knowledge on the scene, the PhD thesis of Nicolas Maignan is aiming at understanding the 3D scene from a monocular video sequence.
7.6 Application of deep learning to image and signal processing
Participants: Nathan Boulangeot, Mehdi Serdoun, Frédéric Sur.
7.6.1 Extensive searches for complex intermetallic catalysts
Materials science is at the heart of many technological revolutions. The search for new materials is crucial to meet the challenges posed by climate change, or the growing global demand for energy and consumer goods. In this context, the discovery of new catalytic materials is a major challenge to improve the performance of industrial processes, for example in the context of the hydrogen economy, in order to provide energy sources with limited environmental impact. Although some theoretical models exist, catalyst research is still largely empirical and based on a trial-and-error approach. This is probably because the application of theoretical models requires particularly expensive calculations based on density functional theory (DFT). Recently, the materials community has been seizing on statistical learning tools to limit the cost of calculations and to accelerate the discovery of new catalysts. Together with Emilie Gaudry at Institut Jean-Lamour (IJL) at Nancy, we have developed a collaboration on this subject since September 2021 through Nathan Boulangeot's PhD thesis. The work of this year concerns developping models for infering the energy of the system and the forces applied to the atoms in the context of predicting the adsorption properties of , based on Gaussian process regression.
7.6.2 Multivariate analysis of geochemical, physical and mineralogical signatures of uranium deposits in the Athabasca Basin (Saskatchewan, Canada)
We are engaged in the co-supervision (together with Julien Mercadier, GéoRessources) of the PhD thesis of Mehdi Serdoun, which is part of the GeoMin3D project funded by ANR and Orano Mining. The goal is to develop statistical learning models to analyze the large amount of data of diverse nature provided during the exploratory drillings in Athabasca basin, the largest known source of uranium. The ultimate goal is to develop new analysis tools to accelerate exploration and reduce its cost, in cooperation with the industrial actors.
8 Partnerships and cooperations
8.1 International initiatives
8.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program
CURATIVE
-
Title:
CompUteR-based simulAtion Tool for mItral Valve rEpair
-
Duration:
2021-2024
-
Coordinator:
Robert Howe (howe@seas.harvard.edu)
-
Partners:
- Harvard University Cambridge (États-Unis)
-
Inria contact:
Pierre-Fredéric Villard
-
Summary:
The mitral valve of the heart ensures one-way flow of oxygenated blood from the left atrium to the left ventricle. However, many pathologies damage the valve anatomy producing undesired backflow, or regurgitation, decreasing cardiac efficiency and potentially leading to heart failure if left untreated. Such cases could be treated by surgical repair of the valve. However, it is technically difficult and outcomes are highly dependent upon the experience of the surgeon. One of the main difficulties of valve repair is that valve tissues must be surgically altered during open heart surgery such that the valve opens and closes effectively after the heart is closed and blood flow is restored. In order to do this successfully, the surgeon must essentially mentally predict the displacement and deformation of anatomically and biomechanically complex valve leaflets and supporting structures [11]. Even if patient-based mitral valve models have been recently used for scientific understanding of its complex physiology, the patient geometry is manually segmented on medical images. This task is long and cumbersome except if the valve has been artificially isolated in-vitro. There is a lack in the literature about the variety of metrics in both anatomy and biomechanics of the valve. In order to study mitral valve behavior or to prepare models for planning, it is necessary to have methods to extract the valve components i) on real clinical data ii) with minor user input and ii) that is mechanically valid.
8.2 International research visitors
8.2.1 Visits of international scientists
Other international visits to the team
Douglas Perrin
-
Status
researcher
-
Institution of origin:
Harvard Medical School
-
Country:
USA
-
Dates:
10/10/2022 - 21/10/2022
-
Context of the visit:
Visit within the context of the associated team CURATIVE. In order to perform mitral valve modeling with real patient's data, our discussions focussed on (i) future experiments in Boston to determine anisotropic constitutive law on porcine data (ii) the analysis of our first results with real patient's data.
-
Mobility program/type of mobility:
research stay
Peter Hammer
-
Status
researcher
-
Institution of origin:
Harvard Medical School
-
Country:
USA
-
Dates:
10/10/2022 - 14/10/2022
-
Context of the visit:
Visit within the context of the associated team CURATIVE. In order to perform mitral valve modeling with real patient's data, our discussions focussed on (i) future experiments in Boston to determine anisotropic constitutive law on porcine data (ii) the analysis of our first results with real patient's data.
-
Mobility program/type of mobility:
research stay
8.2.2 Visits to international teams
Research stays abroad
Pierre-Frédéric Villard
- Visited institution: Uppsala University
-
Country:
Sweden
-
Dates:
04/01 - 04/31
-
Context of the visit:
Visit within the context of the project INVIVE. The aim was to train and supervise a medical student in the segmentation of new medical data, to supervise a PhD student on the mechanical modeling of the diaphragm and to attend the PhD defense of another student for whom he was the co-supervisor.
-
Mobility program/type of mobility:
research stay
Pierre-Frédéric Villard
- Visited institution: Boston Robotic Lab
-
Country:
USA
-
Dates:
11/01 - 11/30
-
Context of the visit:
Visit within the context of the associated team CURATIVE. The aim was mainly to perform experiments on real porcine valves: anisotropy and tension measurements were performed. He also worked on preparing future experiment of a phantom valve closing in a fully-controlled environment and did a CT scan of the same phantom valve. He participated to the Harvard Robotic lab life by attending and giving seminars.
-
Mobility program/type of mobility:
research stay
8.3 European initiatives
8.3.1 Other european programs/initiatives
MOVEON
-
Title:
Towards robust spatial scene understanding in dynamic environments using intermediate representations
-
Partner Institution(s):
DFKI Kaiserslautern
-
Participants:
M.-O. Berger, R. Boisseau, A. Elassam, G. Simon, M. Zins
-
Date/Duration:
2020-2023
-
Additionnal info/keywords:
The aim of the MOVEON project is to push forward the state of the art in vision- based, spatio-temporal scene understanding by merging novel machine-learning approaches with geometrical reasoning. Deep-learning-based recognition and understanding of high-level concepts such as vanishing points or large object classes will serve as unitary building blocks for a spatio-temporal localization and environment reconstruction that will use geometric reasoning as underlying support. This research will lead to a novel generation of visual positioning systems that go beyond classical localization and mapping, which focuses currently only on point cloud reconstruction. In contrast, our aim is to allow for 6DoF positioning and global scene understanding in wild and dynamic environments (e.g. crowded streets) that scales up nicely with the size of the environment, and that can be used persistently over time by reusing consistent maps.
8.4 National initiatives
ANR Arcé
- Title:
- Partner Institution(s): Université de Lorraine
-
Participants:
F. Pierre, N. Maignan, F. Sur
-
Date/Duration:
2019-2022
-
Additionnal info/keywords:
The Arcé project aims at proposing new methods for automatic, fast and perceptually satisfying video colorization. Image colorization methods based on deep learning based have encountered a great success in recent years. These techniques are fully automatic and very fast, but they have not been adopted by colorization industry. The reason is that they do not ensure the temporal coherence of the colorization, which is particularly disturbing for the viewer. The ultimate goal is the use of our work in audiovisual production studios.
ANR JCJC ICaRes
-
Title:
Image correlation for the accurate measurement of residual stress
-
Partner Institution(s):
Université Clermont-Auvergne
-
Participants:
F. Sur
-
Date/Duration:
2019-2022
-
Additionnal info/keywords:
This 3-year project (2019-2022) headed by B. Blaysat (Université Clermont-Auvergne), is supported by the Agence Nationale de la Recherche. It addresses residual stresses, which are introduced in the bulk of materials during processing or manufacturing. Since unintended residual stresses often initiate early failure, it is of utmost importance to correctly measure them. The goal of the ICaRes project is to improve the performance of residual stress estimation through the so-called virtual digital image correlation (DIC) which will be developed. The basic idea of virtual DIC is to mark the specimen with virtual images coming from a controlled continuous image model, instead of the standard random pattern. Virtual DIC is expected to outperform standard DIC by, first, matching real images of the materials with the virtual images, then, to run DIC on the virtual images on which strain fields are estimated, giving ultimately residual stresses.
ANR PRC PreSPIN
-
Title:
Predictive Simulation for Planning Interventional Neuroradiology procedures
-
Partner Institution(s):
CReSTIC (Reims), Creatis (Lyon) and CIC-IT/CHRU Nancy
-
Participants:
R. Jilani, E. Kerrien, P.-F Villard.
-
Date/Duration:
2020-2024
-
Additionnal info/keywords:
This 4-year project is coordinated by E. Kerrien. It aims at improving the planning phase in the therapeutic management of cerebral ischemic strokes thanks to predictive simulation of both the therapeutic interventional gesture and post-interventional perfusion images. The consortium is set to address the challenges of geometrical and topological modeling of the full brain vasculature; physics-based simulation of interventional devices; simulation of MRI perfusion images; and clinical validation.
9 Dissemination
9.1 Promoting scientific activities
9.1.1 Scientific events: selection
Reviewer
- Marie-Odile Berger was a reviewer for ISMAR (International Symposium for Mixed and Augmented Reality), IPCAI (International Conference on Information Processing in Computer-Assisted Interventions), IROS , ICRA and for the French conference RFIAP.
- Pierre-Frederic Villard was a reviewer for the Eurographics Workshop on Visual Computing for Biology and Medicine 2022 and the International Conference on Computer Graphics, Visualization, Computer Vision And Image Processing 2022.
- Gilles Simon was a reviewer for ICRA 2022.
- Erwan Kerrien was a reviewer for MIDL (Medical Imaging with Deep Learning), MICCAI (Medical Image Computing and Computer Assisted Interventions), and ISMAR.
9.1.2 Journal
Member of the editorial boards
- Marie-Odile Berger was a member of the International Program Committee (IPC) for the journal paper track (IEEE TVCG) of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2022.
Reviewer - reviewing activities
- Marie-Odile Berger was a reviewer for Pattern Recognition.
- Frédéric Sur was a reviewer for IEEE Transactions on Image Processing, IEEE Transactions on Instrumentation and Measurement, Experimental Mechanics, Optical Letters, International Journal of Mechanical Sciences, Optics and Lasers in Engineering.
- Gilles Simon was a reviewer for The Visual Computer (Springer) and Measurement (Elsevier).
- Erwan Kerrien was a reviewer for IEEE Transactions on Medical Imaging, the International Journal of Computer Assisted Radiology and Surgery, IEEE Transactions on Robotics and IEEE Robotics and Automation Letters.
9.1.3 Invited talks
- Pierre-Frédéric Villard did a presentation to the Harvard Biorobotics Lab within the CURATIVE collaboration. Title: "Mitral Valve Modeling: From Medical Image Analysis to Fluid Structure Interaction" on Nov 17th 2022.
- Nariman Khaledian did a presentation to the Harvard Biorobotics Lab within the CURATIVE collaboration. Title: "Capturing Contact in Mitral Valve Dynamic Closure with Fluid-Structure Interaction Simulation" on January 14th 2022.
- Pierre-Frédéric Villard gave a seminar at the department of information technology of Uppsala University. Title: "Simulation of Mitral Closing" on May 19th 2022.
- Romain Boisseau and Alain Pagani gave a presentation of the MOVEON project at the VIVATECH event. Title: "Towards Robust Spatial Scene Understanding in Dynamic environments Using Intermediate Representations", on June 17th 2022.
- Erwan Kerrien gave an invited lecture during the scientific days of FLI-RE3 in Strasbourg (French research network on interventional imaging, part of the France Life Imaging infrastructure). Title "Simulation for the planning of endovascular thrombectomy" on June 29th 2022.
9.1.4 Leadership within the scientific community
- Marie-Odile Berger is the president of the Association française pour la reconnaissance et l’interprétation des formes (AFRIF).
9.1.5 Scientific expertise
- Marie-Odile Berger was a member of the recruitment committee for a Professor position at INSA Rennes and at Université de Strasbourg
9.1.6 Research administration
- Marie-Odile Berger was the president of a jury for the recruitement of a research engineer specialized in computer graphics at INRIA Nancy Grand Est
- Gilles Simon is an elected member of the CNU (Conseil National des Universités).
9.2 Teaching - Supervision - Juries
9.2.1 Teaching
The assistant professors of the TANGRAM team actively teach at Université de Lorraine with an annual number of around 200 teaching hours in computer sciences, some of them being accomplished in the field of image processing. INRIA researchers have punctual teaching activities in computer vision and shape recognition mainly in the computer science Master of Nancy and in several Engineering Schools near Nancy (ENSMN Nancy, SUPELEC Metz, ENSG). Our goal is to attract Master students with good skills in applied mathematics towards the field of computer vision.
The list of courses given by staff members which are related to image processing and computer vision is detailed below:
- M.-O. Berger
- Master : Shape recognition, 24 h, Université de Lorraine.
- Master : Introduction to image processing, 12 h, ENSMN Nancy.
- Master : Image processing for Geosciences, 12h, ENSG.
- E. Kerrien
- Master : Introduction to image processing, 15 h, ENSMN Nancy.
- Licence : Initiation au développement, 80h, IUT St Dié-des-Vosges.
- Fabien Pierre
- Master: Introduction à l’apprentissage automatique, 14h, Mines Nancy.
- Master: Vision artificielle et traitement des images, 12h, Polytech Nancy.
- Licence: Introduction au traitement d’image, 30h, IUT Saint-Dié des Vosges.
- Licence: Algorithmique et programmation, 87h, IUT Saint-Dié des Vosges
- Licence: Culture scientifique et traitement de l’information, 69h, IUT Saint-Dié des Vosges
- Licence: Programmation objet et évènementielle, 35h, IUT Saint-Dié des Vosges
- Licence: Initiation à l’intelligence artificielle, 18h, IUT Saint-Dié des Vosges
- G. Simon
- Master: Augmented reality, 9 h, Télécom-Nancy.
- Master: Augmented reality, 24h, M2 Informatique FST
- Master: Visual data modeling, 12h, M1 Informatique FST
- Master: Computer Vision, 12h, M1 Informatique FST
- Licence pro: 3D modeling and augmented reality, 50h FST - CESS d’Epinal
- Licence: Programming methodology, L1 informatique, 48h FST
- F. Sur
- Master: Introduction to machine learning, 40 h, Université de Lorraine (Mines Nancy).
- Licence: Javascript programming, 100h, IUT Charlemagne
- P.-F Villard
- Master : Augmented and Virtual Reality, 16h, M2 Cognitive Sciences and Applications, Institut des Sciences du Digital, Université de Lorraine
- Licence: Computer Graphics with webGL, 30h, IUT Saint-Dié des Vosges.
- Licence: Virtual and Augmented Reality in Industrial Maintenance, 2h, Faculty of Science and Technology, Université de Lorraine
- Licence: Web programming, 20h, IUT Saint-Dié des Vosges.
- Licence: Graphical user interface programming, 30h, IUT Saint-Dié des Vosges.
- Licence: Security and life privacy with internet, 2h, IUT Saint-Dié des Vosges.
- Licence: Parallel programming, 18h, IUT Saint-Dié des Vosges.
- Licence: Initiation to machine learning, 24h, IUT Saint-Dié des Vosges.
- Licence: Initiation to cryptography, 12h, IUT Saint-Dié des Vosges.
- B. Wrobel-Dautcourt
- Master: modélisation objet et conception des systèmes d’information, 30h, Télécom
- Master: projet de conception et développement java, 27h, Télécom 2A
- Master: ingéniérie logicielle, 12h, FST
- Licence: bases de la programmation objet, 44h, FST
- Licence: interfaces graphiques,22h, FST
- Licence: projet de synthèse (activité intégratrice), 30 h, FST
- Licence: système, 24h, FST
- Licence: compilation, 16h, FST
9.2.2 Supervision
- PhD defended: Matthieu Zins, Contribution à la précision et à la robustesse de la localization dans un monde d'objets, December 2022, Marie-Odile Berger, Gilles Simon.
- PhD in progress: Abdelkarim Elassam, Robust visual localization using high level features, October 2020, Marie-Odile Berger, Gilles Simon.
- PhD in progress: Nariman Khaledian, Toward a Functional Model of the Mitral Valve, October 2020, Marie-Odile Berger, Pierre-Frédéric Villard.
- PhD in progress: Youssef Assis, Deep learning for the automated detection of brain aneurysms, November 2020, Erwan Kerrien, René Anxionnat (CHRU Nancy).
- PhD in progress: Radhouane Jilani, Predictive simulation for interventional neuroradiology, October 2021, Erwan Kerrien, Pierre-Frédéric Villard.
- PhD in progress: Liang Liao, Detection of cerebral aneurysms from MRI images using deep learning: deep neural network creation and its clinical evaluation, November 2022, René Anxionnat (CHRU Nancy) and Erwan Kerrien.
- PhD in progress: Nathan Boulangeot, Extensive searches for complex intermetallic catalysts, October 2021, Émilie Gaudry (Institut Jean-Lamour), Frédéric Sur.
- PhD in progress: Mehdi Serdoun, Multivariate analysis of mineralogical, geochemical and physical signatures, January 2022, Julien Mercadier (GéoRessources), Frédéric Sur.
- PhD in progress: Nicolas Maignan, Image and video colorization, October 2022, Fabien Pierre, Frédéric Sur
9.2.3 Juries
- Marie-Odile Berger was president of the PhD committee of Pierre schegg (Inria Lille) and Semyon Efremov (Université de Lorraine) and of Romain Serizel's HDR committe ( Université de Loraine). She was external reviewer for the PhD thesis of Lulin Zhang (IGN), Matthieu Gonzalez (IRISA, Rennes) and Laurane Charrier (LISTIC, Annecy)
- Frédéric Sur was a member of the PhD committees of Mariane Prado Motta (LEM3 Saint-Dié-dès-Vosges) and Yiheng Shi (Université Clermont-Auvergne)
- Erwan Kerrien was an external reviewer for the PhD thesis of Guillaume Mestdagh (Université de Strasbourg) and was a member of the PhD committee of Rémi Decelle and Arturo Consoli (Université de Lorraine).
9.3 Popularization
9.3.1 Internal or external Inria responsibilities
- Erwan Kerrien is Chargé de Mission for scientific mediation at Inria Nancy-Grand Est, and thereby is part of the Inria scientific mediation network. As such, he is a member of the steering committee of "la Maison pour la Science de Lorraine", and member of the IREM Lorraine (Institut de Recherche sur l’Enseignement des Mathématiques - Research Institute for Teaching Mathematics) steering council. He is also the local scientific referent for the "Chiche!" initiative.
9.3.2 Articles and contents
- A report on Gilles Simon's work 6 was broadcast on German television (D. Differdange, Versteckt: Informatiker aus Nancy entschlüsselt Van Eyck-Code, SR, Jan 2022).
9.3.3 Education
- Pierre-Frédéric Villard is the scientific godfather of the secondary school of Champigneulles (France) as a "Collège Pilote" of "La Main à la pâte" foundation. He gave a seminar on augmented and virtual realities to the pupils, he helped the teacher with preparing some activities with augmented and virtual reality technologies. Eventually, he is supervising master students to produce teaching applications with augmented reality technologies that will be used in secondary school classes.
- Pierre-Frédéric Villard presented a workshop on automatic character recognition using deep learning technique at the "Fête de la Science" in St-Dié-des-Vosges.
- Gilles Simon gave a talk at the Rosa Parks high school in Thionville on the theme "Geometry and happiness" as part of the "Regards de géomètres" event.
- Erwan Kerrien was an associate researcher to a MATh.en.JEANS workshop within Henri Loritz high school in Nancy. He presented the MATh.en.JEANS initiative during a DAAC workshop (Culture and Art Academic Direction). He also participates in the "Maths and games" group at IREM Lorraine.
- Erwan Kerrien animated a workshop about "Binary height charts" during the APMEP day (Maths teachers association in public education), and another one about "Images and statistics" during the NSI day (high school computer science teachers association).
- Pierre-Frédéric Villard and Erwan Kerrien did presentations to high school students in the context of the "Chiche!" initiative.
9.3.4 Interventions
- Gilles Simon participated in a conference-debate for the general public on the profession of researcher as part of the event "La Fabrique des chercheurs" at the university library of Vandœuvre-lès-Nancy.
- Erwan Kerrien and Gilles Simon proposed an animation at the "Nuit Européenne des chercheur.e.s" which took place in Nancy in October 2022.
- Erwan Kerrien gave a lecture about "Computer science: A science for every day life" in Strasbourg and again in Lycée Louis Vincent high school in Metz, both in connection with the Homo Numericus exposition designed by Inria Nancy Grand Est.
10 Scientific production
10.1 Major publications
- 1 articleWhen Deep Learning Meets Digital Image Correlation.Optics and Lasers in Engineering136January 2021, 106308
- 2 articleModel-image registration of a building’s facade based on dense semantic segmentation.Computer Vision and Image Understanding206May 2021, 103185
- 3 inproceedingsCamera Relocalization with Ellipsoidal Abstraction of Objects.ISMAR 2019 - 18th IEEE International Symposium on Mixed and Augmented RealityBeijing, ChinaIEEEOctober 2019, 19-29
- 4 articleBlood vessel modeling for interactive simulation of interventional neuroradiology procedures.Medical Image Analysis35January 2017, 685 - 698
- 5 inproceedingsA-Contrario Horizon-First Vanishing Point Detection Using Second-Order Grouping Laws.ECCV 2018 - European Conference on Computer VisionMunich, GermanySeptember 2018, 323-338
- 6 articleJan Van Eyck's Perspectival System Elucidated Through Computer Vision.Proceedings of the ACM on Computer Graphics and Interactive Techniques42July 2021
- 7 articleInertial Alternating Generalized Forward-Backward Splitting for Image Colorization.Journal of Mathematical Imaging and Vision615February 2019, 672–690
- 8 inproceedings3D-Aware Ellipse Prediction for Object-Based Camera Pose Estimation.3DV 2020 - International Virtual Conference on 3D VisionFukuoka / Virtual, JapanNovember 2020
10.2 Publications of the year
International journals
International peer-reviewed conferences
National peer-reviewed Conferences
Conferences without proceedings
Scientific book chapters
Doctoral dissertations and habilitation theses