Keywords
Computer Science and Digital Science
- A5.3. Image processing and analysis
- A5.4. Computer vision
- A5.4.1. Object recognition
- A5.4.5. Object tracking and motion analysis
- A5.4.6. Object localization
- A5.6. Virtual reality, augmented reality
- A5.10.2. Perception
Other Research Topics and Application Domains
- B2.6. Biological and medical imaging
- B5.9. Industrial maintenance
- B9.5.3. Physics
1 Team members, visitors, external collaborators
Research Scientists
- Marie-Odile Berger [Team leader, Inria, Senior Researcher, HDR]
- Erwan Kerrien [Inria, Researcher, HDR]
Faculty Members
- Fabien Pierre [Univ de Lorraine, Associate Professor]
- Gilles Simon [Univ de Lorraine, Associate Professor, HDR]
- Frédéric Sur [Univ de Lorraine, Professor, HDR]
- Pierre-Frédéric Villard [Univ de Lorraine, Associate Professor]
- Brigitte Wrobel-Dautcourt [Univ de Lorraine, Associate Professor]
PhD Students
- Youssef Assis [Univ de Lorraine]
- Nathan Boulangeot [Univ de Lorraine, from Oct 2021]
- Abdelkarim Elassam [Inria]
- Radhouane Jilani [Inria, from Oct 2021]
- Nariman Khaledian [Inria]
- Daryna Panicheva [Inria, until Mar 2021]
- Matthieu Zins [Inria]
Technical Staff
- Romain Boisseau [Inria, Engineer]
Interns and Apprentices
- Idriss Abdallah [Inria, from Jun 2021 until Aug 2021]
- Nathan Boulangeot [Univ de Lorraine, from Feb 2021 until Jun 2021]
- Yasmine Djebiret [Inria, from Apr 2021 until Sep 2021]
- Antoine Martel [École Nationale Supérieure des Mines de Nancy, from Jul 2021 until Sep 2021]
- Pierre Alexandre Simon [Univ de Lorraine, from Feb 2021 until Jul 2021]
Administrative Assistant
- Isabelle Blanchard [Inria]
2 Overall objectives
Visual registration is a research topic with a rich history in computer vision. Though plethora of methods have been developed and can be used for general situations, there are still many open problems which originate in the nature of the scene (poorly textured or specular environments), in the type of motion undergone by the object (tiny motions which hardly emerge from the noise floor, or in contrast, highly deformable objects) and in dissimilarities which may occur in the scene between the time the modeling stage occurs and the application time.
Registration is in practice tightly linked to the choice of the model which represents the scene and the desirable physical properties of the objects. Handling complex –possibly dynamic– scenes thus requires a tradeoff between physical realism of the model, convergence issues and robustness of the registration or tracking tasks.
Recent years have seen a surge in research at the intersection of image and deep learning which has impacted many topics of computer vision. While continuing to explore modeling and registering with traditional approaches derived from signal processing, geometry and robust estimation, one of the aims of the team is to integrate machine learning methods, either as end-to-end methods or as components, into these 2D or 3D geometric tasks.
Targeted trans-disciplinary applications are mixed and augmented reality, computational photomechanics and minimally invasive medical interventions.
3 Research program
3.1 Localization and geometric reasoning with high level features
Our goal is to push forward vision-based scene understanding and localization through the joint use of learning-based methods with geometrical reasoning. Our hypothesis is that the use of intermediate representations instead or in addition to the classical point feature will lead to increased capacity in terms of scale and robustness to changing conditions. These intermediate representations can be concrete objects which are recognized and used directly in the global pose computation, in the continuity of our works on ellipsoid modeling of objects, or conceptual objects such as vanishing points (VP) or horizon lines that are of specific interest both for localization and modeling of urban or industrial scenes.
A first goal is to improve our method for localization from sets of ellipse/ellipsoid correspondences 1. Besides the need to have more accurate prediction of ellipses, another objective is to elaborate robust strategies and associated numerical schemes for refining the initial pose from a set of objects. This requires to develop appropriate metrics for characterizing good reprojection of 3D objects onto 2D ones and study their impact on minimization issues in localization. Another important line of research will aim at defining strategies to integrate into the localization procedure various features such as points, objects and VPs, which each bring information at different levels. We especially want to investigate how predictive uncertainty and explainability mechanisms can be used to select and weight these various features in the estimation process.
3.2 Building dedicated models
In this line of research, our goal is to build physically coherent models with a good accuracy vs. efficiency compromise despite the interactive time constraint set in some targeted applications. Though general purpose solutions exist for building models, such techniques are still greatly challenged in more complex cases when specific constraints on the shape or its deformation must be met. This is especially the case in medical imaging of thin deformable organs, such as the diaphragm, the mitral valve or blood vessels, but also for classical scene modeling where constraints, such as ellipsoidal abstraction of objects, must be introduced. The use of mechanical models have become increasingly important in the team activities in medical imaging, especially for handling organs with large deformations. Our goal is to push forward the development of such models with image-guided procedures or predictive simulation in view.
Facing difficulties of meshing complex geometries, especially thin ones, we want to promote mesh free methods such as implicit models. In the continuity of past works 3, automatic adaptation of node locations and sizes to the image will be investigated to improve compactness, and computational efficiency of implicit models. As the fidelity of a mechanical model is often impaired by approximations required to solve its dynamical system equations at interactive frame rates, a second goal is to take advantage of our implicit models to improve contact and deformation resolution.
The second axis is about investigating shape-aware methods either for shape segmentation or shape recognition, in order to be able to enforce global shape constraints or geometric shape priors in the output of CNNs. This topic is still addressed in the team in the context of localization from 3D ellipsoidal abstraction of objects 7. Two applications are especially targeted: (i) We aim at improving the detection of pathologies (e.g. brain aneurysms that are mostly located at vessel bifurcations), through adapted and guided sampling of input data during training, as well as through mechanisms inspired by visual attention modules (ii) In the context of fluid structure simulation methods for patient-based mitral valve simulation, it often appears that the geometric segmented model leads to divergence of the numerical scheme. Our intention is to identify geometric conditions under which simulation works well with the idea to incorporate them in the segmentation process.
3.3 Estimation and inverse problems
Most aforementioned tasks lead to image-based inverse, possibly ill-posed, problems. While some of them can be solved with well-established estimation techniques, others will necessitate the design of new strategies. In this perspective, we consider in this research axis several fundamental aspects of estimation, common to our problems, such as sampling methods,traditional optimization methods, or end-to-end learning methods for pose estimation.
3.3.1 Optimization, variational calculus and numerical schemes
We are interested in non-convex optimization problems, especially those raised by variational calculus. While the convergence of numerical schemes is well established for convex problems, this is not always the case for non-convex functionals. Our aim is to continue the work already carried out in the biconvex framework 6, and extend it to primal-dual algorithms. We especially want to address energy minimization problems where the energy is convex with respect to each variable, but non-convex with respect to the pair of variables.
Another research topic is to investigate new neural architectures adapted to non-Euclidean data, and also to plug variational methods into deep learning approaches to regularize the results. The obtained theoretical results will be applied to image colorization, with the idea to reduce artefacts caused both by a lack of regularization and by the non-Euclidean structure of color information as perceived by the human visual system.
3.3.2 Machine learning for physical problems
We aim at continuing our efforts towards supervised and unsupervised learning for estimation problems. Concerning supervised learning, we intend to investigate further the opportunities offered by neural network estimation of displacement and strain fields in experimental mechanics that we have recently introduced with colleagues in mechanics and signal processing 8. Besides, we also aim at developing unsupervised learning in problems where a quantity has to be estimated over a spatio-temporal domain, which is a recent trend in several application domains. Neural networks are indeed universal approximators whose derivative can be exactly computed with the backpropagation algorithm, which is supposed to make them robust to acquisition noise.
4 Application domains
Applications on which our program is expected to have an impact are mixed reality, computational photomechanics and minimally invasive medical interventions. These fields correspond to areas where we have established trans-disciplinary collaborations with academic or industrial experts of the applicative fields. Common to these applications are the need for finely characterizing the acquisition context of vision-based applications and the need for accurate registration procedures. Another common point is the availability of a limited number of data for characterizing the variability of the observed phenomena.
Mixed reality
Being able to perform reliable and accurate registration under large viewpoint variations, seasonal or lighting changes opens the way towards challenging mixed reality applications. Urban AR and industrial maintenance in large and cluttered environments are examples of application fields that would successfully capitalize on more robust localization solutions. Improved robustness of camera localization is especially expected for poorly textured, specular environments and in the presence of repeated patterns that are common in industrial contexts
Photomechanics
Photomechanics is the field of experimental mechanics which is dedicated to mechanical measurement from images. In particular, we are interested in contactless image-based methods for extensometry, that is the estimation of displacement and strain fields on the surface of materials subjected to different types of mechanical loads. Full-field extensometry is a challenging task since strains often have tiny values and result in gray level changes at the limit of the sensor noise floor. The economic stakes are high and concern for example the automotive and aeronautics industries, or civil engineering. In order for these methods to be adopted by industry, it is, however, necessary to quantify their metrological performance, which is limited by the registration process or by the image acquisition chain, and especially by sensor noise. This topic is the subject of a long-term transdisciplinary collaboration with Institut Pascal (Clermont- Ferrand Université).
Minimally invasive medical interventions
The trend towards the design and performance of minimally invasive procedures will increase in the near future. But the benefit for the patient is at the expense of the surgeon who can only sense the surgical scene through intra-operative imaging. Commercial solutions now exist to teach this increasingly difficult surgical gesture with interactive simulation technologies. However, challenges remain to fill the gap between the learning environment, where qualitative correctness of the setup is sufficient, and the surgical theater, where accuracy and predictability are required. In this context, we aim at addressing the key problem of modeling the geometry and dynamics of deformable organs and surgical devices, in order to make progress towards a faithful 3D rendition of the surgical scene. To circumscribe practical and experimental difficulties, three specific applications will be addressed with our clinical partners: intra-operative guidance in interventional neuroradiology, augmented reality for laparoscopic liver surgery, and simulation of the mitral valve behaviour.
5 Highlights of the year
Gilles Simon’s article, Jan van Eyck’s perspectival system elucidated through computer vision 13, was presented at SIGGRAPH 2021.
In art history, it is commonly accepted that the Flemish painter Jan Van Eyck knew nothing of the laws of perspective. Gilles Simon analyzed five paintings of the artist between 1432 and 1439 and revealed that the painter was in fact far ahead of his time thanks to computer vision methods. He used an advanced perspective machine with two degrees of freedom to represent space as closely as possible to human vision. This surprising discovery solves a major enigma of art history and means Jan Van Eyck is the pioneer of techniques such as augmented reality or synthetic holography.
6 New software and platforms
6.1 New software
6.1.1 >V<
-
Keywords:
Vanishing points, Image rectification
-
Scientific Description:
In most urban, indoor or industrial scenes, man-made structures are often placed at eye level, facilitating the detection of the horizon line by statistical analysis. More specifically, >V< is based on the a-contrario framework, a mathematical translation of the Gestalt psychology. This framework does not require annotating thousands of images as when using a deep learning method. It follows that our method is not scene-specific and can treat any kind of man-made environment. Moreover, by fractioning the 2-D search of meaningful VPs into three 1-D searches of meaningful events (zenith line, horizon line and VPs), we avoid computationally expensive processes encountered using some previous a-contrario approaches. In 2018, our algorithm reaches top performance in accuracy of the horizon line, on both York Urban and Eurasian Cities datasets. >V< also obtains less missed detections and spurious VPs than with the previous state-of-the-art methods, as we show in our paper.
-
Functional Description:
>V< is a Matlab implementation of the a-contrario method published at ECCV'2018 4. It allows detecting the zenith (vertical vanishing point) and all horizontal vanishing points in uncalibrated images of man made environments (urban, indoor, industrial, …). In addition, >V< can automatically associate a Manhattan frame to the scene, that is three particular vanishing points whose directions are pairwise orthogonal and aligned with some box structures of the scene, e.g. the buildings. This allows getting e.g. the camera focal length and/or the orientation of the camera with regard to these structures. It can also help reconstructing the scene from image analysis. Finally, we added some code to warp (rectify) an image so that all the vertical planes present in this image appear as in a frontal view.
- URL:
-
Contact:
Gilles Simon
6.1.2 DeepEllPose
-
Name:
Deep Ellipses Pose
-
Keywords:
Pose estimation, Visual localization, Ellipses, Neural networks
-
Functional Description:
DeepEllPose contains the training and inference code of the object detection and ellipse prediction networks (PyTorch). This code corresponds to an extended implementation of the article: 3D-Aware Ellipse Prediction for Object-Based Camera Pose Estimation. The 7-Scenes dataset is used as example, but the provided tools make it easily applicable to other datasets. The library also contains the code for camera pose estimation from 3 pairs ellipse-ellipsoid, as well as tools to easily manipulate and visualize such objects.
- URL:
-
Contact:
Matthieu Zins
6.1.3 OBL
-
Name:
Object-based localization
-
Keywords:
Localization, Computer vision, Object detection, 3D modeling
-
Functional Description:
The library EllCV, written in C, aims to perform pose computation, and reconstruction using ellipse (2D) - ellipsoid (3D) correspondences.
A client-server-service model has also been set up to enable remote access to these applications. Services are launched by the server in the form of dedicated docker containers running on a host machine.
-
Contact:
Gilles Simon
7 New results
7.1 Visual localization
Participants: Marie-Odile Berger, Romain Boisseau, Gilles Simon, Matthieu Zins.
Recent years have seen the emergence of very effective ConvNet-based object detectors that have reconfigured the computer vision landscape. As a consequence, new approaches that propose object-based reasoning to solve traditional problems, such as camera pose or SLAM estimation, have appeared. Model-image registration of a building’s facade based on dense semantic segmentation is a first example of our research in this field. In 9, registration and semantic segmentation are jointly refined in an Expectation-Maximization framework. We especially introduced a Bayesian model that uses prior semantic segmentation as well as geometric structure of the facade reference modeled by Generalized Gaussian Mixtures. We showed the advantages of our method in terms of robustness to clutter and change of illumination on urban images from various databases.
Another important research axis of the team is object-based visual localization. Recently, we have proposed a novel object-based pose estimation algorithm from monocular images that is based on an approximate 3D modelling of objects as ellipsoids and 2D object detection as ellipses 1. This past work allowed us to compute a rough pose estimate from a minimum number of ellipse/ellipsoid correspondences. In the context of M. Zins's PhD thesis, we first proposed this year in 20 a 3D-Aware method for predicting more accurate elliptic approximations of objects, thus leading to improved pose estimations. We then addressed the problem of refining the rough pose estimate, computed from the best subset of 2 or 3 ellipse/ellipsoids correspondences, by using several correspondences. Contrary to point correspondences, the definition of a cost function characterizing the adequation of the projection of a 3D object onto a 2D object is not straightforward. We first studied various metrics that are of common use to quantify the matching of ellipses (e.g. IoU, Bbox corners, distance to contour) and evaluated their performance in the context of camera pose refinement. We also proposed a metric based on level sets that outperforms existing metrics. Finally, we showed that the use of a predictive uncertainty on the detected ellipses allows a fair weighting of the contribution of the correspondences which noticeably improves the computed pose.
7.2 Vanishing point computation and applications
Participants: Marie-Odile Berger, Abdelkarim Elassam, Gilles Simon.
7.2.1 Vanishing point detection by deep learning
Automation of vanishing point detection classically involves line segments extraction and the search for their meeting points. We have shown in previous work 4 that the a contrario methodology is the most suitable to detect meeting points that cannot be the result of chance. It remains that some configurations of line segments, which are indeed non-hazardous (e.g. star configurations), do not correspond to vanishing points. For this reason, we are interested in detecting vanishing points by deep learning to be able to exploit the image at a higher semantic level than that of line segments. The main difficulty we had to face is that the expected number of vanishing points is not known in advance. This is why the few researchers who tried to detect them with a CNN only aimed at detecting the dominant vanishing point, or vanishing points constrained to a grid inside the image, even though most are outside it.
In the context of A. Elassam's PhD thesis, our solution 18 was to transform the regression problem into three classification ones concerning respectively the offset of the horizon line, its slope and the position of the vanishing points – warped to a bounded domain – along the horizon line. In each case, the class scores are considered to describe a probability distribution whose peaks – detected using an a contrario approach – correspond to the result values. This method proved to be competitive on three reference datasets, and particularly on the HLW dataset which is the most problematic for methods based on line segments. It is finally the first method based on deep learning that can detect any number of vanishing points along the horizon line.
7.2.2 Vanishing point detection in paintings
Detecting vanishing points in a painting can help to date it (perspective was understood differently at different periods) or even to attribute it (the way of representing space contributes to the style of the painter). But a painting is not a photograph: vanishing points are sometimes represented by very few lines, and the available ones are often inaccurate. Our a contrario vanishing point detection method has therefore been adapted to take into account a small number of segments and an uncertainty on their endpoints. Applied to several paintings by the Flemish painter Jan van Eyck (c. 1390-1441), it unveiled a singular pattern, comprising two to four equidistant principal points aligned along a slightly inclined vertical axis. This authentic hidden signature of Jan van Eyck has made it possible to confirm the still uncertain attribution of two paintings by the Master.
More importantly for art history, it reveals that Jan van Eyck used an optical device (a so-called perspective machine) to represent space: he captured reality through a glass window with transferable carbon ink, from several points of view similar to vanishing points, i.e. equidistant and aligned on an inclined vertical axis. Projective geometry could be used to characterize more precisely the geometry of this device as well as that of the bedroom represented in the Arnolfini portrait. This discovery is of great importance for the history of art, insofar as it is the oldest perspective machine we know, and it was believed until today that the Flemish painters had no knowledge of perspective, at the time when the Italians were inventing it. The method and discovery were published at SIGGRAPH 2021 13.
7.3 Handling non rigid deformation
Participants: Marie-Odile Berger, Nariman Khaledian, Darina Panicheva, Pierre-Frédéric Villard.
7.3.1 Individual mitral valve modeling
In the context of the Inria associated team CURATIVE, we have continued our work on modeling the mitral valve (MV).
Within Daryna Panicheva's PhD 22 we focused on extracting one of the valve components: the chordae. First, a segmentation was obtained with a method based on the topological properties of the structures. Then, a representation of the geometry in a form of trees of connected line segments was extracted using a tracking approach combined with local fitting of a cylindrical model. This year we have specifically developed a method to optimize the chordae architecture in order to reduce the slack in the chordae and to correct false topological configurations by taking into account mechanical and anatomical considerations associated with image data. We have also proposed a validation metric for the segmentation results based on graph similarity 10.
In order to have more realistic valve simulations than those obtained with structural simulations, the goal of N. Khaledian's PhD thesis is to develop fluid-structure interaction simulations (FSI). This year, we first identified important factors influencing convergence issues. We then proposed a model based on immersed boundary methods that captures a map of contacts and allows for perfect closure of the mitral valve. Simulation results do not have orifice holes, which often appear with existing methods. Experiments showed that our method is able to reproduce leakage, bulging, and healthy MV.
7.3.2 Image-based biomechanical simulation of the diaphragm during mechanical ventilation
In the context of the collaboration with Uppsala university within the INVIVE project, we have continued our work on modeling the diaphragm behavior and especially the thin 3D geometry of the diaphragm using RBF formulation 21. This year, an adaptive refinement has been added, based on local curvature. We have also evaluated the resulting reconstruction in terms of residual error with respect to initial data, local curvature, and visual appearance. New geometries were included in our test database: the Stanford Bunny, the Asian Dragon as well as various hollow cube geometries containing different webbings.
7.4 Interventional radiology
Participants: Youssef Assis, Marie-Odile Berger, Yasmine Djebiret, Radhouane Jilani, Erwan Kerrien, Pierre-Frédéric Villard.
Towards our long term objective to improve intra-operative guidance of interventional radiology interventions, the planning phase is addressed as a safer, though similar, research environment. Three key problems were addressed this year: detection of the pathology, identification of pathways, simulation of the navigation, with applications to the three main cerebral vascular pathologies: aneurysms, arteriovenous malformations (AVM) and ischemic strokes.
7.4.1 Detection of brain aneurysms using deep learning
Detecting and locating the pathology is the first step. Aneurysms are small bulges of a few millimeters in the blood vessel walls. They are particularly hard to identity in pre-operative images such as MRI (Magnetic Resonance Imaging). Several works recently considered the deep learning approach, most as variants of the U-net architecture, with promising results but with incremental differences in performance. Youssef Assis's PhD thesis fits into this context, and focused on the data preparation phase with is often overlooked. We have proposed an efficient strategy based on small patches, sample selection and sample synthesis. It was shown to significantly improve the performance of U-net networks 15, 17.
7.4.2 Clinical evaluation of registration and segmentation software
Identifying blood vessels to navigate to the pathology is the second step. AVMs present as a tangle of intertwined tiny vessels. Finding which vessels feed the AVM is a very challenging study case. During her Masters' internship, Yasmine Djebiret designed and proceeded with a clinical evaluation of a vessel segmentation software. She also designed methods to determine the operating zone and conditions of a registration software, and evaluated the interest of using pre-operative MRI images as a complement to intra-operative images. This work was done in collaboration with GE Healthcare and CHRU Nancy.
7.4.3 Predictive simulation of catheter navigation
A wrong choice of catheter or a complex vessel geometry can make the navigation difficult or impossible. This can be resolved in the planning phase with the help of simulation on the condition that the simulation is able to predict the above difficulties. This highly depends on contacts between the device and the vessels. Predictive simulation of the navigation for the treatment of ischemic strokes is one of the main goals of the PreSPIN project. Radhouane Jilani started his PhD at the end of the year, with the aim to improve interactive contact management. The approach is a joint effort on both geometric and physical models of the vessels and the catheter.
7.5 Image and signal processing
Participants: Fabien Pierre, Frédéric Sur.
7.5.1 Computational photomechanics
An important task in photomechanics is to measure displacement and strain fields on the surface of a material specimen subjected to a mechanical load. To achieve this goal, the surface of the specimen is often marked with a contrasting pattern, and images taken before and after deformation are numerically processed thereafter. This year's results concern three different processing methods. First, we have characterized experimentally the influence of the sampling density when specimen are marked with a periodic pattern. For a given sampling density, we have demonstrated that a specific pattern, namely a checkerboard, gives the lowest noise level 12. Second, our contributions also concern digital image correlation (DIC) where images are marked with a random speckle pattern. In this case, the estimated displacement is shown to be impaired by biases related to the interpolation scheme needed to reach subpixel accuracy, the image gradient distribution, sensor noise, as well as the difference between the hypothesized parametric transformation and the true displacement. We have demonstrated the effect of the so-called pattern-induced bias, which is the generalization to DIC problems of the so-called fattening effect which was previously known in stereoscopy 14. Third, we have introduced one of the very first deep convolutional neural network (CNN) to estimate displacement fields from pairs of reference and deformed images of a flat speckled surface, as DIC does. Such a CNN achieves competing results in terms of metrological performance and computing time, and offers a viable alternative to DIC, especially for real-time applications 8.
7.5.2 Variational methods for image processing
We have proposed a detailed analysis and implementation of active contours (also known as snakes). Most of the on-line codes for 2D/3D segmentation, as well as built-in Matlab toolboxes are based on level-set methods. Moreover, in the literature, the implementation details of active contours methods with meshes in three dimensions are tight, making any reproduction of these techniques tedious. In 11, we give some details of the implementation of active contours in 2D/3D with meshes, especially about the choice of the use of a 2D/3D mesh and its refinement. We also explore the choice of the parameters with a quantitative study of their influence on the segmentation results. The 3D segmentation method has been applied to CT scan images of the lungs.
7.6 Application of deep learning to image and signal processing
Participants: Nathan Boulangeot, Fabien Pierre, Pierre Alexandre Simon, Frédéric Sur.
7.6.1 Restauration of old movies
In collaboration with Institut de Recherche en Informatique de Toulouse, we have proposed to detect defects in old movies 19, as the first step of a larger framework of old movies restoration by inpainting techniques developed during the PhD thesis of Arthur Renaudeau. In order to have minimal human interaction and further reduce the time spent for a restoration, we feed a U-Net with consecutive defective frames as input to detect unexpected variations of pixel intensity over space and time. We have created the dataset of mask frames on the basis of restored frames from the software used by the film restorer, instead of classical synthetic ground truth. Our network succeeds in automatically detecting real defects with more precision than the manual selection with an all-encompassing shape, including some the expert restorer could have missed for lack of time.
7.6.2 An application of neural point processes to geophysical data
In a collaboration with Radu Stoica (Pasta Inria project-team), we have supervised the internship of Pierre-Alexandre Simon on the topic of neural point processes. Temporal point processes are a popular approach to model time series. Point processes usually need strong assumptions on the conditional intensity which is often supposed to follow a particular parametric function, hence fixing a priori the structure of the events distribution. Recent papers investigate the use of models from machine learning dedicated to such sequential events analysis, namely recurrent neural networks (RNN). These RNNs are expected to be versatile enough to automatically adapt to the data, without the need for a priori choosing the events distribution. We have presented results on real seismic data at the RING symposium conference 16.
7.6.3 Extensive searches for complex intermetallic catalysts
Materials science is at the heart of many technological revolutions. The search for new materials is crucial to meet the challenges posed by climate change, or the growing global demand for energy and consumer goods. In this context, the discovery of new catalytic materials is a major challenge to improve the performance of industrial processes, for example in the context of the hydrogen economy, in order to provide energy sources with limited environmental impact. Although some theoretical models exist, catalyst research is still largely empirical and based on a trial-and-error approach. This is probably because the application of theoretical models requires particularly expensive calculations based on density functional theory (DFT). Recently, the materials community has been seizing on statistical learning tools to limit the cost of calculations and to accelerate the discovery of new catalysts. Together with Emilie Gaudry at Institut Jean-Lamour (IJL) at Nancy, we have developed a collaboration on this subject since September 2021 through Nathan Boulangeot's PhD thesis. The initial work concerns a faithful description of molecule environment on the surface of catalysts with a complex structure and the prediction of their interactions using statistical learning models based on these descriptions (kernel regression, artificial neural networks, etc).
8 Bilateral contracts and grants with industry
8.1 Bilateral contracts with industry
GE Healthcare
-
Title:
Fusion multi-modalité en Neuroradiologie Interventionnelle
-
Participants:
M.-O. Berger, Y. Djebiret, E. Kerrien
-
Date/Duration:
mars 2021-sept 2021
-
Additionnal info:
In collaboration with GE Healthcare and CHRU Nancy (Pr Anxionnat), a first objective of this contract was the evaluation of a vessel segmentation software in order to identify which vessels feed arterio veinous brain malformations. Another objective was the evaluation of the interest of using pre-operative MRI images as a complement to intra-operative images.
9 Partnerships and cooperations
9.1 International initiatives
9.1.1 Inria associate team not involved in an IIL or an international program
CURATIVE
- Title: CompUteR-based simulAtion Tool for mItral Valve rEpair
- International Partner (Institution - Laboratory - Researcher):
- Harvard University (United States) - Harvard Biorobotics Lab (HBL)- Robert Howe
- Coordinator: Pierre-Frédéric Villard
- Participants: M.-O. Berger, N. Khaledian, D. Panicheva, P.-F. Villard
- Start year: 2020-2022
- See also: https://team.inria.fr/curative/
-
The mitral valve of the heart ensures one-way flow of oxygenated blood from the left atrium to the left ventricle. However, many pathologies damage the valve anatomy producing undesired backflow, or regurgitation, decreasing cardiac efficiency and potentially leading to heart failure if left untreated. Such cases could be treated by surgical repair of the valve. However, it is technically difficult and outcomes are highly dependent upon the experience of the surgeon.
One way to facilitate the repair is to simulate the mechanical behavior of the pathological valve with subject-specific data. Our main goal is to provide surgeons with a tool to study solutions of mitral valve repairs. This tool would be a computer-based model that can simulate a potential surgical repair procedure in order to evaluate its success. The surgeons would be able to customize the simulation to a patient and to a technique of valve repair. Our methodology will be to realistically simulate valve closure based on segmentation methods faithful enough to capture subject-specific anatomy and based on a biomechanical model that can accurately model the range of properties exhibited by pathological valves.
9.2 International research visitors
9.2.1 Visits to international teams
Due to covid, planned visits to the bio-robotic lab (Boston) and to Uppsala University were canceled.
9.3 European initiatives
9.3.1 Other european programs/initiatives
MOVEON
-
Title:
Towards robust spatial scene understanding in dynamic environments using intermediate representations
-
Partner Institution(s):
DFKI Kaiserslautern
-
Participants:
M.-O. Berger, R. Boisseau, A. Elassam, G. Simon
-
Date/Duration:
sept 2020-sept 2023
-
Additionnal info:
The aim of the MOVEON project is to push forward the state of the art in vision-based, spatio-temporal scene understanding by merging novel machine-learning approaches with geometrical reasoning. Deep-learning-based recognition and understanding of high-level concepts such as vanishing points or large object classes will serve as unitary building blocks for a spatio-temporal localization and environment reconstruction that will use geometric reasoning as underlying support. This research will lead to a novel generation of visual positioning systems that go beyond classical localization and mapping, which focuses currently only on point cloud reconstruction. In contrast, our aim is to allow for 6DoF positioning and global scene understanding in wild and dynamic environments (e.g. crowded streets) that scales up nicely with the size of the environment, and that can be used persistently over time by reusing consistent maps.
INVIVE
-
Title:
Biomechanical simulation of the respiratory muscles
-
Partner Institution(s):
Uppsala University, Karolinska Institute
-
Participants:
P.-F Villard
-
Date/Duration:
2021-2024
-
Additionnal info/keywords:
Pierre-Frédéric Villard is currently involved in the INVIVE project funded by the Swedish Research Council and realized within a collaboration with Uppsala University and Karolinska Institute. Within this project, he is the co-supervisor of PhD candidates Igor Tominec and Andreas Michael (with Elisabeth Larsson (Uppsala University) as the main advisor. The aim of the collaboration is to develop a framework to compute mechanical deformations of thin objects using Radial-Based Functions (RBF). A first step is to handle small displacements with linear elasticity (Igor Tominec's Ph.D.). A parallel study has started in September 2021 with Andreas Michael's Ph.D and aims of solving both large deformation and hyperelastic material still within an RBF formulation. The application of this research is modeling the diaphragm behavior when a mechanical ventilator is used on a patient.
9.4 National initiatives
ANR JCJC ICaRes
-
Title:
Image correlation for the accurate measurement of residual stress
-
Partner Institution(s):
Université Clermont-Auvergne
-
Participants:
F. Sur
-
Date/Duration:
2019-2022
-
Additionnal info/keywords:
This 3-year project (2019-2022) headed by B. Blaysat (Université Clermont-Auvergne), is supported by the Agence Nationale de la Recherche. It addresses residual stresses, which are introduced in the bulk of materials during processing or manufacturing. Since unintended residual stresses often initiate early failure, it is of utmost importance to correctly measure them. The goal of the ICaRes project is to improve the performance of residual stress estimation through the so-called virtual digital image correlation (DIC) which will be developed. The basic idea of virtual DIC is to mark the specimen with virtual images coming from a controlled continuous image model, instead of the standard random pattern. Virtual DIC is expected to outperform standard DIC by, first, matching real images of the materials with the virtual images, then, to run DIC on the virtual images on which strain fields are estimated, giving ultimately residual stresses.
ANR PRC PreSPIN
-
Title:
Simulation prédictive pour la planification en neuroradiologie interventionnelle
-
Partner Institution(s):
CReSTIC (Reims), Creatis (Lyon) and CIC-IT/CHRU Nancy
-
Participants:
R. Jilani, E. Kerrien, P.-F Villard.
-
Date/Duration:
2020-2024
-
Additionnal info/keywords:
This 4-year project is coordinated by E. Kerrien. It aims at improving the planning phase in the therapeutic management of cerebral ischemic strokes thanks to predictive simulation of both the therapeutic interventional gesture and post-interventional perfusion images. The consortium is set to address the challenges of geometrical and topological modeling of the full brain vasculature; physics-based simulation of interventional devices; simulation of MRI perfusion images; and clinical validation.
PEPS CNRS-INSIS Jauge2.0
-
Title:
Towards image-based strain jauge
-
Partner Institution(s):
Université Clermont-Auvergne
-
Participants:
F. Sur
-
Date/Duration:
2021
-
Additionnal info/keywords:
This one-year project aims at designing a proof-of-concept smart camera incorporating a lightweight convolutional neural network to measure displacement and strain fields for experimental mechanics applications.
10 Dissemination
10.1 Promoting scientific activities
10.1.1 Scientific events: selection
Member of the conference program committees
- Marie-Odile Berger and Gilles Simon were members of the International Program Committee (IPC) of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2021.
- Fabien Pierre was a s member of Program Committee of the ICPR satellite Workshop on Reproducible Research in Pattern Recognition
Reviewer
- Marie-Odile Berger was a reviewer for ISMAR (International Symposium for Mixed and Augmented Reality), IPCAI (International Conference on Information Processing in Computer-Assisted Interventions), ORASIS 2021 and for the ICCV satellite workshop TradiCV.
- Pierre-Frederic Villard was a reviewer for the Eurographics Workshop on Visual Computing for Biology and Medicine and the International Conference on Computer Graphics, Visualization, Computer Vision And Image Processing.
- Gilles Simon was a reviewer for ORASIS 2021.
- Erwan Kerrien was a reviewer for MICCAI (International Conference on Medical Image Computing and Computer Assisted Intervention), MIDL (Medical Imaging and Deep Learning), ISMAR and ORASIS 2021.
10.1.2 Journal
Reviewer - reviewing activities
- Marie-Odile Berger and Pierre-Frédéric Villard were reviewers for the International Journal of Computer Assisted Radiology and Surgery
- Frédéric Sur was a reviewer for Measurement: Journal of the International Measurement Confederation, and Signal Processing: Image Communication.
- Erwan Kerrien was a reviewer for IEEE Transactions on Medical Imaging, IEEE Transactions on Biomedical Imaging, and the International Journal of Computer Assisted Radiology and Surgery.
- Fabien Pierre was reviewer of SIAM Journal on Imaging Sciences and MDPI journal of imaging.
10.1.3 Invited talks
- Pierre-Frédéric Villard did a talk during the Journée Thématique de la GE@2M “Numerical Simulation in Mechanics” on July 6th 2021. Title: “Simulation of fluid/structure interaction with large deformations and contact in the context of the mitral valve”
- Pierre-Frédéric Villard did various talks to the Harvard Biorobotics Lab within the CURATIVE collaboration. The list of the talks is there.
10.1.4 Leadership within the scientific community
- Marie-Odile Berger is the president of the Association française pour la reconnaissance et l’interprétation des formes (AFRIF).
10.1.5 Scientific expertise
- Marie-Odile Berger was a member of the recruitment committee for a Professor position at Université Savoie Mont-Blanc and for an assistant professor position at Université de Strasbourg
- Frédéric Sur was a member of the recruitment committee for an associate professor position at Université de Strasbourg (IUT d'Haguenau).
- Brigitte Wrobel-Dautcourt was a member of the recruitment committee for an associate professor position at Université de Lorraine (FST).
10.1.6 Research administration
- Gilles Simon is an elected member of the CNU (Conseil National des Universités).
10.2 Teaching - Supervision - Juries
10.2.1 Teaching
The assistant professors of the TANGRAM team actively teach at Université de Lorraine with an annual number of around 200 teaching hours in computer sciences, some of them being accomplished in the field of image processing. INRIA researchers have punctual teaching activities in computer vision and shape recognition mainly in the computer science Master of Nancy and in several Engineering Schools near Nancy (ENSMN Nancy, SUPELEC Metz, ENSG). Our goal is to attract Master students with good skills in applied mathematics towards the field of computer vision.
The list of courses given by staff members which are related to image processing and computer vision is detailed below:
- M.-O. Berger
- Master : Shape recognition, 24 h, Université de Lorraine.
- Master : Introduction to image processing, 12 h, Mines Nancy.
- Master : Image processing for Geosciences, 12h, ENSG.
- E. Kerrien
- Master : Introduction to image processing, 15 h, ENSMN Nancy.
- Licence : Initiation au développement, 80h, IUT St Dié-des-Vosges.
- Fabien Pierre
- Master: Introduction à l’apprentissage automatique, 14h, Mines Nancy.
- Master: Vision artificielle et traitement des images, 12h, Polytech Nancy.
- Licence: Introduction au traitement d’image, 30h, IUT Saint-Dié des Vosges.
- Licence: Algorithmique et programmation, 87h, IUT Saint-Dié des Vosges
- Licence: Culture scientifique et traitement de l’information, 69h, IUT Saint-Dié des Vosges
- Licence: Programmation objet et évènementielle, 35h, IUT Saint-Dié des Vosges
- Licence: Initiation à l’intelligence artificielle, 18h, IUT Saint-Dié des Vosges
- G. Simon
- Master: Augmented reality, 24 h, Télécom-Nancy.
- Master: Augmented reality, 24h, M2 Informatique FST
- Master: Visual data modeling, 12h, M1 Informatique FST
- Licence pro: 3D modeling and integration, 40h FST - CESS d’Epinal
- Licence: Programming methodology, L1 informatique, 48h FST
- F. Sur
- Master: Introduction to machine learning, 40 h, Université de Lorraine (Mines Nancy).
- Licence: Javascript programming, 100h, IUT Charlemagne
- P.-F Villard
- Master : Augmented and Virtual Reality, 16h, M2 Cognitive Sciences and Applications, Insti- tut des Sciences du Digital, Université de Lorraine
- Licence: Computer Graphics with webGL, 30h, IUT Saint-Dié des Vosges.
- Licence: Game design with Unity3D, 15h, IUT Saint-Dié des Vosges.
- Licence: Virtual and Augmented Reality in Industrial Maintenance, 2h, Faculty of Science and Technology, Université de Lorraine
- Licence: Web programming, 20h, IUT Saint-Dié des Vosges.
- Licence: Graphical user interface programming, 30h, IUT Saint-Dié des Vosges.
- Licence: Object-oriented programming, 20h, IUT Saint-Dié des Vosges.
- Licence: UML modeling,16h, IUT Saint-Dié des Vosges.
- Licence: Security and life privacy with internet, 2h, IUT Saint-Dié des Vosges.
- Licence: Parallel programming, 18h, IUT Saint-Dié des Vosges.
- Licence: Initiation to machine learning, 24h, IUT Saint-Dié des Vosges.
- B. Wrobel-Dautcourt
- Master: modélisation objet et conception des systèmes d’information, 30h, Télécom
- Master: projet de conception et développement java, 27h, Télécom 2A
- Master: ingéniérie logicielle, 12h, FST
- Licence: bases de la programmation objet, 44h, FST
- Licence: interfaces graphiques,22h, FST
- Licence: projet de synthèse (activité intégratrice), 30 h, FST
- Licence: système, 24h, FST
- Licence: compilation, 16h, FST
10.2.2 Supervision
- PhD defended: Daryna Panicheva, Image-based Biomechanical Simulation of Mitral Valve Closure, March 2021, Marie-Odile Berger, Pierre-Frédéric Villard.
- PhD in progress: Matthieu Zins, Localization in a world of objects, October 2019, Marie-Odile Berger, Gilles Simon.
- PhD in progress: Abdelkarim Elassam, Robust visual localization using high level features, October 2020, Marie-Odile Berger, Gilles Simon.
- PhD in progress: Nariman Khaledian, Toward a Functional Model of the Mitral Valve, October 2020, Marie-Odile Berger, Pierre-Frédéric Villard.
- PhD in progress: Youssef Assis, Deep learning for the automated detection of brain aneurysms, November 2020, Erwan Kerrien, René Anxionnat (CHRU Nancy).
- PhD in progress: Radhouane Jilani, Predictive simulation for interventional neuroradiology, October 2021, Erwan Kerrien, Pierre-Frédéric Villard.
- PhD in progress: Nathan Boulangeot, Extensive searches for complex intermetallic catalysts, October 2021, Émilie Gaudry (Institut Jean-Lamour), Frédéric Sur.
10.2.3 Juries
- Marie-Odile Berger was president of the PhD committee of Tong Yu (University de Strasbourg), Toby Collins (Université Clermont-Auvergne), Raphael Groscot (Université Paris-Dauphine), Farouk Achakir (Université Jules Verne Picardie)
- Fabien Pierre was a member of the PhD committee of Arthur Renaudeau
10.3 Popularization
10.3.1 Internal or external Inria responsibilities
Erwan Kerrien is Chargé de Mission for scientific mediation at Inria Nancy-Grand Est, and thereby is part of the Inria scientific mediation network. As such, he is a member of the steering committee of "la Maison pour la Science de Lorraine", and member of the IREM Lorraine (Institut de Recherche sur l’Enseignement des Mathématiques - Research Institute for Teaching Mathematics) steering council. He is also the local scientific referent for the "Chiche!" initiative.
10.3.2 Articles and contents
- Gilles Simon wrote an article for The Conversation France 25 about Jan van Eyck's perspective system elucidated through computer vision – an English translation is also available on their website. To date it has been read by 40,000 readers. An extended version of this article, addressed mainly to secondary school pupils, was also published in the Inria journal Interstices 24. This work has also been widely publicized, with articles in the magazines Pour la Science and Sciences & Avenir, as well as television reports for France 3 Grand Est and the German public television SR. It was also reported in a radio program on France Culture.
- Erwan Kerrien participated in the updated version of the MOOC "Les métiers du numérique pour inventer le monde de demain", which aims at providing more insight into the reality and variety of digital jobs to help students choose undergraduate studies to pursue after high school. This MOOC is part of the PIA3-funded MOOCFOLIO project.
10.3.3 Education
- Pierre-Frédéric Villard is the scientific godfather of the secondary school of Champigneulles (France) as a "Collège Pilote" of "La Main à la pâte" foundation. He gave a seminar on augmented and virtual realities to the pupils, he helped the teacher with preparing some activities with augmented and virtual reality technologies. Eventually, he is supervising master students to produce teaching applications with augmented reality technologies that will be used in secondary school classes.
- Pierre-Frédéric Villard did a presentation to high school students in the context of the "Chiche!" initiative.
- Erwan Kerrien was an associate researcher to a MATh.en.JEANS workshop within Henri Loritz high school in Nancy. He animated a workshop about "Images and statistics" during the APMEP day (Maths teachers association in public education). He also participates in the "Maths and games" group at IREM Lorraine.
11 Scientific production
11.1 Major publications
- 1 inproceedingsCamera Relocalization with Ellipsoidal Abstraction of Objects.ISMAR 2019 - 18th IEEE International Symposium on Mixed and Augmented RealityBeijing, ChinaIEEEOctober 2019, 19-29
- 2 articleImpact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery.IEEE Transactions on Visualization and Computer Graphics2152015, 584 - 597
- 3 articleBlood vessel modeling for interactive simulation of interventional neuroradiology procedures.Medical Image Analysis35January 2017, 685 - 698
- 4 inproceedingsA-Contrario Horizon-First Vanishing Point Detection Using Second-Order Grouping Laws.ECCV 2018 - European Conference on Computer VisionMunich, GermanySeptember 2018, 323-338
- 5 articleMeasuring the Noise of Digital Imaging Sensors by Stacking Raw Images Affected by Vibrations and Illumination Flickering.SIAM Journal on Imaging Sciences81March 2015, p. 611-643
- 6 articleInertial Alternating Generalized Forward-Backward Splitting for Image Colorization.Journal of Mathematical Imaging and Vision615February 2019, 672–690
- 7 inproceedings3D-Aware Ellipse Prediction for Object-Based Camera Pose Estimation.3DV 2020 - International Virtual Conference on 3D VisionFukuoka / Virtual, JapanNovember 2020
11.2 Publications of the year
International journals
International peer-reviewed conferences
National peer-reviewed Conferences
Conferences without proceedings
Doctoral dissertations and habilitation theses
Reports & preprints
11.3 Other
Scientific popularization