3.1 FAULTS R GEMS: Properties of faults, a key to realistic generic earthquake modeling and hazard simulation

Decades of research on earthquakes have yielded meager prospects for earthquake predictability: we cannot predict the time, location and magnitude of a forthcoming earthquake with sufficient accuracy for immediate societal value. Therefore, the best we can do is to mitigate their impact by anticipating the most “destructive properties” of the largest earthquakes to come: longest extent of rupture zones, largest magnitudes, amplitudes of displacements, accelerations of the ground. This topic has motivated many studies in last decades. Yet, despite these efforts, major discrepancies still remain between available model outputs and natural earthquake behaviors. An important source of discrepancy is related to the incomplete integration of actual geometrical and mechanical properties of earthquake causative faults in existing rupture models. We first aim to document the compliance of rocks in natural permanent damage zones. These data –key to earthquake modeling– are presently lacking. A second objective is to introduce the observed macroscopic fault properties –compliant permanent damage, segmentation, maturity– into 3D dynamic earthquake models. A third objective is to conduct a pilot study aiming at examining the gain of prior fault property and rupture scenario knowledge for Earthquake Early Warning (EEW). This research project is partially funded by the ANR Fault R Gems, whose PI is Prof. I. Manighetti from Geoazur. Two sucessive postdocs (Barham Jafrasteh and Bilel Kanoun) have worked on this research topic funded by UCA-Jedi.

3.2 Probabilistic models on graphs and machine learning in remote sensing applied to natural disaster response

We currently develop novel probabilistic graphical models combined with machine learning in order to manage natural disasters such as earthquakes and flooding. The first model will introduce a semantic component to the graph at the current scale of the hierarchical graph, and will necessitate a new graph probabilistic model. The quad-tree proposed by Ihsen Hedhli in AYIN team in 2016 is no longer fit to resolve this issue. Applications from urban reconstruction or reforestation after natural disasters will be achieved on images from Pleiades optical satellites (provided by the French Space Agency, CNES) and CosmoSKyMed radar satellites (provided by the Italian Space Agency, ASI). This project is conducted in partnership with the University of Genoa (Prof. G.Moser and Prof. S. Serpico) via the co-supervision of a PhD student, financed by the Italian government. The PhD student, Martina Pastorino, has worked with Josiane Zerubia and Gabriele Moser in 2020 during her double Master degree at both University Genoa and IMT Atlantique. She started her PhD in co-supervision between University of Genoa DITEN (Prof. Moser) and Inria (Prof. Zerubia) in November 2020.

3.3 Marked point process models for object detection and tracking in temporal series of high resolution images

The model proposed by Paula Craciun's PhD thesis in 2015 in AYIN team, supposed the speed of tracked objects of interest in a series of satellite images to be quasi-constant between two frames. However this hypothesis is very limiting, particularly when objects of interest are subject to strong and sudden acceleration or deceleration. The model we proposed within AYIN team is then no longer viable. Two solutions will be considered within AYANA team : either a generalist model of marked point processes (MPP), supposing an unknown and variable velocity, or a multi-model of MPPs, simpler with regard to the velocity that can have a very limited amount of values (ie, quasi-constant velocity for each MPP). The whole model will have to be redesigned, and is to be tested with data from a constellation of small satellites, where the objects of interest can be for instance "speed-boats" or "go-fast cars". Some comparaisons with deep learning based methods belonging to Airbus Defense and Space (Airbus DS) will be done. Then this new model should be brought to diverses platforms (ground based or on-board). The modeling and ground-based application part will be studied within a PhD thesis (Jules Mabon), and the on-board version will be developed as part of a postdoctoral project (Camilo Aguilar-Hererra), both being conducted in partnership with Airbus DS, and the company Erems for the onboard hardware. This project is financed by BPI France within the LiChIE contract. This research project has started in 2020 for a duration of 4 years.

5.1 Awards

• Martina Pastorino was finalist for the Prix Laffitte, 6th edition, Ecole Mines Paris-PSL, 2022

6.1 New software

7.1 Semantic segmentation of remote sensing images combining hierarchical probabilistic graphical models and deep convolutional neural networks

Participants: Martina Pastorino, Josiane Zerubia.

External collaborators: Gabriele Moser [Genoa University, DITEN, Professor], Sebastiano Serpico [Genoa University, DITEN, Professor].

Keywords: image processing, stochastic models, deep learning, fully convolutional networks, multiscale analysis.

The work carried out consisted in the development of novel methods based on fully convolutional networks (FCNs) and a hierarchical probabilistic graphical model (PGM) to tackle the semantic labelling of very high resolution (VHR) remote sensing data. The logic was to take advantage of the spatial modeling capabilities of hierarchical PGMs to mitigate the impact of incomplete ground truths (GTs) and obtain accurate classification results in scenarios where spatially exhaustive GTs do not exist, such as real-world remote sensing applications. In fact, deep learning techniques have shown outstanding performances in semantic segmentation tasks, but their accuracy depends on the quantity and quality of GT used to train them. This approach makes use of a quadtree to model the interactions of the pixels, inter-layer and intra-layer and exploit the intrinsically multiscale nature of FCNs. The marginal posterior mode (MPM) criterion for inference was used in the proposed framework. The method was experimentally validated on the ISPRS 2D Semantic Labeling Challenge dataset, which includes aerial images of the cities of Vaihingen and Potsdam in Germany. As these datasets are “ideal": the class label is known for all pixels in the images, the FCN has been further trained with a dataset with several modifications, aimed at approximating the most common GTs found in real remote sensing applications (which are far from being spatially exhaustive). The results are significant, as the proposed methodology has a higher producer accuracy (the map accuracy from the point of view of the map maker) compared to the standard FCNs considered, in particular when the input training data are scarce and approach the real-world GTs available for land-cover mapping applications. The results were published on IEEE TGRS 3.

(a)

 

(b)

 

(c)

 

(d)

 

Figure

Figure

Figure

Figure

The method was further extended with the addition of fully connected neural networks (dense layers) at different convolutional blocks of the FCN, allowing to incorporate information at different spatial resolutions and define an end-to-end architecture. Spatial-contextual information is favored in this framework through a supplementary convolutional layer modeling the interactions between neighboring pixels at the same resolution. The loss function of the proposed model consists of a linear combination of the weighted cross-entropy losses of the fully connected neural networks and of the FCN. The modeling of spatial information is further addressed by the introduction of an additional loss term which allows to integrate spatial information between neighboring pixels. The goal of the proposed method is threefold: firstly, to take advantage of the intrinsic multiscale behavior of FCNs to integrate multiscale information through the addition of fully connected neural networks; secondly, to strengthen the modeling of the spatial information through an additional convolutional layer; finally, to define a loss function capable to consider both this multiscale and spatial information within an end-to-end neural model. The experimental validation was conducted again with the ISPRS 2D Semantic Labeling Challenge data set over the city of Vaihingen, Germany (see Fig.1). This approach obtains higher average classification results than the state-of-the-art techniques considered, especially in the case of scarce, suboptimal GTs. This work was submitted to ICIP’22 9.

Current work involves the definition of an end-to-end neural model for semantic segmentation tasks as a fully convolutional conditional random field (CRF) network, and for the future the idea is to integrate the proposed method with transfer learning and test it with various datasets related to real world applications, such as disaster management.

7.2 Hierarchical probabilistic graphical models and fully convolutional networks for the semantic segmentation of multimodal SAR images

Participants: Martina Pastorino, Josiane Zerubia.

External collaborators: Gabriele Moser [Genoa University, DITEN, Professor], Sebastiano Serpico [Genoa University, DITEN, Professor].

Keywords: image processing, stochastic models, deep learning, fully convolutional networks, multimodal radar images.

The method developed and tested on the optical dataset in 3 was extended to deal with synthetic aperture radar (SAR) images (see Fig. 2, 3), in the framework of a research project funded by the Italian Space Agency (ASI). This SAR images are multimission (SAOCOM and Cosmo-SkyMed) and were collected with different modalities (TOPSAR and StripMap), bands (L and X), polarizations, and several trade-offs between resolution and coverage. The possibility to deal with multimodal images offers great prospects for land cover mapping applications in the field of remote sensing. However, the development of a supervised method for the classification of SAR images – which suffer from speckle and are determined by complex scattering phenomena – presents some issues.

Figure 2

The results obtained by the proposed technique were also compared with those of HRNet 23, a network consisting of multiresolution subnetworks connected in parallel, and of the light-weight attention network (LWN-Attention) 19, a multiscale feature fusion method making use of multiscale information through the concatenation of feature maps associated with different scales. The proposed method, leveraging both hierarchical and long-range information attained generally higher average classification results, thus suggesting that the proposed integration of FCN and hierarchical PGM can be advantageous. The work validated on Cosmo-SkyMed data was accepted for a presentation at IGARSS’22 10.

(a)

 

(b)

 

(c)

 

(d)

 

(e)

 

Figure

Figure

Figure

Figure

Figure

7.3 Extraction of curvilinear structure networks in image data using an innovative deep learning approach

Participants: Bilel Kanoun, Josiane Zerubia.

External collaborators: Isabelle Manighetti [OCA, Géoazur, Senior Physicist].

Keywords: curvilinear feature extraction, machine learning, VHR optical imagery, remote sensing.

Fractures and faults are ubiquitous on Earth and contribute to a number of important processes: small scale fractures control the permeability of rock reservoirs, and larger-scale faults may produce damaging earthquakes. Therefore, identifying and mapping fractures and faults are important in geosciences, especially in earthquake hazard and geological reservoir studies. However, the delineation of fractures and faults is a challenging task due to complexity of the faults. So far, mapping fractures and faults has been done manually in optical images of the Earth surface, yet it is time consuming and it requires an expertise that may not be available. To this purpose, the idea is to explore deep learning approaches for the automatic extraction of fractures and faults. Building upon a recent prior study namely TM-MRef model 20, we developed a deep learning approach, called BK-MRef (presented at ICASSP 2022 5), based on a variant of a U-Net neural network, and applied it to automate fracture and fault mapping in optical images and topographic data. We performed the training of the model with a realistic knowledge of fracture and fault uneven distributions and trends (a “balanced training”), using a loss function, called the focal loss 18 , that “down-weights” the most represented features (not-a-fault) to focus on the less represented but most meaningful features (the fracture/fault pixels). In addition, we proceeded the study with another loss function that operates at both pixel and larger scales through a combined use of weighted binary Cross-entropy (wBCE) 14 and Jaccard loss 15, noted as combined loss $L^{\text{'}\text{'}}$. Such a loss function greatly improves the predictions, both quantitatively (see ROC curves in Figure 4) and qualitatively (see Figure 5). As we applied the model to a site different from those used for training, we have demonstrated it enhanced generalization capacity.

Figure 4

Figure 5

7.4 Combining Stochastic geometry and deep learning for small objects detection in remote sensing datasets

Participants: Jules Mabon, Josiane Zerubia.

External collaborators: Mathias Ortner [Airbus Defense and Space, Senior Data Scientist].

Keywords: object detection, deep learning, fully convolutional networks, stochastic models, energy based models.

Unmanned aerial vehicles and low-orbit satellites, including cubesats, are increasingly used for wide area surveillance, which results in large amounts of multi-source data that have to be processed and analyzed. These sensor platforms capture vast ground areas at frequencies varying from 1 to 60 Hz depending on the sensor. The number of moving objects in such data is typically very high, accounting for up to thousands of objects. On one hand, stochastic geometry has proven to be extremely powerful for capturing object positions within images using a prior model. For instance, it is possible to introduce a regularizing term that accounts for relative positions of objects (see previous research results in both AYIN and ARIANA teams), in order to represent specific patterns. One major drawback of this approach is that the amount of marks that can be handled has to be limited in order to avoid an explosion of the problem dimensionality. Moreover, parameter optimization is more complex when objects have complex shapes. On the other hand, deep learning approaches have largely proven over the recent years to be extremely efficient in building representations that match object signatures. We first tackled the problem of static detection. We propose to merge both approaches and build a stochastic point process model using a data term learned through a convolutional neural network (CNN); more specifically a Unet architecture, allowing for dense predictions on any image size. This approach allows us to leverage the generalization capabilities of neural networks while regularizing the results through prior energies in the point process model. The availability of detection map sourced from the CNN -used for the data term of the point process- allows us to leverage a birth map within the sampling procedure thus reduce the inference time. Our overall goal is to intertwine the sampling and learning procedure as much as feasible, in order to avoid the ad-hoc tweaking of parameters that is often necessary with point process modeling. We perform tests of our CNN and point process joint model on the DOTA dataset and on aerial data provided by Airbus DS (as part of our collaboration within the LiChIE contract funded by BPI France), subsampled to a spatial resolution of 50 cm per pixel. In our approach, we build a model to extract oriented rectangles (position, length, with and angle) that match the vehicles in the images. At first, we set the relative weights of the energies of our model (weighting the relative importance of the data term and the various prior terms), these results are published in 8, 11. We elaborate on that model by introducing a procedure to learn the weights; this method is presented in 7.

In figure 6 we compare our results on DOTA with the CNN method from 27, along with MPP constrast based methods from 17 and 16. The model MPP+CNN${}^{☆}$ corresponds to manual energy weights, while MPP+CNN${}^{†}$ uses learned energy weights. In Figure 7 we compare results of our method against 27 on aerial data provided by Airbus DS.

Ground truthMPP+T-test 17MPP+Gradient 16BBA-Vec. 27MPP+CNN${}^{☆}$MPP+CNN${}^{†}$

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure

Figure 7

7.5 Small Object Detection and Tracking in Satellite Videos with Neural Networks and Random Finite Sets

Participants: Camilo Aguilar, Josiane Zerubia.

External collaborators: Mathias Ortner [Airbus Defense and Space, Senior Data Scientist].

Keywords: object detection, object tracking, convolutional neural networks, Random Finite Sets.

This work presents a novel object detection and tracking approach for satellite videos. Specifically, the project focuses on tracking moving ground vehicles in low-resolution images (1 m/pixel). The main challenge arises from detecting and tracking dense populations of small objects in cluttered scenes. Vehicles span few pixels of width and height, lack distinctive features, and are often surrounded by texture rich backgrounds. We propose a track-by-detection approach to detect and track small moving targets by using a convolutional neural network and a Bayesian tracker. Our object detection consists of a two-step process based on motion and a patch-based convolutional neural network (CNN). The first stage performs a lightweight motion detection operator to obtain rough target locations. The second stage uses this information combined with a CNN to refine the detection results. In addition, we adopt an online track-by-detection approach by using the Probability Hypothesis Density (PHD) 25 filter to convert detections into tracks. The PHD filter offers a robust multi-object Bayesian data-association framework that performs well in cluttered environments, keeps track of missed detections, and presents remarkable computational advantages over different Bayesian filters. We test our method across various cases of a challenging dataset: a low-resolution satellite video comprising numerous small moving objects. We demonstrate the proposed method outperforms competing approaches across different scenarios with both object detection and object tracking metrics (Figure 8). These have been published in MLSP 2022 4 and Frontiers in Signal Processing2. Currently, our work involves developing better and computationally feasible approximations for the Random Finite Set (RFS) posterior using state-of-the-art filters 26 and novel CNN architectures 24.

Figure 8

7.6 A Two-Stage Road Segmentation Approach for Remote Sensing Images

Participants: Josiane Zerubia.

External collaborators: Tianyu Li [Purdue University, US, PhD student], Mary Comer [Purdue University, US, Associate professor].

Keywords: remote sensing, road segmentation, convolutional neural network (CNN), two stage learning.

This work consisted of the development of a two-stage learning strategy for road segmentation in remote sensing images. With the development of new technology in satellites, cameras and communications, massive amounts of high-resolution remotely-sensed images of geographical surfaces have become accessible to researchers. To help in analyzing these massive image datasets, automatic road segmentation has become more and more important. Many road segmentation methods based on CNNs have been proposed for remote sensing images in recent years. Although these techniques show great performance in various applications, there are still problems in road segmentation, due to the existence of complex backgrounds, illumination changes, and occlusions due to trees and cars. To alleviate such problems, we propose a two-stage strategy for road segmentation. The diagram of the two-stage segmentation approach is given in Figure 9. A selected network (such as ResUnet) is trained in stage one with the original RGB training samples. Then the network trained in stage one is used to generate the training samples for stage two. To be specific, when an RGB training sample is fed to the trained network in stage one, a probability map and a weight map are generated. The probability map is attached to the RGB training sample as a 4 dimension input to the U-Net-like network in stage two. The weight map is used for calculating the loss function in stage two. Our experiments on the Massachusetts road dataset 21 and the DeepGlobe dataset 13 show the average IoU can increase up to 3% from stage one to stage two, which achieves state-of-the-art results on these datasets. Moreover, from the qualitative results, such as the examples in Figure 10, obvious improvements from stage one to stage two can be seen: fewer false positives and better connection of road lines.

This work was presented at ICPRw 2022 - 26th International Conference on Pattern Recognition workshops (PRRS 2022) Montréal, Canada, August 21, 2022 6.

Figure 9

Figure 10

8.1 Bilateral contracts with industry

9.1 National initiatives

Participants: Bilel Kanoun, Josiane Zerubia.

External collaborators: Isabelle Manighetti [OCA, Géoazur, Senior Physicist].

AYANA team is part of the ANR project FAULTS R GEMS (2017-2023, PI Geoazur) dedicated to realistic generic earthquake modeling and hazard simulation.

10.1 Promoting scientific activities

10.2 Teaching - Supervision - Juries

10.3 Popularization

• Participation to the day “Girls, Mathematics and Computer Science” at Lycée du Val d'Argens au Muy dans le Var (83), Apr. 2022 – Martina Pastorino , Josiane Zerubia

11.1 Major publications

• 1 inbookI.Ihsen Hedhli, G.Gabriele Moser, S. B.Sebastiano B Serpico and J.Josiane Zerubia. Hierarchical Markov random fields for high resolution land cover classification of multisensor and multiresolution image time series.2Change Detection and Image Time Series AnalysisSupervised MethodsISTE-WileyDecember 2021, 1-27
  HAL

11.2 Publications of the year

11.3 Cited publications

• 12 inproceedingsA.Alex Bewley, Z.Zongyuan Ge, L.Lionel Ott, F.Fabio Ramos and B.Ben Upcroft. Simple online and realtime tracking.2016 IEEE international conference on image processing (ICIP)IEEE2016, 3464--3468
  back to text
• 13 articleI.I. Demir, K.K. Koperski, D.D. Lindenbaum, G.G. Pang, J.J. Huang, S.S. Basu, F.F. Hughes, D.D. Tuia and R.R. Raskar. DeepGlobe 2018: A challenge to parse the earth through satellite images.IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)2018
  back to text
• 14 articleY.Yaoshiang Ho and S.Samuel Wookey. The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling.IEEE Access82020, 4806-4813
  DOIback to text
• 15 articleP.Paul Jaccard. The distribution of the flora in the Alpine zone.1.New Phytologist1121912, 37-50
  DOIback to text
• 16 inproceedingsM. S.Maria S Kulikova, I. H.Ian H Jermyn, X.Xavier Descombes, E.Elena Zhizhina and J.Josiane Zerubia. Extraction of Arbitrarily-Shaped Objects Using Stochastic Multiple Birth-and-Death Dynamics and Active Contours.Proc. Computational Imaging VIIISPIE2010
  back to textback to text
• 17 articleC.Caroline Lacoste, X.Xavier Descombes and J.Josiane Zerubia. Point processes for unsupervised line network extraction in remote sensing.IEEE TPAMI27102005, 1568--1579
  back to textback to text
• 18 inproceedingsT.-Y.Tsung-Yi Lin, P.Priya Goyal, R.Ross Girshick, K.Kaiming He and P.Piotr Dollar. Focal Loss for Dense Object Detection.Proceedings of the IEEE International Conference on Computer Vision (ICCV)2017, 2980-2988
  back to text
• 19 inproceedingsS.Siyu Liu, C.Changtao He, H.Haiwei Bai, Y.Yijie Zhang and J.Jian Cheng. Light-Weight Attention Semantic Segmentation Network for High-Resolution Remote Sensing Images.IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing SymposiumWaikoloa, HI, USASep 2020, 2595-2598
  back to text
• 20 articleL.Lionel Mattéo, I.Isabelle Manighetti, Y.Yuliya Tarabalka, J.-M.Jean-Michel Gaucel, M.Martijn van den Ende, A.Antoine Mercier, O.Onur Tasar, N.Nicolas Girard, F.Frédérique Leclerc, T.Tiziano Giampetro, S.Stéphane Dominguez and J.Jacques Malavieille. Automatic Fault Mapping in Remote Optical Images and Topographic Data With Deep Learning.Journal of Geophysical Research: Solid Earth1264e2020JB021269 2020JB0212692021, e2020JB021269
  DOIback to textback to text
• 21 phdthesisV.V. Mnih. Machine Learning for Aerial Image Labeling.University of Toronto2013
  back to text
• 22 inproceedingsS.Shaoqing Ren, K.Kaiming He, R. B.Ross B. Girshick and J.Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks..NIPSMIT Press2015, 91-99
  back to text
• 23 inproceedingsK.K. Sun, B.B. Xiao, D.D. Liu and J.J. Wang. Deep High-Resolution Representation Learning for Human Pose Estimation.IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)5686-5696, 2019
  back to text
• 24 inproceedingsA.Ashish Vaswani, N.Noam Shazeer, N.Niki Parmar, J.Jakob Uszkoreit, L.Llion Jones, A. N.Aidan N Gomez, Ł.Łukasz Kaiser and I.Illia Polosukhin. Attention is All you Need.Advances in Neural Information Processing Systems30Curran Associates, Inc.2017, 5998-6008
  back to text
• 25 articleB.-N.Ba-Ngu Vo and W.-K.Wing-Kin Ma. The Gaussian Mixture Probability Hypothesis Density Filter.IEEE Transactions on Signal Processing5412 2006, 4091 - 4104
  DOIback to text
• 26 articleB.-N.Ba-Ngu Vo, B.-T.Ba-Tuong Vo and H. G.Hung Gia Hoang. An Efficient Implementation of the Generalized Labeled Multi-Bernoulli Filter.IEEE Transactions on Signal Processing6582017, 1975-1987
  DOIback to text
• 27 inproceedingsJ.Jingru Yi, P.Pengxiang Wu, B.Bo Liu, Q.Qiaoying Huang, H.Hui Qu and D.Dimitris Metaxas. Oriented Object Detection in Aerial Images with Box Boundary-Aware Vectors.Proc. IEEE Winter Conf. on Applications of Computer Vision2021
  back to textback to textback to text