Convergent storage for HPC systems and clouds through transactional blobs.

We believe that HPC-Big Data convergence can efficiently be tackled from the angle of HPC/cloud storage convergence. Since 2016 we have been investigating how object-based storage could serve as a common abstraction for both HPC simulations (traditionally running on supercomputers) and for Big Data analytics (traditionally running on clouds). We argue that transactional blob storage is a strong candidate for future data platforms combining these infrastructures  36.

We proposed a pioneering approach to the convergence of HPC and Big Data from the perspective of storage, by introducing Týr — the first blob storage system to provide built-in, multiblob transactions, while retaining sequential consistency and high throughput under heavy access concurrency. It leverages novel version management and read protocols allowing applications to read a consistent version of a blob spanning multiple chunks in presence of concurrent, conflicting writers. Large-scale experiments on the Microsoft Azure cloud with a production application from CERN LHC show Týr throughput outperforming state-of-the-art solutions by more than 75 % 10 (Best Student Paper Award Finalist at the ACM/IEEE SC16 conference — the reference international conference in the HPC area). We investigate this approach on various large-scale platforms with a diverse spectrum of applications.

Provisioning storage resources for hybrid supercomputer/cloud infrastructures.

Another recent idea we are developing in the context of HPC and Big Data convergence concerns the provisioning of storage resources. The way these resources are accessed on supercomputers and clouds opposes a complex low-level vision that requires tight user control (on supercomputers) and a very abstract vision that implies uncertainty for performance modeling (on clouds). Nevertheless, taking full advantage of all available resources is critical in a context where storage is central for coupling workflow components. This line of work is still preliminary.

A first effort has begun in 2021 on how to make hybrid heterogeneous storage resources allocatable and elastic to meet the needs of I/O-intensive hybrid workloads. A publication is in preparation to share the first results. A PhD thesis started in October 2021 on this topic, with a focus on scheduling algorithms in the context of a storage-aware job scheduler.

Malleable storage systems.

Using storage resources in an elastic, cloud-inspired manner on supercomputers or on hybrid supercomputer/cloud infrastructures requires adequate modeling. Naturally, energy and cost savings can be obtained by reducing idle resources. Malleability, which is the possibility for resource managers to dynamically increase or reduce the resources of jobs, appears as a promising means to progress towards this goal. However, state-of-the-art parallel and distributed file systems have not been designed with malleability in mind. This is mainly due to the supposedly high cost of storage decommission, which is considered to involve expensive data transfers. Nevertheless, as network and storage technologies evolve, old assumptions on potential bottlenecks can be revisited.

We investigate and evaluate the viability of malleability as a design principle for a distributed file system. We specifically model the minimal duration of the rescaling operations for storage resources. Furthermore, we introduce Pufferbench  26, a benchmark for evaluating how fast one can scale up and down a distributed storage system on a given infrastructure and, thereby, how viably can one implement storage malleability on it. We explore how it can serve to quickly prototype and evaluate mechanisms for malleability in existing distributed storage systems.

Towards unified data processing techniques for Extreme Computing and Big Data applications.

As part of our contribution to HPC-Big Data convergence, another direction we explore regards how to efficiently leverage HPC infrastructures for data-intensive applications. We propose a solution based on burst buffers, which enables efficient analytics that can run concurrently to simulations on the same supercomputer. This is achieved thanks to a prefetching technique that fetches the input data for analytics applications to be stored close to computing nodes thus reducing the latency of reading data inputs  39.

Scalable stream storage across the continuum.

As myriads of frameworks emerge for for stream processing 8, we argue that a plausible path to follow towards scalable stream storage is the careful design and implementation of a unified architecture for stream ingestion and storage which can lead to the optimization of the processing of Big Data applications across the continuum. This approach minimizes data movement within the analytics architecture, finally leading to better utilized resources.

We propose a set of design principles for a scalable, unified architecture for data ingestion and storage: (1) dynamic partitioning based on semantic grouping and sub- partitioning, which enables more flexible and elastic management of stream partitions; (2) lightweight offset indexing (i.e., reduced stream offset management overhead) optimized for sequential record access; (3) adaptive and fine-grained replication to trade-off in-memory storage with performance (low-latency and high throughput with durability).

We implement and evaluate this architecture through the KerA prototype with the goal of efficiently handling diverse access patterns: low-latency access to streams and/or high throughput access to unbounded streams and/or objects 7. KerA serves a playground for our experimental research in the area of cloud/edge storage.

Cost-efficient execution plans for the uniform placement of stream analytics on the Edge-to-Cloud continuum.

In order to execute a request over hybrid Edge-Cloud deployments, one needs a specific execution plan for the Edge engines, another one for the Cloud engines and to ensure the right interconnection between them thanks to an ingestion system. Manually and empirically deploying this analytics pipeline (Edge-Ingestion-Cloud) can lead to sub-optimal computation placement with respect to the network cost (i.e., high latency, low throughput) between the Edge and the Cloud.

We argue that a uniform approach is needed to bridge the gap between Cloud SPEs (Stream Processing Engines) and Edge analytics frameworks in order to leverage a single, transparent execution plan for stream processing in both environments. We introduce Planner, a streaming middleware capable of finding cost-efficient cuts of execution plans between Edge and Cloud. Our goal is to find a distributed placement of operators on Edge and Cloud nodes to minimize the stream processing makespan.

Edge benchmarking.

While a plethora of frameworks for Edge processing were recently proposed, the distributed systems community has no clear means today to discriminate between them. Some preliminary surveys exist, focusing on a feature-based comparison. We claim that a step further is needed, to enable a performance-based comparison.

To this purpose, the definition of a benchmark is a necessity. We make a step towards the definition of a methodology for benchmarking Edge processing frameworks  37. Using two representative real-life stream processing applications and state-of-the-art processing engines, we perform an experimental evaluation based on the analysis of the execution of those applications in fully-Cloud computing and hybrid Cloud-Edge computing infrastructures.

Exploring the continuum through reproducible experiments.

Understanding end-to-end performance in such a complex continuum is challenging. This breaks down to reconciling many, typically contradicting, application requirements and constraints with low-level infrastructure design choices. One important challenge is to accurately reproduce relevant behaviors of a given application workflow and representative settings of the physical infrastructure underlying this complex continuum.

We introduce a rigorous methodology for such a process and validate it through E2Clab  11. It is the first platform to support the complete analysis cycle of an application on the Computing Continuum: (i) the configuration of the experimental environment, libraries and frameworks; (ii) the mapping between the application parts and machines on the Edge, Fog and Cloud; (iii) the deployment of the application on the infrastructure; (iv) the automated execution; and (v) the gathering of experiment metrics.

We further propose a methodology to support the optimization of real-life applications on the Edge-to-Cloud Continuum. Our approach 16, implemented as an extension of E2Clab, relies on a rigorous analysis of possible configurations in a controlled testbed environment to understand their behavior and related performance trade-offs. We illustrate our methodology by optimizing Pl@ntNet, a world-wide plant identification application. This methodology can be generalized to other applications in the Edge-to-Cloud Continuum.

Supporting AI across the continuum.

An increasing number of AI applications turn to the Computing Continuum to speed-up decision making, by processing data close to their sources. One such example is the Earthquake Early Warning (EEW) which detects and characterizes medium and large earthquakes before their damaging effects reach a certain location.

Our research aims to improve the accuracy of EEW systems by means of machine learning across the continuum. We introduce the Distributed Multi-Sensor Earthquake Early Warning (DMSEEW) system, a novel machine learning-based approach that combines data from both types of sensors (GPS stations and seismometers) to detect medium and large earthquakes. The system builds on a geographically distributed infrastructure, which can be deployed on clouds and edge systems. This ensures an efficient computation in terms of response time and robustness to partial infrastructure failures.

Our experiments show that DMSEEW is more accurate than the traditional seismometer-only approach and the combined-sensors (GPS and seismometers) approach that adopts the rule of relative strength. These results have been accepted for publication at AAAI (Special Track on AI for Social Impact), a "A*" conference in the area of Artificial Intelligence 5. This paper won the Outstanding Paper Award - Special Track on AI for Social Impact.

Scalable I/O and in situ processing with Damaris: leveraging dedicated resources for HDF-based storage and for data processing for geoscience and climate applications.

Supercomputers are expected to reach Exascale by 2022. The Fugaku supercomputer (now first in the Top500 supercomputer list) reached 442 PFlop/s (LINPACK performance, which serves to established the ranking) and was reported to have already overcome 1 EFlop/s as early as June 2020 on a specific benchmark (HPC-AI). Extreme-scale scientific simulations deployed on pre-Exascale HPC machines usually store the resulted datasets in standard formats such as HDF5 or NetCDF. In the data storage process, two different approaches are traditionally employed: 1) file-per-process; and 2) collective I/O.

In our research we explore an approach based on dedicated resources for data aggregation and storage. The computing resources are partitioned such that a subset of cores in each node or a subset of nodes of the underlying platform are dedicated to data management. The data generated by the simulation processes are transferred to these dedicated cores/nodes either through shared memory (in the case of dedicated cores) or through the MPI calls (in the case of dedicated nodes) and can be processed asynchronously, with data aggregated and stored in HDF5 format using a specific plug-in.

This approach is implemented in Damaris (3 and  29), a middleware software developed in our team. It originated in our collaboration with the JLESC, and was the first software resulted from this joint lab validated in 2011 for integration into the Blue Waters supercomputer project. It scaled up to 16,000 cores on the Titan supercomputer (first in the Top500 supercomputer list in 2013) before being validated on other major HPC systems.

We recently have focused on two main targets:

1. Validating the benefits of the dedicated-core approach for new types of storage backends, e.g., HDF5, where it enables performance improvements approaching the order of 300 % compared to the standard file-per-process approach.
2. Validating Damaris for in situ processing for new simulations codes from different scientific domains, i.e. geoscience (wave propagation) and ocean modeling (e.g., the CROCO coastal and ocean simulation).

Ongoing efforts are supporting the integration of Damaris with the Code_Saturne CFD application from EDF for in-situ visualization 21, within the PRACE6IP project; the integration with the OPM Flow simulator, developed at Sintef (Norway), used for reservoir simulations and carbon sequestration (within the ACROSS EuroHPC project). Future plans include the development of the Damaris as a key technology for asynchronous I/O and in-situ processing on future European Exascale machines.

Scalable communication: exploring topology-awareness on Dragonfly-based supercomputers.

Recent network topologies in supercomputers have motivated new research on topology-aware collective communication algorithms for MPI. But such an endeavor requires betting on the fact that topology-awareness is the primary factor to accelerate these collective operations. Besides, showing the benefit of a new topology-aware algorithm requires not only access to a leadership-scale supercomputer with the desired topology, but also a large resource allocation on this supercomputer.

Event-driven network simulations can alleviate such constraints and speed up the search for appropriate algorithms by providing early answers on their relative merit. In our studies, we focused on the Scatter and AllGather operations in the context of the Theta supercomputer's Dragonfly topology  27. We proposed a set of topology-aware versions of these operations as well as optimizations of the old, non-topology-aware ones. A trivial implementation of Scatter using non-blocking point-to-point communications can be faster than state-of-the art algorithms by up to a factor of 6. Traditional AllGather algorithms can also be improved by the same principle and exhibit a 4x speedup in some situations. These results highlight the need to rethink the collective operations under the light of non-blocking communications.

For this work, Nathanaël Cheriere received the Third Prize at the ACM Graduate Student Research Competition organized at the SC'16 conference.

Achieving fast atomic RDMA-based replication.

Performance and fault tolerance are often contradictory goals. Replication is essential for fault-tolerance for in-memory storage systems, which are increasingly present on HPC systems as well as on cloud infrastructures. However, it is a source of high overhead. Remote direct memory access (RDMA) is an attractive technology to support the fast creation of copies of data, since it is low-latency and has no CPU overhead at the target. To ensure atomic data transfers, receivers check and apply only fully received messages. In contrast to such an approach, we introduced Tailwind 12, a zero-copy recovery-log replication protocol for scale-out in-memory databases.

Tailwind is the first replication protocol that eliminates all CPU-driven data copying and fully bypasses target server CPUs, thus leaving backups idle. Tailwind ensures all writes are atomic by leveraging a protocol that detects incomplete RDMA transfers. It substantially improves replication throughput and response latency compared with conventional RPC-based replication. Experiments show Tailwind improves RAMCloud's normal-case request processing throughput by 1.7x. It also cuts down writes median and 99th percentile latencies by 2x and 3x respectively. This work is carried out in collaboration with Ryan Stutsman, a coauthor of RAMCloud.

Social impact.

One of our target applications is Early Earthquake Warning. We proposed a solution that enables earthquakes classification with an outstandingly perfect accuracy. By enabling accurate identification of strong earthquakes, it becomes possible to trigger adequate measures and save lifes. For this reason, our work was distinguished with an Outstanding Paper Award — Special Track for Social Impact at AAAI-20, an A* conference in the area of Artificial Intelligence. This result has been highlighted by the Le Monde journal in its edition of December 28, 2020, in a section entitled: Ces découvertes scientifiques que le Covid-19 a masquées en 2020. This collaborative work continued in 2021.

Environmental impact.

As presented in Section 4, we have started a collaboration with CybeleTech that we plan to materialize in the framework of the EUPEX EuroHPC project. CybeleTech is a French company specialized in precision agriculture. Within the framework of our collaboration, we propose to focus our efforts on a scale-oriented data management mechanism targeting two CybeleTech use-cases. They address irrigation scheduling for orchards and optimal harvest date for corn, and their models require the acquisition of large volumes of remote data. The results of our collaboration will have concrete applications as they will improve the accuracy of plant growth models and improve decision making for precision agriculture, which directly aims to contribute to sustainable development.

7.1.1 Damaris

• Keywords:
  Visualization, I/O, HPC, Exascale, High performance computing
• Scientific Description:

  Damaris is a middleware for I/O and data management targeting large-scale, MPI-based HPC simulations. It initially proposed to dedicate cores for asynchronous I/O in multicore nodes of recent HPC platforms, with an emphasis on ease of integration in existing simulations, efficient resource usage (with the use of shared memory) and simplicity of extension through plug-ins.

  Over the years, Damaris has evolved into a more elaborate system, providing the possibility to use dedicated cores or dedicated nodes to in situ data processing and visualization. It proposes a seamless connection to the VisIt visualization framework to enable in situ visualization with minimum impact on run time. Damaris provides an extremely simple API and can be easily integrated into the existing large-scale simulations.

  Damaris was at the core of the PhD thesis of Matthieu Dorier, who received an Accessit to the Gilles Kahn Ph.D. Thesis Award of the SIF and the Academy of Science in 2015. Developed in the framework of our collaboration with the JLESC – Joint Laboratory for Extreme-Scale Computing, Damaris was the first software resulted from this joint lab validated in 2011 for integration to the Blue Waters supercomputer project. It scaled up to 16,000 cores on Oak Ridge’s leadership supercomputer Titan (first in the Top500 supercomputer list in 2013) before being validated on other top supercomputers. Active development is currently continuing within the KerData team at Inria, where it is at the center of several collaborations with industry as well as with national and international academic partners.

• Functional Description:
  Damaris is a middleware for data management and in-situ visualization targeting large-scale HPC simulations. Damaris enables: In-situ data analysis by using selected dedicated cores/nodes of the simulation platform. Asynchronous and fast data transfer from HPC simulations to Damaris. Semantic-aware dataset processing through Damaris plug-ins, Writing aggregated data (by hdf5 format) or visualizing them either by VisIt or ParaView.
• Release Contributions:
  This version includes the following changes: A Spack package.py file is available through the official Spack repository allowing automated source builds from source. Support for unstructured mesh model types are available for Paraview in-situ visualization.
• URL:
  https://­project.­inria.­fr/­damaris/
• Contact:
  Gabriel Antoniu
• Participants:
  Gabriel Antoniu, Lokman Rahmani, Luc Bouge, Matthieu Dorier, Orçun Yildiz, Hadi Salimi, Joshua Charles Bowden
• Partner:
  ENS Rennes

7.1.2 KerA

• Name:
  KerAnalytics
• Keywords:
  Big data, Distributed Storage Systems, Streaming, Real time
• Scientific Description:

  Current state-of-the-art Big Data analytics architectures are built on top of a three-layer stack: data streams are first acquired by the ingestion layer (e.g., Kafka) and then they flow through the processing layer (e.g., Flink) which relies on the storage layer (e.g., HDFS) for storing aggregated data or for archiving streams for later processing. Unfortunately, in spite of potential benefits brought by specialized layers (e.g., simplified implementation), moving large quantities of data through specialized layers is not efficient: instead, data should be acquired, processed and stored while minimizing the number of movements. To that end, we design a unified architecture for stream ingestion and storage which leads to better utilized resources.

  We implement and evaluate this architecture through the KerA prototype with the goal of efficiently handling diverse access patterns: low-latency access to streams and/or high-throughput access to objects. We further introduce a shared virtual log-structured storage approach for improving the cluster throughput when multiple producers and consumers write and read data streams in parallel. Stream partitions are associated with shared replicated virtual logs transparently to the user, effectively separating the implementation of stream partitioning (and data ordering) from data replication (and durability).

  KerA was at the core of the PhD thesis of Ovidiu-Cristian Marcu. It was developed on top of RAMCloud, a low-latency key-value distributed system, and serves as a playground for our experimental research in the area of cloud/edge storage.

• Functional Description:
  KerA is a software prototype which addresses the limitations of state-of-the-art storage solutions for stream processing. It illustrates the idea of a unified storage strategy for data ingestion and storage, while allowing low-latency access to streams and/or high throughput access to unbounded streams and objects.
• News of the Year:
  Latest developments: in 2021, KerA was added support for containers allowing to facilitate further deployments. In particular, we aim at providing automatic scaling for stream processing applications.
• URL:
  https://­kerdata.­gitlabpages.­inria.­fr/­Kerdata-Codes/­kera-website/
• Publications:
  hal-01773799, tel-02127065, hal-01532070
• Authors:
  Ovidiu-Cristian Marcu, Gabriel Antoniu, Alexandru Costan, Thomas Bouvier
• Contact:
  Thomas Bouvier

7.1.3 E2Clab

• Name:
  Edge-to-Cloud lab
• Keywords:
  Distributed Applications, Distributed systems, Computing Continuum, Large scale, Experimentation, Evaluation, Reproducibility
• Functional Description:

  E2Clab is a framework that implements a rigorous methodology that provides guidelines to move from real-life application workflows to representative settings of the physical infrastructure underlying this application in order to accurately reproduce its relevant behaviors and therefore understand end-to-end performance. Understanding end-to-end performance means rigorously mapping the scenario characteristics to the experimental environment, identifying and controlling the relevant configuration parameters of applications and system components, and defining the relevant performance metrics.

  Furthermore, this methodology leverages research quality aspects such as the Repeatability, Replicability, and Reproducibility of experiments through a well-defined experimentation methodology and providing transparent access to the experiment artifacts and experiment results. This is an important aspect that allows that the scientific claims are verifiable by others in order to build upon them.

• URL:
  https://­gitlab.­inria.­fr/­E2Clab/­e2clab
• Contact:
  Gabriel Antoniu

8.1.1 Provisioning storage resources for hybrid supercomputer/cloud infrastructures

Participants: François Tessier, Julien Monniot, Gabriel Antoniu, Matthieu Robert.

• Collaboration.
  This work has been carried out in close co-operation with Rob Ross, Argonne National Lab.

One of the recent axis we are developing in the context of HPC and Big Data convergence concerns the provisioning of storage resources. The way these resources are accessed on supercomputers and clouds opposes a complex low-level vision that requires tight user control (on supercomputers) and a very abstract vision that implies uncertainty for performance modeling (on clouds). Nevertheless, taking full advantage of all available resources is critical in a context where storage is central for coupling workflow components. Our goal is then to make heterogeneous storage resources distributed across HPC+Cloud infrastructures allocatable and elastic to meet the needs of I/O-intensive hybrid workloads.

A first effort has begun in 2021 through the MS internship of Matthieu Robert. Matthieu explored ways of extracting relevant data from I/O traces so they can be replayed in StorAlloc, our simulator of a storage-aware job scheduler. The PhD thesis of Julien Monniot also started in October 2021 on this topic, with a focus on scheduling algorithms for storage-aware job schedulers. It will build upon and extend the preliminary work that was started during the internship.

A publication is in preparation to share our first results. StorAlloc, still under very active development, serves as a testbed for our dynamic storage scheduling algorithms. This simulator is already able to replay and visualize the scheduling of I/O intensive jobs from Theta at Argonne National Laboratory, a 10 PFlops supercomputer.

8.1.2 Supporting seamless execution of HPC-enabled workflows across the Computing Continuum

Participants: Juliette Fournis d'Albiat, Alexandru Costan, François Tessier, Gabriel Antoniu.

• Collaboration.
  This work has been carried out in close co-operation with Rosa Badia, Barcelona Supercomputing Center

Scientific applications have recently evolved to more complex workflows combining traditional HPC simulations with Data Analytics (DA) and ML algorithms. While these applications were traditionally executed on separate infrastructures (e.g., simulations on HPC supercomputers, DA/ML on cloud/edge), the new combined ones need to leverage myriads of resources from the edge to the HPC in order to promptly extract insights. Our goal is to enable seamless deployment, orchestration and optimization of HPC workflows across the Computing Continuum.

To this end, we have started to leverage the building blocks developed at BSC and Inria. In particular, the E2Clab framework already enables the deployment and execution of DA/ML workflows on cloud and edge resources following a reproducibility-oriented methodology. We are exploring the possibility to extend this support workflows including simulations and other compute intensive applications on HPC infrastructures. The layers and services abstractions introduced by E2Clab fit naturally to the approach based on containers for scientific codes pushed forward by BSC. Beyond the ease of deployment, this approach aims to enable the reproducibility of HPC experiments, one of the core design principles of E2Clab.

In the framework of the internship of Juliette Fournis d'Albiat, one direction explored during this year towards such integration is supporting TOSCA-based HPC workflows in E2Clab. Of particular interest are the links and interoperability issues with TOSCA, the Ystia orchestrator and the PyCOMPSs programming model proposed by BSC. A second step consists in investigating the semantics for workflow description that would be compatible with E2Clab (which could possibly be extended) in order to fit the HPC Workflow as a Service paradigm. Finally, a proof-of-concept integrating the support for HPC with E2Clab should be designed and evaluated with real-life applications.

8.1.3 Artificial intelligence-based data analysis in heterogeneous and volatile environments

Participants: Cédric Prigent, Alexandru Costan, Gabriel Antoniu.

• Collaboration.
  This work has been carried out in close co-operation with Loïc Cudennec (DGA) and with DFKI in the context of the ENGAGE Inria-DFKI project.

One of the particularities of AI is to interfere – positively - in all domains: computer-aided design, the search for optimized solutions and of course all business applications with tedious, repetitive and sufficiently complex tasks that cannot be performed by a conventional computer program. Application domains benefiting greatly from these advances are the distributed heterogeneous systems, like the cooperative and autonomous vehicles. One challenge in this context lies in the ability to have relevant data at a given location and at a given time. For this, three mechanisms must be finely studied: data locality, task scheduling and orchestration. These issues call upon many well-studied research themes in homogeneous distributed systems such as supercomputers or large-scale file-sharing systems. But these themes run into new problems once the targeted system is made up of several heterogeneous systems, each bringing different paradigms for task and data management. The research context of the PhD thesis of Cédric Prigent started this year is related to the management of this heterogeneity.

The objective is to identify and adapt emerging approaches from the "computing continuum" in order to address the issues of distribution of calculations and processing, particularly in the case of workflows involving AI. This exploratory thesis has concrete applications such as the SCAF project or on civil warning systems and is situated upstream of these to help guide future technical choices.

During this year we have started to investigate and characterize different middleware for managing shared computations and data on HPC, Cloud, Fog, Edge systems, in centralized (client-server), decentralized (peer-to-peer), or hybrid configurations. We have also identified the communicating systems present in the targeted use cases and characterized them according to the reading grid established previously. We are currently surveying emerging scientific work on "continuum computing", and we are selecting and proposing adaptations for relevant contributions in the case of data intensive workflows.

8.1.4 Memory and data-awareness in hybrid workflows

Participant: François Tessier.

HPC and HPDA opens up the opportunity to solve a wide variety of questions and challenges. The number and complexity of challenges that these two domains can help with are limited by the performance of computer software and hardware. Increasingly, performance is now limited by how fast data can be moved within the memory and storage of the hardware. However, for years, the focus has been primarily on optimizing floating point operations. In the framework of the Maestro EU Project, we proposed a novel memory- and data-aware middleware called whose goal is to improve the performance of data movement in HPC and HPDA  32.

In particular, we have developed a data abstraction layer and an API designed for data orchestration. Our framework has been evaluated on synthetic benchmarks and on components of a production workflows for weather forecast.

8.2.1 Reproducible Performance Optimization of Complex Applications on the Edge-to-Cloud Continuum

Participants: Daniel Rosendo, Alexandru Costan, Gabriel Antoniu.

In more and more application areas, we are witnessing the emergence of complex workflows that combine computing, analytics and learning. They often require a hybrid execution infrastructure with IoT devices interconnected to cloud/HPC systems (aka Computing Continuum). Such workflows are subject to complex constraints and requirements in terms of performance, resource usage, energy consumption and financial costs. This makes it challenging to optimize their configuration and deployment 22.

We propose a methodology 16 to support the optimization of real-life applications on the Edge-to-Cloud Continuum. We implement it as an extension of E2Clab  11, a previously proposed framework supporting the complete experimental cycle across the Edge-to-Cloud Continuum. Our approach relies on a rigorous analysis of possible configurations in a controlled testbed environment (Grid’5000  24) to understand their behavior and related performance trade-offs. We illustrate our methodology by optimizing Pl@ntNet  35, a world-wide plant identification application. Our methodology can be generalized to other applications in the Edge-to-Cloud Continuum.

Since real-life Edge-to-Cloud deployments spans among IoT, Edge, Fog, and Cloud/HPC resources, currently we are extending E2Clab to support multiple large-scale testbeds (e.g., Grid’5000  24 + CHI@Edge + FIT IoT LAB  23). The ultimate goal is to allow more realistic experimental evaluations through cross-testbed large-scale deployments combining small wireless sensors (FIT IoT LAB), Fog/Edge devices (CHI@Edge), and Cloud/HPC machines (Grid’5000).

8.2.2 Enabling Provenance Capture in Edge-to-Cloud Workflows

Participants: Daniel Rosendo, Alexandru Costan, Gabriel Antoniu.

• Collaboration.
  This work has been carried out in close co-operation with Marta Mattoso, Federal University of Rio de Janeiro.

Supporting reproducibility of experiments carried out on large scale distributed and heterogeneous infrastructures is non-trivial. The experimental methodology, the artifacts used, and the data captured should provide additional context that more accurately explains the experiment execution and results.

In particular, one relevant challenge is the provenance capture (with a focus on debugging and data analysis support) on such highly heterogeneous and distributed infrastructures 18. It requires the design and development of novel provenance systems to efficiently capture data from resource constrained hardware resources like IoT/Edge devices (e.g. requires smart data capture strategies to reduce the capture overhead).

In this context, meaningful questions that arise are: How do the existing provenance capture systems (e.g., Komadu, DfAnalyzer) perform in resource-constrained environments? How to minimize the provenance capture overhead in IoT/Edge environments? Regarding this research direction, we started to explore the performance (e.g., time required to capture provenance; amount of main memory used; amount of data transmitted) of reference provenance systems such as Komadu and DfAnalyzer to capture provenance in resource constrained IoT/Edge devices. Experimental evaluations will be carried out on real-life IoT/Edge hardware (e.g., CHI@Edge, FIT IoT LAB  23).

The ultimate goal is that the novel approach could allow users to instrument their application code to capture provenance during the execution of a variety of tasks performed by IoT/Edge devices (e.g., model training, data processing, among others). The proposed approach will be integrated in E2Clab to enable provenance capture in Edge-to-Cloud experiments and allow users to debug and analyze experiment data to better understand the results.

8.2.3 Towards Data Parallel Rehearsal-based Continual Learning

Participants: Thomas Bouvier, Hugo Chaugier, Alexandru Costan, Gabriel Antoniu.

• Collaboration.
  This work has been carried out in close co-operation with Bogdan Nicolae (Argonne National Laboratory, USA), who co-advised the internship of Hugo Chaugier.

Although not a new concept, the adoption of deep learning accelerated in recent years because of the ever-increasing amount of data being generated. Extracting valuable knowledge from it requires training large learning models using parallel and distributed strategies. However as more data accumulates in data centers over time, it becomes prohibitively expensive to re-train deep learning models from scratch to update them with new data. This concern is emphasized in the context of the Edge-to-Cloud Continuum, where 1) real-world sensing applications requires the need for agents that can adapt to continuously evolving environments and 2) compute capacities might be constrained across processing elements.

Besides, standard deep neural networks (DNNs) suffer from the well-known issue of catastrophic forgetting, making continual learning difficult. Specifically, incremental training introduces a bias towards newly learned patterns which makes the model forget about the previously learned ones.

These two challenges, distributing the training process and preventing catastrophic forgetting, have not been studied together yet are essential in the implementation of large scale deep learning. A first effort on that topic has begun in 2021 through the MS internship of Hugo Chaugier. Since then, we studied how different continual learning strategies behave at scale using Grid'5000. In particular, we showed that rehearsal-based methods did not suffer from data parallelization but on the contrary benefited from it, as increasing the total rehearsal memory size improves the ability of the DNN to remember previous tasks.

8.2.4 Deploying Heterogeneity-aware Deep Learning Workloads on the Computing Continuum

Participants: Thomas Bouvier, Alexandru Costan, Gabriel Antoniu.

The Computing Continuum led to the evolution of centralized data centers towards interconnected processing elements spanning from edge devices to cloud data centers. While this type of infrastructure benefits real-time analysis by reducing latency, the network and compute heterogeneity across and within clusters may negatively impact deep learning workloads.

We conducted preliminary experiments on Grid’5000 using the methodology implemented in E2Clab to help reproduce both relevant behaviors of the given DL workloads and representative settings of the physical infrastructure underlying the continuum. The goal is to understand the end-to-end performance of deep learning training in such heterogeneous settings in order to identify the main factors leading to stragglers. We propose such an approach in 20 and 17. The architectural differences of DNNs are compared to help predict inefficient gradients computation and propagation across nodes.

The outcomes will allow to devise novel intra- and inter-cluster strategies to address infrastructure bottlenecks in DL scenarios. This should lead to the design of a middleware helping to distribute DL training, accounting for deterministic and dynamic of both network and compute heterogeneity. Various policies should improve the following objectives while ensuring system scalability: makespan, cost and fairness. Eventually, it will be interesting to study the extent to which these strategies apply to continual learning.

8.2.5 Modeling distributed stream processing across the Edge-to-Cloud Continuum

Participants: Alexandru Costan, Gabriel Antoniu.

• Collaboration.
  This work has been carried out in close co-operation with Daniel Balouek, University of Utah, and Pedro De Souza Bento Da Silva, formerly a post-doc student in the team, and now at Hasso Plattner Institute, Berlin, Germany.

The growth of the Internet of Things is resulting in an explosion of data volumes at the Edge of the Internet. To reduce costs incurred due to data movement and centralized cloud-based processing, it is becoming increasingly important to process and analyze such data closer to the data sources. Exploiting Edge computing capabilities for stream-based processing is however challenging. It requires addressing the complex characteristics and constraints imposed by all the resources along the data path, as well as the large set of heterogeneous data processing and management frameworks. Consequently, the community needs tools that can facilitate the modeling of this complexity and can integrate the various components involved.

To tap into the power of the Computing Continuum, we propose MDSC 13, an approach for modeling distributed stream-based applications running across the Edge-to-Cloud continuum. The objective of MDSC is to facilitate the definition and evaluation of alternative deployment options for such applications atop the Edge-to-Cloud Continuum, in a versatile way. To enable this vision, MDSC leverages a set of simple but expressive architectural concepts (e.g., layers, processing units, buses), which aim to describe the essential components of the applications and to make more explicit their inter-relations and functionalities. These abstractions further enable grouping for components of similar nature and allow to characterize the communication among those groups. Subsequently, this model facilitates the definition of alternative mappings onto the underlying edge/fog/cloud infrastructure.

We demonstrate how MDSC can be applied to a concrete real-life ML-based application - early earthquake warning - to help answer questions such as: when is it worth decentralizing the classification load from the Cloud to the Edge and how?

8.3.1 Exploring Alternative Strategies Leveraging Dedicated Resources for in-situ visualization for Large Scale Simulations

Participants: Joshua Bowden, Gabriel Antoniu, François Tessier.

• Collaboration.
  This work has been carried out in close co-operation with collaborators within the PRACE 6IP project, Simone Bna (CINICA), Miroslav Puskaric, Martin Bidner (HLRS), and with support through the EDF Code_Saturne user group.

As the exascale era approaches, maintaining scalable performance in data management tasks (storage, visualization, analysis, etc.) remains a key challenge in enabling high performance for the application execution. To address this challenge, the Damaris approach proposed by our team leverages dedicated computational resources in multicore nodes to offload data management tasks, including I/O, data compression, scheduling of data movements, in-situ analysis, and visualization. In this study, we evaluate the benefits of Damaris to improve the efficiency of in-situ visualization for Code_Saturne, a fluid dynamics modeling environment. The experiments show Damaris to adequately hide the I/O processing of various visualization processing pipelines in Code_Saturne using the Paraview visualization engine. In all cases the Damaris enabled version of Code_Saturne was found to be more efficient than the identical non-Damaris capable version when running the same Paraview pipeline.

This work investigates and evaluates alternative methods for allocation of resources for Damaris using either dedicated nodes or dedicated cores for visualization within multicore nodes. It shows what problems asynchronous processing can encounter due to under-allocation of resources. The paper also presents an analysis of the integration effort required to add the Damaris functionality to Code_Saturne, which is another important consideration when deciding to integrate a library. The Damaris Paraview interface has now implemented support for an unstructured-grid mesh type, thus opening up a large domain of simulation types relying on this mesh model. This mesh type was a requirement for the interface with Code_Saturne which uses these type extensively. Results have presented to the Code_Saturne development team at EDF and were published as a PRACE white paper 21 available from the PRACE website.

8.3.2 Scalable asynchronous I/O and in-situ processing with Damaris for Carbon Sequestration

Participants: Joshua Bowden, Alexandru Costan, François Tessier, Gabriel Antoniu.

• Collaboration.
  This work has been carried out in close co-operation with Atgeirr Rasmussen (SINTEF) within the framework of the EuroHPC H2020 ACROSS project.

Carbon capture and storage (CCS) is one of the technologies that have a large potential for mitigating CO2 emissions, and can also enable carbon-negative processes. Before one can commit to large-scale carbon storage operations, it is essential to do simulation studies to assess the storage potential and safety of the operation and to optimize the placement and operation of injection wells. Such a simulation is done by computer programs that solve the equations that describe the motion and state of the fluids within the porous rocks. In the ACROSS project, we use OPM Flow, an open-source reservoir simulator program suitable for both industrial uses as well as research.

As such large-scale simulations can take a long time to run, and require significant high-performance computing resources, we investigate how asynchronous I/O and in-situ processing can help improve the performance, scaling, and efficiency of OPM Flow and of workflows using the program. The ACROSS project is using a co-development method, where software requirements inform the hardware design process for next generation HPC systems. The flow software is typical of MPI based simulation software where I/O inhibits the scaling of the simulation to larger machine sizes due to its serialized nature. We started to investigate how the Damaris approach could be leveraged by OPM Flow to provide asynchronous I/O.

In addition, we started to investigate methods for asynchronous analytics that could be integrated into the Damaris library. This includes exploratory work for integration of a Python interface to a distributed server component based on Ray or Dask. The result of this exploratory work has converged on the use of Python with Dask as Dask offers a suite of useful distributed analytic methods using familiar Pythonesque interfaces, similar to NumPy and Pandas. Pursuing this direction will enable new usage possibilities for Damaris that could be further studied in collaboration with CEA within the future NumPEx exploratory PEPR project, currently under discussion.

9.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

9.1.2 Inria associate team not involved in an IIL or an international program

9.1.3 Participation in other International Programs

9.2.1 FP7 & H2020 projects

9.2.2 Collaborations in European Programs, except FP7 and H2020

9.2.3 Other European programs/initiatives

9.2.4 Collaborations with Major European Organizations

Participants: Gabriel Antoniu, Alexandru Costan.

9.3.1 ANR

9.3.2 Other National Projects

10.1.1 Scientific events: organisation

10.1.2 Scientific events: selection

• Gabriel Antoniu:
  Member of the Best Paper Award and Best Student Paper Award Committee for ACM/IEEE SC21.

10.1.3 Journal

10.1.4 Invited talks

• Gabriel Antoniu
  was invited to give a keynote talk on the 10-year collaboration with Argonne National Laboratory within the JLESC international Laboratory (virtual, February 2021).
• Daniel Rosendo:
  • E2Clab: Optimizing Complex Workflow Deployments on the Edge-to-Cloud Continuum. 12th JLESC Workshop, February 24th, 2021.
  • Enabling Reproducible Analysis of Complex Application Workflows on the Edge-to-Cloud Continuum. Experimentation Users-Group Meetings at Inria, December 7th, 2021.
  • Reproducible Performance Optimization of Complex Applications on the Edge-to-Cloud Continuum. 13th JLESC Workshop, December 14th, 2021.
• Thomas Bouvier:
  • Deploying Heterogeneity-aware Deep Learning Workloads on the Computing Continuum. 13th JLESC Workshop, December 16th, 2021.
• François Tessier:
  • Storage allocation over hybrid HPC/Cloud Infrastructures. 12th JLESC Workshop, February 25th, 2021.
• Joshua Bowden:
  • Code_Saturne: Integration with Damaris - The potential of asynchronous I/O. Code_Saturne User Group Meeting, September 15th, 2021.
  • Asynchronous I/O using Damaris - Results from integration with Code_Saturne CFD code. 13th JLESC Workshop, December 16th, 2021.

10.1.5 Leadership within the scientific community

• Gabriel Antoniu:
  • TCI:
    Since 2020, co-leader of the Use-Case Analysis Working Group. TCI (The Transcontinuum Initiative) emerged as a collaborative initiative of ETP4HPC, BDVA, CLAIRE and other peer organizations, aiming to identify joint research challenges for leveraging the HPC-Cloud-Edge computing continuum and make recommendations to the European Commission about topics to be funded in upcoming calls for projects.
  • ETP4HPC:
    Since 2019, co-leader of the working group on Programming Environments and co-lead of two research clusters, contributing to the Strategic Research Agenda of ETP4HPC (next edition to appear in 2022).
  • International lab management:
    Vice Executive Director of JLESC for Inria. JLESC is the Joint Inria-Illinois-ANL-BSC-JSC-RIKEN/AICS Laboratory for Extreme-Scale Computing. Within JLESC, he also serves as a Topic Leader for Data storage, I/O and in situ processing for Inria.
  • Team management:
    Head of the KerData Project-Team (INRIA-ENS Rennes-INSA Rennes).
  • International Associate Team management:
    Leader of the UNIFY Associate Team with Argonne National Lab (2019–2021).
  • Technology development project management:
    Coordinator of the Damaris 2 ADT project (2019–2022).
• Luc Bougé:
  • SIF:
    Co-Vice-President of the French Society for Informatics (Société informatique de France, SIF), in charge of the Teaching Department.
  • CoSSAF:
    Member of the Foundation Committee for the College of the French Scientific Societies (CoSSAF, Collège des sociétés savantes académiques de France) gathering more than 40 French societies from all domains.
• Alexandru Costan:
  • International Associate Team management:
    Leader of the SmartFastData Associate Team with Instituto Politécnico Nacional, Mexico City (2019–2021).
• François Tessier:
  • JLESC:
    Member of the organizing committee of the bi-annual workshops of the Joint Laboratory for Extreme Scale Computing.

10.1.6 Scientific expertise

• Alexandru Costan:
  • Estonian Research Council:
    Reviewer for the Postdoctoral and Young Researcher Grants.
  • GDR RSD:
    Member of the juries for the PhD award and Young Researcher award.
• Thomas Bouvier:
  Participated in the hackAtech as a technical coach. The goal was to promote Inria technologies by conceiving a startup project within 72 hours, from September 30th to October 2nd. "Partner" challenge prize awarded by Ouest France.

10.2.1 Teaching

• Alexandru Costan:
  • Bachelor: Software Engineering and Java Programming, 28 hours (lab sessions), L3, INSA Rennes.
  • Bachelor: Databases, 68 hours (lectures and lab sessions), L2, INSA Rennes.
  • Bachelor: Practical case studies, 24 hours (project), L3, INSA Rennes.
  • Master: Big Data Storage and Processing, 28h hours (lectures, lab sessions), M1, INSA Rennes.
  • Master: Algorithms for Big Data, 28 hours (lectures, lab sessions), M2, INSA Rennes.
  • Master: Big Data Project, 28 hours (project), M2, INSA Rennes.
• Gabriel Antoniu:
  • Master (Engineering Degree, 5th year): Big Data, 24 hours (lectures), M2 level, ENSAI (École nationale supérieure de la statistique et de l'analyse de l'information), Bruz.
  • Master: Scalable Distributed Systems, 10 hours (lectures), M1 level, SDS Module, EIT ICT Labs Master School.
  • Master: Infrastructures for Big Data, 10 hours (lectures), M2 level, IBD Module, SIF Master Program, University of Rennes.
  • Master: Cloud Computing and Big Data, 14 hours (lectures), M2 level, Cloud Module, MIAGE Master Program, University of Rennes.
  • Master (Engineering Degree, 5th year): Big Data, 16 hours (lectures), M2 level, IMT Atlantique, Nantes.
• François Tessier:
  • Bachelor: Computer science discovery, 18 hours (lab sessions), L1 level, DIE Module, ISTIC, University of Rennes.
  • Bachelor: Introduction to operating systems, 12 hours (lab sessions), L3 level, SIN Module, MIAGE Licence Program, University of Rennes.
  • Master (Engineering Degree, 4th year): Storage on Clouds, 5 hours (lecture and lab session), M2 level, IMT Atlantique, Rennes.
• Daniel Rosendo:
  • Master: Miage BDDA, 24 hours (lab sessions), M2, ISTIC Rennes.
  • Bachelor: Algorithms for Big Data, 10 hours (lectures, lab sessions), INSA Rennes.
• Thomas Bouvier:
  • Bachelor: Databases, 36 hours (lectures and lab sessions), L2, INSA Rennes.

10.2.2 Supervision

10.2.3 Juries

• Gabriel Antoniu:
  member of the PhD jury of Xenia Human, ENS Lyon, 16 December 2021.
• François Tessier:
  PhD mid-term evaluation of Amal Gueroudji, Maison de la simulation.

10.3.1 Internal or external Inria responsibilities

• Gabriel Antoniu:
  • ETP4HPC and BDVA:
    Inria representative in the working groups of BDVA and ETP4HPC dedicated to HPC-Big Data convergence.
• Luc Bougé:
  • ANR:
    Scientific officer at the National Research Agency (ANR, Agence nationale de la recherche) in charge of data management.
  • PNRIA:
    Programme National de Recherche en Intelligence Artificielle (National Research Program for Artificial Intelligence). Member of the management team for the IA for Humanity governmental program launched in March 2018. Bertrand Braunschweig, Inria, and now Isabelle Herlin, are the scientific directors of the program.
• Alexandru Costan:
  • In charge of internships at the Computer Science Department of INSA Rennes.
  • In charge of the organization of the IRISA D1 Department Seminars.
  • In charge of the management of the KerData team access to Grid'5000.

10.3.2 Interventions

• Luc Bougé:
  • ANR:
    Organization of various seminars and training about Large-Scale Data Management at National Research Agency (ANR, Agence nationale de la recherche). It focused on using Web API to access databases, for instance HAL and scanR, the search engine for French research and innovation.
• François Tessier:
  • ENS Rennes Student Seminar:
    "Dynamic Provisioning of Storage Resources on Hybrid Infrastructures", March 2021.

International peer-reviewed conferences

• 13 inproceedingsD.Daniel Balouek-Thomert, P.Pedro Silva, K.Kevin Fauvel, A.Alexandru Costan, G.Gabriel Antoniu and M.Manish Parashar. MDSC: Modelling Distributed Stream Processing across the Edge-to-Cloud Continuum.DML-ICC 2021 workshop (held in conjunction with UCC 2021)Leicester, United KingdomDecember 2021
  HALback to text
• 14 inproceedingsC.Christopher Haine, U.-U.Utz-Uwe Haus, M.Maxime Martinasso, D.Dirk Pleiter, F.François Tessier, D.Domokos Sarmany, S.Simon Smart, T.Tiago Quintino and A.Adrian Tate. A Middleware Supporting Data Movement in Complex and Software-Defined Storage and Memory Architectures.ISC 2021 SuperCompCloud - 4th Workshop on Interoperability of Supercomputing and Cloud TechnologiesFrankfurt, GermanyJuly 2021, 1-12
  HAL
• 15 inproceedingsO.-C.Ovidiu-Cristian Marcu, A.Alexandru Costan, B.Bogdan Nicolae and G.Gabriel Antoniu. Virtual Log-Structured Storage for High-Performance Streaming.Cluster 2021 - IEEE International Conference on Cluster ComputingPortland / Virtual, United StatesSeptember 2021, 1-11
  HAL
• 16 inproceedingsD.Daniel Rosendo, A.Alexandru Costan, G.Gabriel Antoniu, M.Matthieu Simonin, J.-C.Jean-Christophe Lombardo, A.Alexis Joly and P.Patrick Valduriez. Reproducible Performance Optimization of Complex Applications on the Edge-to-Cloud Continuum.Cluster 2021 - IEEE International Conference on Cluster ComputingPortland, OR, United StatesSeptember 2021
  HALback to textback to textback to text

National peer-reviewed Conferences

• 17 inproceedingsT.Thomas Bouvier, A.Alexandru Costan and G.Gabriel Antoniu. Deploying Heterogeneity-aware Deep Learning Workloads on the Computing Continuum.BDA 2021 - 37e Conférence sur la Gestion de Données - Principes, Technologies et ApplicationsProceedings of the BDA 2021 conferenceParis, FranceOctober 2021
  HALback to text
• 18 inproceedingsD.Daniel Rosendo, A.Alexandru Costan, G.Gabriel Antoniu and P.Patrick Valduriez. Enabling Reproducible Analysis of Complex Workflows on the Edge-to-Cloud Continuum.BDA 2021 - 37e Conférence sur la Gestion de Données - Principes, Technologies et ApplicationsProceedings of the BDA 2021 conferenceParis, FranceOctober 2021
  HALback to text

Other scientific publications

• 19 miscG.Gabriel Antoniu, P.Patrick Valduriez, H.-C.Hans-Christian Hoppe and J.Jens Krüger. Towards Integrated Hardware/Software Ecosystems for the Edge-Cloud-HPC Continuum.2021
  HALDOI
• 20 inproceedingsT.Thomas Bouvier, A.Alexandru Costan and G.Gabriel Antoniu. Heterogeneity-aware Deep Learning Workload Deployments on the Computing Continuum.IPDPS 2021 - 35th IEEE International Parallel and Distributed Processing SymposiumVirtual / Portland, United StatesMay 2021
  HALback to text
• 21 miscJ. C.Joshua C Bowden, F.François Tessier, C.Charles Deltel, S.Simone Bnà and G.Gabriel Antoniu. In-situ visualization using Damaris: the Code Saturne use case.PRACE: Partnership for Advanced Computing in EuropeSeptember 2021
  HALback to textback to text
• 22 inproceedingsD.Daniel Rosendo, A.Alexandru Costan, G.Gabriel Antoniu and P.Patrick Valduriez. E2Clab: Reproducible Analysis of Complex Workflows on the Edge-to-Cloud Continuum.IPDPS 2021 - 35th IEEE International Parallel and Distributed Processing SymposiumVirtual, FranceMay 2021
  HALback to text