2023Activity reportProject-TeamTOPAL

3.3.1 Use of Runtime systems

Participants: Olivier Beaumont, Aurélien Esnard, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Philippe Swartvagher.

In previous works, our main goal was to study the methodology needed to efficiently exploit the new generation of high-performance computers with all the constraints that it induces (number of cores, heterogeneity, co-scheduling effects, etc.). To achieve this goal, we successfully proposed a methodology based on the use of modern task-based runtime systems to ensure both portability and performance portability (the ability to achieve high performance by only tuning few parameters of the application). This work was done in the context of several projects (ANR Solhar, ANR SOLHARIS, Projet Région HPC Scalable Ecosystem, etc.). The work done mainly targeted single multicore nodes equipped with several accelerator devices and the extension of these techniques to the multi-node case will be the focus of our future works, especially with the arrival of Philippe Swartvagher in the team. Indeed, it has been observed that in the context of distributed nodes, the placement strategies of runtime systems are insufficient and generate too much communication. In this context, it is therefore crucial to develop efficient placement strategies  33, 26. The extension of these mixed (static/dynamic) strategies in the case of tensors is largely open.

3.3.2 Design of compression techniques

Participants: Abdou Guermouche, Mathieu Faverge, Pierre Ramet, Philippe Swartvagher.

The memory consumption of the applications has been and will remain an important challenge for solving larger problems that will lead to exascale computations. In the recent years we have demonstrated the interest of data compression techniques in linear solvers, both to save space and computations. Increasingly complex compression schemes require programming models to evolve to properly express the parallelism of these formats and to accommodate the increasing irregularity of applications. In TOPAL, we propose to continue the study of data compression techniques (low-rank, mixed precision, ...) in the context of solvers, but also in the context of training and multi-linear algebra. This part will be a very pertinent field for the study of applications over runtime systems, because of the strong irregularities that make the load balancing more complicated. At the same time, it is an original and promising approach for energy reduction. Representing convolutional / fully-connected weights in tensor formats is an effective way to reduce the parameters/FLOP in neural networks. However, post-quantization (reduction of parameters precision, for example, from float32 to int8) of networks with factorized weights yields a significant drop in accuracy. Due to memory/power consumption limitations of real devices, the quantization step is necessary, when pre-trained models are deployed. Therefore, our goal is to find algorithms that build tensorized neural networks, where weight factors are directly contain elements in low-precision format. Efficient implementation of operations on tensors represented in low-bit format will be required, as well as development of regularization techniques to tackle instability issues when training deep learning models with low-bit weights.

3.3.3 Energy minimization

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche.

Running computations with resource frugality is an important challenge, both for the upcoming exascale shift and for generally reducing the carbon impact of scientific computing. In addition to the usual objective of making computations run faster, we thus intend to design and evaluate our techniques and algorithms with the purpose of limiting their carbon footprint. In particular, given the lasting trend that the time and energy costs of computing are becoming ever lower than the costs of accessing and communicating data, we want to explore the tradeoffs of trading more computation for less data movements. This can be achieved in several ways: compression techniques as described above, replication of some computations, or use of lower precision. We are planning to work on this issue from two points of views: more frugal numerical algorithms, and energy-aware scheduling techniques. As for the embedded architectures in the phone, but also in the latest generation of laptops (Apple M1 Pro and Max chips), we are starting to see the emergence of Big-Little type technologies in the design of HPC oriented chips. In general, thermal design power (TDP) constraints push architects to increase the diversity and number of energy efficient circuits, even if they cannot all be powered simultaneously. If this hardware solution is very debatable from the point of view of carbon impact, it raises difficult and original questions about the optimization of computing performance under energy constraints. This kind of approach opens new perspectives, both from the point of view of scheduling algorithms but also in the design of computational kernels in linear algebra. We are also seeing the emergence of new processors (ARM or RISC-V technologies, Rhea from the SiPearl company within the EPI consortium, which should seriously compete with the supremacy of x86 architectures (Intel and AMD) with Nvidia accelerator cards in the search for a compromise between pure performance and energy sobriety.

In the field of training, a complementary opportunity is available. Indeed, contrary to classical HPC, the renewal of computational resources is often linked to the need to run larger models (and data with a better resolution to a lesser extent), rather than by the acceleration of computations. In this context, the possibility offered by tools such as Rotor 7.1.4 to limit memory requirements contributes to limiting the carbon footprint. Our goal is to extend the scope of these techniques, including to other fields of application than training. Our collaboration with Qarnot Computing is consistent with this objective. The co-design environment of the TextaRossa and Eupex projects 10 are also great avenues to explore these questions.

3.4.1 Task-based Linear Algebra and Tensor Computations

Participants: Olivier Beaumont, Aurélien Esnard, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Pierre Ramet, Philippe Swartvagher.

We plan to continue our activity on task-based linear algebra to find solutions for expressing high level algorithms in an elegant way while ensuring high performance. First, we want to consider the expressivity of the algorithms for large scale distributed architectures while considering the specific problems of scheduling, data and task mapping, and data granularity. This work will be done in tight collaboration with the Storm and Tadaam teams and is a key objective of the ANR SOLHARIS project. Moreover, the foundations of this topic fall back to the HiePACS project. Thus, we plan to collaborate and exchange with the CONCACE team on topics which are of interest to both teams (mainly expressivity and scalability). Second, as mentioned above, we plan to study data compression techniques in linear algebra  40, 45, 49, which brings new algorithmic schemes that are outside of the scope of the classical programming model used until now. As mid and long term objectives, we would like to find new ways to express these linear algebra algorithms to efficiently exploit large heterogeneous architectures. A second research topic focuses on the extension of the techniques developed in the framework of linear algebra, in particular with the Chameleon library, to multi-linear algebra and tensors. The idea is to build on the expertise we have in the field of compression and in the use of runtimes to use heterogeneous resources in particular.

Another challenge would be to redesign the graph partitioning & matrix ordering algorithms in a task-based runtime, in order to facilitate the integration of this basic building block in modern tasked-based solvers. This work has already been initiated in the StarPart 7.1.2 project.

3.4.2 Multi-Linear Algebra and Tensor Decompositions

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Julia Gusak, Pierre Ramet.

Tensor decompositions are a natural extension of SVD-type decompositions in linear algebra. Unlike linear algebra, there are several types of decompositions, which play an important role in the analysis of large data and in the compression of networks, in particular to increase the efficiency of inference. The arrival of Julia Gusak in the project allows us to reinforce this competence. In addition to the basic kernels to be integrated in Chameleon proposed in the Topic 3.4.1, we will propose distributed tensor decomposition algorithms compression algorithms, focusing mainly on the case of small but large tensors, which is the most common in the context of neural networks.

3.4.3 Energy Minimization in Linear Solvers

Participants: Mathieu Faverge, Abdou Guermouche.

We plan to investigate how to reduce the energy consumption of linear algebra libraries (either sparse or dense). To do so we will rely on an algorithmic approach rather than a system approach. The idea, in a first step, is to consider several implementations of a same kernel and select the implementation while taking into account energy consumption  24, 23, 25. For instance a low-rank implementation of a given operation will be slower than a regular high-performance implementation but it will tend to require less energy. In the longer term, we plan also to investigate how to design energy efficient implementations of basic kernels. They will then be used within higher level algorithms in order to find a better trade-off between energy consumption and high performance. In the context of developing linear algebra solvers using compression techniques, a research axis we would like to develop is the energy consumption study of these solvers: is it possible to provide computation kernels with different energy consumption levels that can be easily exchanged to lower the final energy consumption of the application while keeping the same numerical accuracy. Low-rank compression techniques, as well as mixed-precision solution are envisioned toward this objective.

3.4.4 Task-based Approaches for Deep Learning

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Pierre Ramet, Philippe Swartvagher.

In popular Deep Learning frameworks like TensorFlow or PyTorch, the parallelization of the training process is performed with a large granularity, mostly relying on Data Parallelism. Specialized frameworks have been proposed to explore finer parallel schemes, like PipeDream for model parallelism  51. These implementations are however very static and require explicit and error-prone data management policies. We believe that our expertise in using task-based runtime systems can be used to provide much simpler approaches for a finer grain control on the execution of the corresponding task graphs and communications patterns, for both training and inference phases. We plan to design a prototype implementation that would allow to easily use clever scheduling and optimization techniques to improve the performance of inference. In the longer term, we expect that this approach will provide better scalability and flexibility, and unlock new opportunities for optimization, for a wide range of deep learning applications.

3.4.5 Tensor Compression for Inference

Participants: Olivier Beaumont, Julia Gusak.

We envision a more exploratory research activity around the use of tensor compression for inference. Initially, the objective is to use tensor compression techniques and quantization to allow inference to be performed with little memory or low latency. These techniques can also be further extended in the context of online training performed after installation on the device itself, which requires in particular memory-saving approaches. Finally, an even more ambitious goal would be to combine these approaches with techniques for designing neural networks that are inherently efficient in terms of memory needs, such as extensions of RevNets  41, 44, 53.

3.4.6 Carbon Saving and Energy-Efficient Training

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Julia Gusak.

The training phase of Deep Neural Networks is notoriously very resource-hungry, especially regarding its energy consumption. In the last years, we have proposed several algorithmic solutions (re-materialization  27, offloading  30, their combination  28, pipelining  31) to reduce the resource consumption of this training phase, with a focus on reducing the training time. We plan to broaden the scope of these studies, by also taking into account the energy usage. A heterogeneous context and a flexible runtime system, as planned in Topic 3.4.4, may also be an opportunity to reduce energy consumption by allocating some tasks, typically the non-critical ones, to the most efficient resources for them, or by selecting a different implementation with better energy efficiency. This can be seen as a generalization of mixed-precision techniques, which are also very popular in this context to help achieving a better frugality. However, care must be taken to not degrade the convergence of the training phase. Moreover, the carbon footprint comes essentially from the manufacturing  52, 43 of the computing resources (GPUs) and the main goal is to facilitate their non-renewal, as enabled by memory saving techniques.

5.2.1 Carbon Impact of Cloud Platforms

To limit the environmental impact of Qarnot focuses on re-using the heat produced by computations in heat circuits or boilers. As part of the Pulse Inria challenge, we are working with Qarnot on algorithms for placing computations on their infrastructure, so as to maximize the use of reusable heat sources, depending on computation demand and task characteristics. The aim is to enable users of the Qarnot platform to specify their objective function on the (carbon footprint, time, cost) axes, and to be able to meet it.

5.2.2 Democratization of Large Models Training

In the context of training, at one end of the spectrum we see the provision of computing resources, such as the Jean Zay supercomputer, whose efficient use requires large-scale parallel training algorithms and frameworks to optimize resource utilization and accelerate time to discovery. At the other end of the spectrum, we see the importance of enabling researchers from different communities to use the resources at their disposal (often just a few GPUs) to develop original models without being constrained by hardware limitations. In particular, recent transformer-based models are very heavy-weight, and techniques must be employed to run them on GPUs that are only a few years old, without compromising data quality, computational accuracy, or model size. In particular, the Topal team has been working for several years on memory-saving strategies to enable the training of large models on limited-capacity resources (re-materialization and offloading), and on software 7 such as Rotor and Rockmate, which are recognized and visible in the AI applications community and enable researchers with access to limited capacity resources to train large models.

7.1.1 Chameleon

• Keywords:
  Runtime system, Task-based algorithm, Dense linear algebra, HPC, Task scheduling
• Scientific Description:

  Chameleon is part of the MORSE (Matrices Over Runtime Systems @ Exascale) project. The overall objective is to develop robust linear algebra libraries relying on innovative runtime systems that can fully benefit from the potential of those future large-scale complex machines.

  We expect advances in three directions based first on strong and closed interactions between the runtime and numerical linear algebra communities. This initial activity will then naturally expand to more focused but still joint research in both fields.

  1. Fine interaction between linear algebra and runtime systems. On parallel machines, HPC applications need to take care of data movement and consistency, which can be either explicitly managed at the level of the application itself or delegated to a runtime system. We adopt the latter approach in order to better keep up with hardware trends whose complexity is growing exponentially. One major task in this project is to define a proper interface between HPC applications and runtime systems in order to maximize productivity and expressivity. As mentioned in the next section, a widely used approach consists in abstracting the application as a DAG that the runtime system is in charge of scheduling. Scheduling such a DAG over a set of heterogeneous processing units introduces a lot of new challenges, such as predicting accurately the execution time of each type of task over each kind of unit, minimizing data transfers between memory banks, performing data prefetching, etc. Expected advances: In a nutshell, a new runtime system API will be designed to allow applications to provide scheduling hints to the runtime system and to get real-time feedback about the consequences of scheduling decisions.

  2. Runtime systems. A runtime environment is an intermediate layer between the system and the application. It provides low-level functionality not provided by the system (such as scheduling or management of the heterogeneity) and high-level features (such as performance portability). In the framework of this proposal, we will work on the scalability of runtime environment. To achieve scalability it is required to avoid all centralization. Here, the main problem is the scheduling of the tasks. In many task-based runtime environments the scheduler is centralized and becomes a bottleneck as soon as too many cores are involved. It is therefore required to distribute the scheduling decision or to compute a data distribution that impose the mapping of task using, for instance the so-called “owner-compute” rule. Expected advances: We will design runtime systems that enable an efficient and scalable use of thousands of distributed multicore nodes enhanced with accelerators.

  3. Linear algebra. Because of its central position in HPC and of the well understood structure of its algorithms, dense linear algebra has often pioneered new challenges that HPC had to face. Again, dense linear algebra has been in the vanguard of the new era of petascale computing with the design of new algorithms that can efficiently run on a multicore node with GPU accelerators. These algorithms are called “communication-avoiding” since they have been redesigned to limit the amount of communication between processing units (and between the different levels of memory hierarchy). They are expressed through Direct Acyclic Graphs (DAG) of fine-grained tasks that are dynamically scheduled. Expected advances: First, we plan to investigate the impact of these principles in the case of sparse applications (whose algorithms are slightly more complicated but often rely on dense kernels). Furthermore, both in the dense and sparse cases, the scalability on thousands of nodes is still limited, new numerical approaches need to be found. We will specifically design sparse hybrid direct/iterative methods that represent a promising approach.

  Overall end point. The overall goal of the MORSE associate team is to enable advanced numerical algorithms to be executed on a scalable unified runtime system for exploiting the full potential of future exascale machines.

• Functional Description:
  Chameleon is a dense linear algebra software relying on sequential task-based algorithms where sub-tasks of the overall algorithms are submitted to a Runtime system. A Runtime system such as StarPU is able to manage automatically data transfers between not shared memory area (CPUs-GPUs, distributed nodes). This kind of implementation paradigm allows to design high performing linear algebra algorithms on very different type of architecture: laptop, many-core nodes, CPUs-GPUs, multiple nodes. For example, Chameleon is able to perform a Cholesky factorization (double-precision) at 80 TFlop/s on a dense matrix of order 400 000 (i.e. 4 min 30 s).
• Release Contributions:

  Chameleon includes the following features:

  - BLAS 3, LAPACK one-sided and LAPACK norms tile algorithms - Support QUARK and StarPU runtime systems and PaRSEC since 2018 - Exploitation of homogeneous and heterogeneous platforms through the use of BLAS/LAPACK CPU kernels and cuBLAS/MAGMA CUDA kernels - Exploitation of clusters of interconnected nodes with distributed memory (using OpenMPI)

• URL:
  https://­gitlab.­inria.­fr/­solverstack/­chameleon
• Contact:
  Mathieu Faverge
• Participants:
  Cédric Castagnede, Samuel Thibault, Emmanuel Agullo, Florent Pruvost, Mathieu Faverge
• Partners:
  Innovative Computing Laboratory (ICL), King Abdullha University of Science and Technology, University of Colorado Denver

7.1.2 StarPart

• Keywords:
  High performance computing, HPC, Parallel computing, Graph algorithmics, Graph, Hypergraph
• Functional Description:
  StarPart is a flexible and extensible framework that integrates state-of-the-art methods for graph partitioning and sparse matrix ordering. More precisely, StarPart is a framework that offers a uniform API to manipulate graph, hypergraph and mesh structures. It is designed to be easily extensible by adding new methods and to plug all these methods into a comprehensive framework. It is initially designed to provide graph partitioning and sparse matrix ordering methods, that come from sate-of-the-art software such as Metis, Scotch, Patoh, Zoltan, etc. Besides, it provides some facilities for IO, diagnostic, benchmark, visualization (VTK, SVG, ...). StarPart is the core of the MetaPart project. It is built upon the LibGraph library.
• URL:
  https://­gitlab.­inria.­fr/­metapart/­starpart
• Contact:
  Aurelien Esnard
• Participant:
  Aurelien Esnard

7.1.3 PaStiX

• Name:
  Parallel Sparse matriX package
• Keywords:
  Direct solvers, Parallel numerical solvers, Linear Systems Solver
• Scientific Description:
  PaStiX is based on an efficient static scheduling and memory manager, in order to solve 3D problems with more than 50 million of unknowns. The mapping and scheduling algorithm handles a combination of 1D and 2D block distributions. A dynamic scheduling can also be applied to take care of NUMA architectures while taking into account very precisely the computational costs of the BLAS 3 primitives, the communication costs and the cost of local aggregations.
• Functional Description:

  PaStiX is a scientific library that provides a high performance parallel solver for very large sparse linear systems based on block direct and block ILU(k) methods. It can handle low-rank compression techniques to reduce the computation and the memory complexity. Numerical algorithms are implemented in single or double precision (real or complex) for LLt, LDLt and LU factorization with static pivoting (for non symmetric matrices having a symmetric pattern). The PaStiX library uses the graph partitioning and sparse matrix block ordering packages Scotch or Metis.

  The PaStiX solver is suitable for any heterogeneous parallel/distributed architecture when its performance is predictable, such as clusters of multicore nodes with GPU accelerators or KNL processors. In particular, we provide a high-performance version with a low memory overhead for multicore node architectures, which fully exploits the advantage of shared memory by using a hybrid MPI-thread implementation.

  The solver also provides some low-rank compression methods to reduce the memory footprint and/or the time-to-solution.

• URL:
  https://­gitlab.­inria.­fr/­solverstack/­pastix
• Publications:
  inria-00346017, inria-00346018, hal-01485507, hal-01824275, hal-03361299
• Contact:
  Pierre Ramet
• Participants:
  Alycia Lisito, Grégoire Pichon, Mathieu Faverge, Pierre Ramet

7.1.4 rotor

• Name:
  Re-materializing Optimally with pyTORch
• Keywords:
  Deep learning, Optimization, Python, GPU, Automatic differentiation
• Scientific Description:

  This software implements in PyTorch a new activation checkpointing method which allows to significantly decrease memory usage when training Deep Neural Networks with the back-propagation algorithm. Similarly to checkpointing techniques coming from the literature on Automatic Differentiation, it consists in dynamically selecting the forward activations that are saved during the training phase, and then automatically recomputing missing activations from those previously recorded. We propose an original computation model that combines two types of activation savings: either only storing the layer inputs, or recording the complete history of operations that produced the outputs (this uses more memory, but requires fewer recomputations in the backward phase), and we provide in https://hal.inria.fr/hal-02352969 an algorithm to compute the optimal computation sequence for this model.

  Our PyTorch implementation processes the entire chain, dealing with any sequential DNN whose internal layers may be arbitrarily complex and automatically executing it according to the optimal checkpointing strategy computed given a memory limit. In https://hal.inria.fr/hal-02352969, through extensive experiments, we show that our implementation consistently outperforms existing checkpoint-ing approaches for a large class of networks, image sizes and batch sizes.

• Functional Description:
  Allows to train very large convolutional networks on limited memory by optimally selecting which activations should be kept and which should be recomputed. This code is meant to replace the checkpoint.py utility available in pytorch, by providing more efficient rematerialization strategies. The algorithm is easier to tune: the only required parameter is the available memory, instead of the number of segments.
• URL:
  https://­gitlab.­inria.­fr/­hiepacs/­rotor
• Publication:
  hal-02352969
• Contact:
  Lionel Eyraud Dubois
• Participants:
  Olivier Beaumont, Alena Shilova, Alexis Joly, Lionel Eyraud Dubois, Julien Herrmann

7.1.5 StarPU

• Name:
  The StarPU Runtime System
• Keywords:
  Runtime system, High performance computing
• Scientific Description:

  Traditional processors have reached architectural limits which heterogeneous multicore designs and hardware specialization (eg. coprocessors, accelerators, ...) intend to address. However, exploiting such machines introduces numerous challenging issues at all levels, ranging from programming models and compilers to the design of scalable hardware solutions. The design of efficient runtime systems for these architectures is a critical issue. StarPU typically makes it much easier for high performance libraries or compiler environments to exploit heterogeneous multicore machines possibly equipped with GPGPUs or Cell processors: rather than handling low-level issues, programmers may concentrate on algorithmic concerns.Portability is obtained by the means of a unified abstraction of the machine. StarPU offers a unified offloadable task abstraction named "codelet". Rather than rewriting the entire code, programmers can encapsulate existing functions within codelets. In case a codelet may run on heterogeneous architectures, it is possible to specify one function for each architectures (eg. one function for CUDA and one function for CPUs). StarPU takes care to schedule and execute those codelets as efficiently as possible over the entire machine. In order to relieve programmers from the burden of explicit data transfers, a high-level data management library enforces memory coherency over the machine: before a codelet starts (eg. on an accelerator), all its data are transparently made available on the compute resource.Given its expressive interface and portable scheduling policies, StarPU obtains portable performances by efficiently (and easily) using all computing resources at the same time. StarPU also takes advantage of the heterogeneous nature of a machine, for instance by using scheduling strategies based on auto-tuned performance models.

  StarPU is a task programming library for hybrid architectures.

  The application provides algorithms and constraints: - CPU/GPU implementations of tasks, - A graph of tasks, using StarPU's rich C API.

  StarPU handles run-time concerns: - Task dependencies, - Optimized heterogeneous scheduling, - Optimized data transfers and replication between main memory and discrete memories, - Optimized cluster communications.

  Rather than handling low-level scheduling and optimizing issues, programmers can concentrate on algorithmic concerns!

• Functional Description:
  StarPU is a runtime system that offers support for heterogeneous multicore machines. While many efforts are devoted to design efficient computation kernels for those architectures (e.g. to implement BLAS kernels on GPUs), StarPU not only takes care of offloading such kernels (and implementing data coherency across the machine), but it also makes sure the kernels are executed as efficiently as possible.
• Release Contributions:
  StarPU is a runtime system that offers support for heterogeneous multicore machines. While many efforts are devoted to design efficient computation kernels for those architectures (e.g. to implement BLAS kernels on GPUs), StarPU not only takes care of offloading such kernels (and implementing data coherency across the machine), but it also makes sure the kernels are executed as efficiently as possible.
• URL:
  https://­starpu.­gitlabpages.­inria.­fr/
• Publications:
  tel-04213186, inria-00326917, inria-00378705, inria-00384363, inria-00411581, inria-00421333, inria-00467677, inria-00523937, inria-00547614, inria-00547616, inria-00547847, inria-00550877, inria-00590670, inria-00606195, inria-00606200, inria-00619654, hal-00643257, hal-00648480, hal-00654193, hal-00661320, hal-00697020, hal-00714858, hal-00725477, hal-00772742, hal-00773114, hal-00773571, hal-00773610, hal-00776610, tel-00777154, hal-00803304, hal-00807033, hal-00824514, hal-00851122, hal-00853423, hal-00858350, hal-00911856, hal-00920915, hal-00925017, hal-00926144, tel-00948309, hal-00966862, hal-00978364, hal-00978602, hal-00987094, hal-00992208, hal-01005765, hal-01011633, hal-01081974, hal-01101045, hal-01101054, hal-01120507, hal-01147997, tel-01162975, hal-01180272, hal-01181135, hal-01182746, hal-01223573, tel-01230876, hal-01283949, hal-01284004, hal-01284136, hal-01284235, hal-01316982, hal-01332774, hal-01353962, hal-01355385, hal-01361992, hal-01372022, hal-01386174, hal-01387482, hal-01409965, hal-01410103, hal-01473475, hal-01474556, tel-01483666, hal-01502749, hal-01507613, hal-01517153, tel-01538516, hal-01616632, hal-01618526, hal-01718280, tel-01816341, hal-01842038, tel-01959127, hal-02120736, hal-02275363, hal-02296118, hal-02403109, hal-02421327, hal-02872765, hal-02914793, hal-02933803, hal-02943753, hal-02970529, hal-02985721, hal-03144290, hal-03273509, hal-03290998, hal-03298021, hal-03318644, hal-03348787, hal-03552243, hal-03609275, hal-03623220, hal-03773486, hal-03773985, hal-03789625, hal-03936659, tel-03989856, hal-04005071, hal-04088833, hal-04115280, hal-04236246
• Contact:
  Olivier Aumage
• Participants:
  Corentin Salingue, Andra Hugo, Benoît Lize, Cédric Augonnet, Cyril Roelandt, François Tessier, Jérôme Clet-Ortega, Ludovic Courtes, Ludovic Stordeur, Marc Sergent, Mehdi Juhoor, Nathalie Furmento, Nicolas Collin, Olivier Aumage, Pierre Wacrenier, Raymond Namyst, Samuel Thibault, Simon Archipoff, Xavier Lacoste, Terry Cojean, Yanis Khorsi, Philippe Virouleau, LoÏc Jouans, Leo Villeveygoux, Maxime Gonthier, Philippe Swartvagher, Gwenole Lucas, Romain Lion

7.1.6 VITE

• Name:
  Visual Trace Explorer
• Keywords:
  Visualization, Execution trace
• Functional Description:
  ViTE is a trace explorer. It is a tool made to visualize execution traces of large parallel programs. It supports Pajé, a trace format created by Inria Grenoble, and OTF and OTF2 formats, developed by the University of Dresden and allows the programmer a simpler way to analyse, debug and/or profile large parallel applications.
• URL:
  https://­solverstack.­gitlabpages.­inria.­fr/­vite/
• Contact:
  Mathieu Faverge
• Participant:
  Mathieu Faverge

7.1.7 pmtool

• Keywords:
  Scheduling, Task scheduling, StarPU, Heterogeneity, GPGPU, Performance analysis
• Functional Description:
  Analyse post-mortem the behavior of StarPU applications. Provide lower bounds on makespan. Study the performance of different schedulers in a simple context. Provide implementations of many scheduling algorithms from the literature
• URL:
  https://­gitlab.­inria.­fr/­eyrauddu/­pmtool
• Publications:
  hal-01386174, hal-01878606
• Contact:
  Lionel Eyraud Dubois
• Participant:
  Lionel Eyraud Dubois

7.1.8 rockmate

• Name:
  rockmate
• Keywords:
  Deep learning, Optimization, Python, Pytorch, GPU, Automatic differentiation
• Scientific Description:

  We propose Rockmate to control the memory requirements when training PyTorch DNN models. Rockmate is an automatic tool that starts from the model code and generates an equivalent model, using a predefined amount of memory for activations, at the cost of a few re-computations. Rockmate automatically detects the structure of computational and data dependencies and rewrites the initial model as a sequence of complex blocks. We show that such a structure is widespread and can be found in many models in the literature (Transformer based models, ResNet, RegNets,...). This structure allows us to solve the problem in a fast and efficient way, using an adaptation of Checkmate (too slow on the whole model but general) at the level of individual blocks and an adaptation of Rotor (fast but limited to sequential models) at the level of the sequence itself. We show through experiments on many models that Rockmate is as fast as Rotor and as efficient as Checkmate, and that it allows in many cases to obtain a significantly lower memory consumption for activations (by a factor of 2 to 5) for a rather negligible overhead (of the order of 10% to 20%). Rockmate is open source and available at https://github.com/topal-team/rockmate.

  Complete paper: https://openreview.net/pdf?id=wLAMOoL0KD

• Functional Description:

  Given a PyTorch model, a sample input, and a GPU memory budget, Rockmate builds a new torch.nn.Module, which performs forward and backward pass while keeping the memory of activations under the given budget.

  The new model produces the same outputs and gradients as the original one. Training the model with a lower memory than PyTorch Autodiff is achieved by re-computing some of the activations instead of storing them for gradient calculation. Based on the budget, Rockmate determines automatically which activations should be recomputed.

• URL:
  https://­github.­com/­topal-team/­rockmate
• Contact:
  Lionel Eyraud Dubois

9.1.1 PhD Theses

Some on the ongoing PhD thesis are developed within bilateral contract with industry for PhD advisory:

• Airbus (2022-). This collaboration concerns the parallelization and optimization of the Flusepa application, which models the separation of boosters for space launchers at Airbus Safran Launchers. Flusepa combines computational fluid mechanics, algorithms (AMR) and task-based parallelism based on the StarPU runtime system. We are involved in the supervision of the PhD. of Alice Lasserre in this context.
• CEA-Cesta for the PhD of Clément Richefort. The aim of this thesis is to open a research work on an alternative method to domain decomposition. The basic principle of multigrid method is to use a collection of coarser problems which permit to accelerate the convergence to the fine solution. These methods are iterative with an optimal linear scalability. However, they are not efficient for oscillatory kernel problems such as electromagnetism or acoustic, which lead to indefinite matrices. The aim of this thesis is to draw up a first analysis of this method applied to indefinite Helmholtz equation, then to find the appropriates operators and finally to adapt them to Maxwell equations.
• CEA-Cesta for the PhD of Abel Calluaud. A direct solver developed at CEA relies on the approximation by hierarchical matrices to reduce both computational and memory costs. Although these developments have met a growing demand for increased simulation accuracy, there are still open problems to pursue these research efforts in an HPC context. In this thesis, we propose to develop and compare several approaches to adapt the granularity of hierarchical tasks and extract parallelism to exploit the multicore computational nodes associated with massively parallel architectures such as GPUs.

For over a year, we have been collaborating with Eviden on the development of an HPL benchmark on top of runtime systems. This work will be continued next year as part of Alycia Lisito's thesis funded by a CIFRE contract.

9.1.2 Research Engineers

We are also involved in a bilateral collaboration with Atos as part of the recovery plan, which has led in particular to the recruitment of Marc Sergent and Ahmed Abdourahmane as research engineers.

• ATOS within the framework of French Plan de Relance: Supercomputers are equipped with gas pedals to meet computing capacity requirements. Most of these accelerators are GPUs, and the computations they perform concern traditional scientific applications, but also more recently artificial intelligence training, in particular deep neural networks, and the recognition of collective communication patterns such as broadcasts. With regard to artificial intelligence and the training of deep neural networks on supercomputers, the use of the Horovod library makes it possible to distribute artificial intelligence codes based on TensorFlow2 or PyTorch on different computing nodes. Horovod then relies on an underlying MPI communication library to optimize communications during training. Given the complexity of the memory hierarchy of supercomputers, it is therefore necessary to propose and develop new collective communications adapted to these machines and capable of accelerating the training of deep-type artificial intelligence. As part of the stimulus package, ATOS and the Inria-TOPAL project-team have set up a collaboration to develop improvements to hierarchical collective communications within the Open MPI communication library, in order to accelerate the training of deep-type artificial intelligence on supercomputers with multiple types of accelerator.

10.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

ELF Associate Team on on Efficient deep Learning Frameworks.

Partners

• TOPAL
• California Institute of Technology (Caltech)

Nowadays, Deep Learning (DL) and Artificial Intelligence (AI) technologies are incorporated in more and more areas to solve various problems of video, audio, natural language processing, content generation, etc. Frameworks based on neural networks, which are core modules of deep learning models, have been already successfully used for action recognition, weather forecasting, robotic surgery and other inspiring applications [24, 44, 48]. The drawbacks of modern neural networks are that they usually require a significant amount of data and a lot of GPU devices to be trained, which makes them expensive in terms of energy and money costs, and harmful in terms of air emissions [27]. The general question we are going to address during the work of the associate team is: given your application and your computation platform, how to perform the model training efficiently in terms of time/energy?

10.2.1 Visits of international scientists

10.2.2 Visits to international teams

10.3.1 H2020 projects

10.4.1 ANR

10.4.2 Inria Challenge

11.1.1 Scientific events: organisation

• Olivier Beaumont and Emmanuel Jeannot (Tadaam team) organized the 15th JLESC workshop in Talence from March 21st to March 23rd. It gathered 128 participants from the different JLESC institutions (Inria, BSC, Jülisch, Riken, ANL, U.Tennessee, NCSA). It featured discussions and exchanges on: Artificial intelligence, Big Data, I/O and in-situ visualization, Numerical methods and algorithms, Resilience, Performance tools, Programming Languages, Advanced architectures, among others.
• Yulia Gusak and Olivier Beaumont co-organized the Workshop on Advancing Neural Network Training WANT on Computational Efficiency, Scalability, and Resource Optimization. The workshop gathered around 250 participants and focused on practically addressing challenges to enhance computational efficiency, scalability, and resource optimization, with invited talks given by researches from Oakridge, DeepSpeed, ColossalAI, NVidia and Cerebras.

11.1.2 Scientific events: selection

11.1.3 Journal

11.1.4 Invited talks

• Olivier Beaumont gave an invited talk at the 24th IEEE International Workshop on Parallel and Distributed Scientific and Engineering Computing IPDPS workshop, entitled « Exotic Data Distributions for (Symmetric Dense) Linear Algebra Kernels »
• Mathieu Faverge gave an ICL Lunch Talk on March 24th entitled Programming Heterogeneous Architectures using Hierarchical Tasks.
• Lionel Eyraud Dubois was an invited speaker at the WANT workshop of NeurIPS 2023.
• Lionel Eyraud Dubois gave a talk at the Department of Mathematical and Statistical Sciences Seminar at CU Denver, on December 4th, entitled “Optimal rematerialization algorithms for Memory-efficient learning”
• Olivier Beaumont gave an invited talk entitled "Memory Saving Techniques for Training" at the 16th Scheduling for Large Scale Systems Workshop held at The University of Tennessee in Knoxille Tennessee, May 22-May 24 2023
• Mathieu Faverge gave an invited talk entitled "Programming Heterogeneous Architectures using Hierarchical Tasks" at the 16th Scheduling for Large Scale Systems Workshop held at The University of Tennessee in Knoxille Tennessee, May 22-May 24 2023

11.1.5 Scientific expertise

• Pierre Ramet is Scientific Advisor at the CEA-DAM CESTA.
• Olivier Beaumont participated in the HCERES evaluation committee of the Mathématiques et Systèmes (MathSys) unit at the Ecole de Mines de Paris.
• Olivier Beaumont participated to the evaluation committee of the Inno4scale EuroHPC call.

11.1.6 Research administration

• Aurélien Esnard is responsible for the second year of the computer science bachelor degree (L2 Informatique), which involves managing about 200 students each year.
• Aurélien Esnard is in charge of the Numerical Transformation mission at the College ST (Science & Technology) of the University of Bordeaux. In this context, he assists the management of the College in its decisions, informs the College's advisors about the current issues in various projects, and leads a working group to propose improvements to business software for education and its administration.
• Abdou Guermouche is member of Scientific comittee of the LaBRI.
• Mathieu Faverge is member of the Study committee of Bordeaux INP.
• Pierre Ramet is the head of the CNRS Satanas department.
• Pierre Ramet is member of Scientific comittee of the LaBRI.

11.2.1 Supervision

• Defended PhD: Aboul-Karim Mohamed El Maarouf ; Parallel fine grain imcomplete LU factorization for the solution of sparse linear systems; defended March 2023; L. Giraud, A. Guermouche .
• Defended PhD: Gwénolé Lucas ; Programmation des architectures hétérogènes à l’aide de tâches divisibles; defended Oct. 2023; A. Guermouche , R. Namyst , M. Faverge , N. Furmento , P.-A. Wacrenier .
• PhD in progress: Xunyi Zhao ; Memory optimization for Deep Learning Applications; started Sept. 2020; L. Eyraud Dubois , O. Beaumont .
• PhD in progress: Jean Francois David ; Task-based inference for heterogeneous architectures; started Sept. 2020; L. Eyraud Dubois , O. Beaumont .
• PhD in progress: Clément Richefort; Multigrid methods applied to electromagnetism problems; started Nov. 2021; P. Ramet, M. Lecouvez (CEA Cesta).
• PhD in progress: Hayfa Tayeb ; Optimization of high-performance applications on heterogeneous computing nodes; started Nov. 2021; A. Guermouche , B. Bramas , M. Faverge .
• PhD in progress: Abel Calluaud; Combined compiler and runtime approach for a direct hierarchical solver; started Nov. 2022; P. Ramet, M. Faverge, D. Lugato (CEA Cesta).
• PhD in progress: Alice Lasserre; Optimisation d'un code de calcul écrit en tâche sur calculateur à mémoire distribuée; started Oct. 2022; A. Guermouche, R. Namyst (with Airbus)
• PhD in progress: Thomas Morin; Scheduling recursive task graphs; started Oct. 2023; A. Guermouche, S. Thibault

11.2.2 Juries

• Olivier Beaumont : Reviewer for the PhD Thesis of Matthias Beaupère, advised by Laura Grigori, Sorbonne University
• Olivier Beaumont : Reviewer for the PhD Thesis of Matthias Beaupère, entitled "Algorithmes parallèles pour le calcul des décompositions de rang faible des matrices et tenseurs", advised by Laura Grigori, Sorbonne University
• Olivier Beaumont : Reviewer for the PhD Thesis of Paul Youssef, entitled "Online Learning At The Edge", advised by Nguyen Kim Thang and Denis Trystram at the University of Grenoble Alpes.
• Olivier Beaumont : examiner for the PhD thesis of Redouane Elghazi "Theoretical bounds for scheduling problems and their application to asymptotic analysis and energy consumption minimization", advised by Louis-Claude Canon and Loris Marchal at ENS-Lyon.
• Aurélien Esnard : Member of the thesis jury for Hubert Hirtz (Mesh partitioning for load balancing multiphysics simulations, CEA & Univ. Paris-Saclay), defended on 12 December 2023 in Bruyères-le-Chatel.

11.3.1 Education

• As part of the science festival and the Bordeaux scientific circuit, Philippe Swartvagher lead a workshop with high school pupils about being a citizen in a digital world.
• As part of the "Fête de la rentrée" of Bordeaux University, Philippe Swartvagher presented Inria activities to students.
• As part of the SIF (Société Informatique de France) Education Day 2023, Aurélien Esnard presented feedback on project-based learning in the computer science degree course at the University of Bordeaux.
• Olivier Beaumont participated to the "Maths En Jeans" program with a group of students from the Lycée d'Andernos.

International journals

• 6 articleE.Emmanuel Agullo, A.Alfredo Buttari, A.Abdou Guermouche, J.Julien Herrmann and A.Antoine Jego. Task-based parallel programming for scalable matrix product algorithms.ACM Transactions on Mathematical Software2023HALDOIback to text
• 7 articleA.Alexandre Denis, E.Emmanuel Jeannot, P.Philippe Swartvagher and S.Samuel Thibault. Tracing task-based runtime systems: Feedbacks from the StarPU case.Concurrency and Computation: Practice and ExperienceOctober 2023, 24HALDOISoftware Heritage
• 8 articleM.Mathieu Faverge, N.Nathalie Furmento, A.Abdou Guermouche, G.Gwenolé Lucas, R.Raymond Namyst, S.Samuel Thibault and P.Pierre‐andré Wacrenier. Programming Heterogeneous Architectures Using Hierarchical Tasks.Concurrency and Computation: Practice and Experience35252023HALDOIback to text
• 9 articleA.Aboul‐karim Mohamed El Maarouf, L.Luc Giraud, A.Abdou Guermouche and T.Thomas Guignon. Combining reduction with synchronization barrier on multi‐core processors.Concurrency and Computation: Practice and Experience351January 2023, e7402HALDOIback to text

National journals

• 10 articleA.Aurélien Esnard and N.Nicolas Bonichon. Retour d'expérience sur une UE Projet en licence informatique.1024 : Bulletin de la Société Informatique de France22November 2023, 73-87HALDOI

International peer-reviewed conferences

• 11 inproceedingsE.Emmanuel Agullo, A.Alfredo Buttari, O.Olivier Coulaud, L.Lionel Eyraud-Dubois, M.Mathieu Faverge, A.Alain Franc, A.Abdou Guermouche, A.Antoine Jego, R.Romain Peressoni and F.Florent Pruvost. On the Arithmetic Intensity of Distributed-Memory Dense Matrix Multiplication Involving a Symmetric Input Matrix (SYMM).International Parallel and Distributed Processing SymposiumIPDPS 2023 - 37th International Parallel and Distributed Processing SymposiumSt. Petersburg, FL, United StatesJune 2023, 357-367HALback to text
• 12 inproceedingsO.Olivier Beaumont, J.-A.Jean-Alexandre Collin, L.Lionel Eyraud-Dubois and M.Mathieu Vérité. Data Distribution Schemes for Dense Linear Algebra Factorizations on Any Number of Nodes.Proceedings of the 37th IEEE International Parallel & Distributed Processing SymposiumIPDPS 2023 - 37th IEEE International Parallel & Distributed Processing SymposiumSt. Petersburg, Florida, United StatesIEEEMay 2023HALback to text
• 13 inproceedingsC.Clément Richefort, M.Matthieu Lecouvez, R.Robert Falgout and P.Pierre Ramet. Toward a multilevel method for the Helmholtz equation.21st SIAM Copper Mountain Conference on Multigrid MethodCopper Mountain, CO, United StatesApril 2023HALback to text

Conferences without proceedings

• 14 inproceedingsR.Robert Falgout, M.Matthieu Lecouvez, P.Pierre Ramet and C.Clément Richefort. Toward a Multigrid Method for the Indefinite Helmholtz Equation.CSE 2023 - SIAM Conference on Computational Science and EngineeringAmsterdam, NetherlandsFebruary 2023HALback to text
• 15 inproceedingsX.Xunyi Zhao, T.Théotime Le Hellard, L.Lionel Eyraud-Dubois, J.Julia Gusak and O.Olivier Beaumont. Rockmate: an Efficient, Fast, Automatic and Generic Tool for Re-materialization in PyTorch.ICML 2023Honolulu (HI), United StatesJuly 2023HALback to text

Doctoral dissertations and habilitation theses

• 16 thesisG.Gwenolé Lucas. On the Use of Hierarchical Task for Heterogeneous Architectures.Université de BordeauxOctober 2023HALback to text

Reports & preprints

• 17 reportN.Nicolas Bonichon and A.Aurélien Esnard. Retour d'Expérience sur une UE Projet en Licence Informatique: Exposé à la SIF lors des Journées Enseignements 2023..Université de bordeauxMay 2023HAL
• 18 miscJ.Julia Gusak, X.Xunyi Zhao, T.Théotime Le Hellard, Z.Zhe Li, L.Lionel Eyraud-Dubois and O.Olivier Beaumont. H-Rockmate: Hierarchical Approach for Efficient Re-materialization of Large Neural Networks.May 2023HALback to text

Other scientific publications

• 19 thesisH.Hicham Nekt. Énergie ou performance : impact de l'implémentation sur la consommation.Bordeaux-INPOctober 2023HALback to text