2025Activity report​​​‌Project-TeamMEGAVOLT

Keywords

1 Team members,‌​‌ visitors, external collaborators

2 Overall objectives​​

The high-level objective of​​​‌ the MEGAVOLT team is​ to bring together an​‌ expertise on evolution equations​​ and their numerical analysis,​​​‌ with an expertise on​ machine learning (ML). Traditionally,​‌ these two communities have​​ had limited interaction; however,​​​‌ some recent works demonstrate​ that there is a​‌ large untapped potential in​​ crossing perspectives.

3 Research​​​‌ program

Our research program​ is currently structured in​‌ three major axes, with​​ sub-axes describing the specific​​​‌ objectives, as follows:

• Axis​ I. Neural network architectures​‌ as dynamics The rise​​ of deep learning, for​​​‌ example through innovations like​ skip connections in ResNets,​‌ has led to a​​ view of neural networks​​​‌ as discretized differential equations,​ offering a clearer temporal​‌ interpretation of layers. Our​​ horizons include:
  • A mathematical​​​‌ theory of Transformers. The​ introduction of Transformers in​‌ marked a turning point​​ in the artificial intelligence​​​‌ (AI) revolution, powering breakthroughs​ in natural language modeling​‌ and computer vision. With​​ remarkable practical success, Transformers​​​‌ like ChatGPT have revolutionized​ natural language and image​‌ processing. Now, as the​​ size of these models​​​‌ grows at an astonishing​ rate, the need to​‌ understand their inner workings​​ has never been more​​​‌ urgent. The temporal perspective​ on Transformers leads to​‌ an interpretation as interacting​​ particle systems: ${\stackrel{\dot{}‌}{x}}_i(t)=𝒳\left[(‌x_1\left(\mathrm{t}\right),...,‌x_n\left(\mathrm{t}\right))\right](‌x_i\left(\mathrm{t}\right)).$ The​​​‌ particles $x_i(t)\in {\mathrm{ℝ‌}}^d$ play the role​​ of tokens (fragments of​​​‌ words in language modeling,​ or patches of an​‌ image in image processing),​​ and the time variable​​​‌ $t$ again plays the​ role of a layer.​‌ In this regard, the​​ input of a Transformer​​​‌ is the sequence of​ tokens $(x_{\mathrm{1‌}}\left(0\right),...,x_{\mathrm{n‌}}\left(0\right))$, instead of a​‌ single $d$-dimensional vector​​ as in conventional neural​​​‌ networks. In a nutshell,​ the key novelty of​‌ Transformers is the introduction​​ of the self-attention mechanism​​​‌ $𝒳[({\mathrm{x}}_1,...,‌x_n)](x_i)‌={\sum }_{i=1}^n\left({\displaystyle \frac{e^{\mathrm{\beta ‌}\langle Q{x}_{\mathrm{i}},K{x}_{\mathrm{j‌}}\rangle }}{{\sum }_{\ell =1}^n{e}^{\mathrm{\beta ‌}\langle K{x}_{\mathrm{i}},Q{x}_{\mathrm{\ell ‌}}\rangle }}}\right)V{x}_{\mathrm{j}},$ which depends on​‌ the empirical measure of​​ all particles $\mu (‌t,·)=\frac{1}{n}{\sum ‌}_{i=1}^{\mathrm{n}}{\delta }_{x_i(‌t)}(·),$ and entails​‌ permutation-equivariance properties of the​​ flow.
  • Continuous/discrete modelling of​​​‌ neural ODEs and SDEs.​ Residual neural networks (ResNets)​‌ are state-of-the-art deep learning​​ models. Their continuous-depth analog,​​​‌ neural ordinary differential equations​ (neural ODEs) are widely​‌ used as an idealization​​ for which theoretical guarantees​​​‌ can be shown more​ easily. However, a rigorous​‌ mathematical framework bridging these​​ discrete and continuous realms​​ remains elusive. Establishing a​​​‌ rigorous link between these‌ two models is not‌​‌ merely a technical endeavor;​​ it promises novel insights​​​‌ into the workings of‌ ResNets. Should we establish‌​‌ that ResNets, once trained,​​ act as discretized neural​​​‌ ODEs, it would enable‌ the application of neural‌​‌ ODE findings to a​​ broad spectrum of ResNets.​​​‌ Theoretically, the prowess of‌ neural ODEs in approximation‌​‌ and the ease of​​ deriving their generalization bounds​​​‌ are well documented. Practically,‌ neural ODEs offer benefits‌​‌ like training with lower​​ memory demands and the​​​‌ potential for weight reduction,‌ addressing the critical issue‌​‌ of memory constraints in​​ residual network training. This​​​‌ inquiry marks an initial‌ step in deciphering the‌​‌ implicit regularization effects of​​ gradient descent on deep​​​‌ ResNets.

• Axis II. Training‌ dynamics of neural networks‌​‌

  Neural networks accumulate two​​ major challenges: they are​​​‌ non-convex and often heavily‌ over-parameterized, with little or‌​‌ no explicit regularization. However​​ it has been observed​​​‌ that the dynamics of‌ neural networks converge to‌​‌ prediction rules that generalize​​ well. This research axis​​​‌ proposes several paths to‌ apprehend this apparent contradiction.‌​‌

  • Incremental/dynamical learning and implicit​​ regularization. Incremental learning qualifies​​​‌ training dynamics that can‌ be decomposed into several‌​‌ phases, during which differents​​ components are learned. This​​​‌ phenomenon occurs frequently in‌ ML, but theoretical understanding‌​‌ is lacking. In this​​ section, we propose a​​​‌ theoretical path of simplified‌ models to apprehend this‌​‌ phenomenology. Actually we show​​ that incremental learning builds​​​‌ the estimator in a‌ specific way, and thus‌​‌ selects a specific solution:​​ it induces implicit regularization.​​​‌ We present an analysis‌ strategy based on heteroclinic‌​‌ dynamics.
  • Structure of​​ neural networks and function​​​‌ approximation. The existing theoretical‌ approaches for the non-convex‌​‌ dynamics of neural networks​​ study restricted regimes where​​​‌ the dynamics simplify. The‌ “neural tangent kernel” approach‌​‌ linearises the dynamics around​​ its initialization but does​​​‌ not explain variable selection‌ or, more generally, the‌​‌ excellent practical performances of​​ neural networks. The mean​​​‌ field approach only applies‌ in the limit of‌​‌ a large number of​​ neurons. The research axis​​​‌ that we propose proposes‌ to use a new‌​‌ regime, called the two-timescale​​ regime, to study​​​‌ non-linear dynamics (i) with‌ a moderate number of‌​‌ neurons and (ii) performing​​ variable selection.
• Axis III.​​​‌ Solving PDEs with ML‌

  • PINNs as a global‌​‌ approach.

    Despite notable advancements,​​ modern ML models pose​​​‌ challenges in interpretation and‌ may not adhere to‌​‌ the fundamental mathemtical laws​​ governing physical systems. Additionally,​​​‌ they often struggle to‌ extend their predictions beyond‌​‌ the scenarios they were​​ trained on. Conversely, numerical​​​‌ or purely physical methods‌ encounter difficulties in capturing‌​‌ nonlinear relationships within complex​​ and high-dimensional systems, while​​​‌ lacking adaptability and being‌ susceptible to computational issues.‌​‌ This situation has prompted​​ a growing consensus that​​​‌ data-driven ML methods should‌ be integrated with prior‌​‌ scientific knowledge rooted in​​ physics. Our focus will​​​‌ be specifically on neural‌ networks that incorporate a‌​‌ physical regularization, known as​​ PINNs.

  • VOFML as a​​​‌ local approach.

    Another line‌ of research concerns the‌​‌ extension of learning methods​​​‌ for local non-linear modules,​ which are set to​‌ boost the accuracy and​​ stability of complex physics​​​‌ simulation codes. This theme​ concerns the interaction between​‌ ML and PDE resolution,​​ which is a subject​​​‌ of very active international​ research. There is a​‌ very wide variety of​​ points of view. The​​​‌ aim is to get​ closer to the numerical​‌ simulation of hyperbolic problems.​​ The aim is to​​​‌ focus on well-targeted computational​ fluid dynamics (CFD) problems,​‌ in this case interface​​ reconstruction for volume-finite numerical​​​‌ flow reconstruction. More generally,​ complex physics codes are​‌ tricky to handle, and​​ they present many numerical​​​‌ or physical sub-mesh problems​ for which supervised learning​‌ can provide. This topic​​ is becoming increasingly active​​​‌ at Sorbonne Université and​ some interactions are natural.​‌

  • Numerical stability of neural​​ networks and time problems.​​​‌

    Stability is a central​ concept in time evolution​‌ PDE and numerical analysis,​​ but one encounters fundamental​​​‌ difficulties when it comes​ to evaluating it for​‌ methods or functions created​​ by ML. The aim​​​‌ of the research to​ be carried out is​‌ first to evaluate the​​ Lipshitz constant of functions​​​‌ created in neural networks​ with the ReLU activation​‌ function (which has interesting​​ properties), and then to​​​‌ incorporate this constant into​ the learning phase in​‌ order to enhance tje​​ stability of neural networks.​​​‌

4 Application domains

Since​ the project is oriented​‌ mostly towards methodological questions,​​ there is no specific​​​‌ applicative domain linked to​ our research.

However we​‌ aim at interacting with​​ scientific community involved in​​​‌ ML for fluids with​ the objective of developing​‌ a basis of common​​ research, and also with​​​‌ the engineering community.

5​ Social and environmental responsibility​‌

Machine learning and artificial​​ intelligence may contribute positively​​​‌ to the environment for​ example by measuring climate​‌ change effect or reducing​​ the carbon footprint of​​​‌ other sciences and activities.​ But it may also​‌ contribute negatively, notably by​​ the ever-increasing sizes of​​​‌ machine learning models. Within​ the team, we work​‌ on these two aspects​​ through our work on​​​‌ climate science and on​ frugal algorithms.

6 Highlights​‌ of the year

7 Latest software​​ developments, platforms, open data​​​‌

No software has been​ developed so far.

8​‌ New results

Participants: Raphael​​ Berthier, Bruno Després​​​‌, Borjan Geshkovski.​

Raphaël Berthier has worked​‌ on the theory of​​ neural networks by drawing​​​‌ a new connection with​ sparse regression. His work​‌ focused on diagonal linear​​ networks that are neural​​​‌ networks with linear activation​ and diagonal weight matrices.​‌ The theoretical interest of​​ these neural networks is​​​‌ that their implicit regularization​ can be rigorously analyzed:​‌ from a small initialization,​​ the training of diagonal​​​‌ linear networks converges to​ the linear predictor with​‌ minimal 1-norm among minimizers​​ of the training loss.​​​‌ In the paper 2​, RB deepened this​‌ analysis showing that the​​ full training trajectory of​​ diagonal linear networks is​​​‌ closely related to the‌ lasso regularization path. In‌​‌ this connection, the training​​ time plays the role​​​‌ of an inverse regularization‌ parameter.

Bruno Després has‌​‌ shown that the Murat-Trombetti​​ Theorem is a simple​​​‌ and efficient mathematical framework‌ for nonsmooth automatic differentiation‌​‌ of maxpooling functions. In​​ particular it gives a​​​‌ the chain rule formula‌ which correctly defines the‌​‌ composition of Lipschitz-continuous functions​​ which are piecewise ${\mathrm{C‌}}^1$. The formalism‌ is applied to four‌​‌ basic examples, with some​​ tests in PyTorch. A​​​‌ self contained proof of‌ an important Stampacchia formula‌​‌ is in the appendix​​ of the article published​​​‌ at TMLR.

Borjan Geshkovski‌ gives a precise optimization-theoretic‌​‌ and dynamical-systems interpretation of​​ a Transformer self-attention layer’s​​​‌ forward pass in the‌ “hardmax” (zero-temperature regime: in‌​‌ that limit, the token​​ update can be rewritten​​​‌ as a Frank–Wolfe (conditional‌ gradient) step for a‌​‌ quadratic objective over the​​ convex hull of the​​​‌ current token embeddings (with‌ the value matrix playing‌​‌ a preconditioning role), which​​ lets him sharply characterize​​​‌ the geometry and long-time‌ behavior of token motion‌​‌ depending on the sign​​ of the (symmetric) key–query​​​‌ matrix—showing linear contraction to‌ a single cluster at‌​‌ the origin in the​​ negative semidefinite case, and​​​‌ (after extending the rule‌ to the whole convex‌​‌ hull) a Voronoi-cell structure​​ in the positive semidefinite​​​‌ case where vertices are‌ stationary, points remain in‌​‌ their initial cells, and​​ tokens move straight toward​​​‌ the cell’s vertex with‌ (super-)exponential convergence; they additionally‌​‌ prove well-posedness of the​​ associated singular ODE limit​​​‌ and then connect back‌ to finite-temperature (softmax) attention‌​‌ by modeling it as​​ a Markov chain and​​​‌ proving a dynamic metastability‌ result: with high probability‌​‌ the finite-temperature dynamics rapidly​​ reaches and then stays​​​‌ near the hardmax “near-vertex”‌ configurations for times exponential‌​‌ in the inverse temperature​​ so the hardmax analysis​​​‌ accurately predicts behavior over‌ very long horizons before‌​‌ eventual collapse.

9 Partnerships​​ and cooperations

10 Dissemination

11 Scientific‌ production