2023Activity reportProject-TeamEMERAUDE

Audio-to-fixed easier than float-to-fixed

In audio processing, we know that the inputs and outputs are fixed-point data, and we also have a lot of domain knowledge about audio physics. This gives serious hope that Faust audio can be compiled directly to fixed-point. This is a requirement for FPGAs, but it will also reduce the latency and power consumption on software targets if we can use their integer units. It will also enable the compilation of Faust to ultra-low-power microcontrollers without floating-point hardware.

“Domain-specific” is the key word here making us confident that a problem that is generally intractable (float-to-fixed conversion) can be addressed with little or no modification to a Faust program. The challenge here is to keep this additional work so high-level and sound-related that it is not a burden for a musician or a sound engineer. A central objective is that Faust programmers should not need to become fixed-point experts. They should actually not be anymore aware of the underlying arithmetic than they currently are with floating-point. Being high-level is a key reason for the success of Faust .

Automated error analysis for hardware computing just right

The main issue is to understand how arithmetic errors propagate, are amplified, are accumulated, etc. in a computation and in a circuit. This is called error analysis. Then a general technique 32 is to add enough bits to the right of internal fixed-point formats so that errors accumulate in these bits and the overall error accumulation does not hinder the final quality of the result. Error analysis is also managed by a worst-case combination, but here there is nothing implicit or hidden. This is therefore a comparatively well understood problem, and there is no reason to believe it cannot be fully automated in a compiler that is already able to derive the format information, building on the experience accumulated when designing complex FloPoCo operators 31, 30, 21, 85, 90.

Digital filters as arithmetic objects

Digital filters are essential components of everyday electronics like radios, mobile phones, etc., but also in audio systems of course. Their design is a core topic in digital signal processing and control theory, one that has received significant research interest for the better part of the last half century. A lot of effort has gone into constructing flexible filter design methods. For designing software-based digital filters with floating-point coefficients, there are many powerful approaches that are relatively easy to use by the filter designer (all the more as they rely on over-dimensioned floating-point operators). When designing hardware, things are not that simple for several reasons:

• algorithms developed for software-implemented filters cannot be transferred directly to hardware: what is a constraint in software (e.g., “use a 32-bit fixed-point format”) becomes a degree of freedom in hardware design (“What is the smallest fixed-point format that can be used?”);
• another degree of freedom comes from different available realization techniques to implement the arithmetic itself, for instance the construction of multipliers by constants.

A consequence is that popular tools, such as the popular fdatool (filter design and analysis tool) from Matlab's Signal Processing toolbox, offer a complex interface, requiring a tedious hand-tuning process, and expect some domain expertise. Such tools input a frequency response, and decompose the filter implementation problem in three steps: 1/ the filter design (FD) step consists in finding a filter with ideal (high precision) coefficients that adheres to the frequency response; 2/ the quantization (Q) step converts the obtained coefficients to hardware-friendly fixed-point values ; 3/ the implementation (I) step generates a valid hardware description (e.g., a VHDL or Verilog description) using the quantized coefficients.

The objective of this research axis is to offer an optimal solution to the global FD + Q + I problem. Optimal techniques exist for each of the FD, Q and I steps in isolation. The combination of the FD & Q steps have been studied since the 1960's   48, and can even be regarded as solved for certain practical instances of fixed-point Finite Impulse Response (FIR) design 49. A large body of work also exists for the I step, with recent optimal approaches  13, 50, 51. However, these approaches are only optimal for a given set of coefficients, and therefore strongly depend on the FD and Q steps.

Arithmetic-oriented scheduling and tiling for low-latency audio

Finally, we also want to formally insert arithmetic considerations in the global problem of distributing a very heavy computation between space (we have up to several thousands multipliers in an FPGA, and many more if we multiply by a constant) and time (we have thousands of FPGA cycles within one audio cycle). These are well-researched compilation issues, called the scheduling and tiling problems. There is local expertise in Lyon (in particular in the CASH team and its spin-off XtremLogic21) who have worked on these problems for FPGAs. However, scheduling and tiling techniques so far consider each operation as having a standard, constant cost (e.g., multiplication costs $c_m$ and has latency $t_m$, addition costs $c_a$ in space and $t_a$ in time). This is a very crude simplification if one attempts to optimize each operator, think for multiplications by constant for instance. The availability of many audio-related case studies in Emeraude will allow us (hopefully in collaboration with CASH) to develop arithmetic-aware scheduling and tiling techniques that will eventually prove useful well beyond the world of digital audio.

Physical modeling

One of the most active areas of research in audio DSP is physical modeling of musical instruments, mechanic vibrations and analog electronic circuits (also called “virtual analog”). Because these techniques have a direct link with the real acoustical-world, they make a lot of sense in the context of Emeraude which develops skills in embedded systems, hardware and electronics.

While waveguide 82, mass-interaction 25, and modal models 12 are relatively well understood and can be easily implemented in Faust , we would like to focus on algorithms using the Finite-Difference Time-Domain (FDTD) method 18, 17. Similarly, we would like to work on Wave Digital Filter (WDF) algorithms 37 in the context of the modeling of analog electronic circuits for audio processing and synthesis 92. This is a very hot topic 91.

Virtual acoustics, acoustic active control, and spatial audio

A large portion of the tools developed as part of Emeraude target real-time ultra-low-latency audio (i.e., FPGAs, etc.). Some of the application areas for this type of systems are virtual room acoustics (e.g., artificial reverberation, etc.), acoustic active control, and spatial audio (i.e., Ambisonics  39 and binaural  66).

Artificial reverberation can be implemented using a wide range of techniques such as feedback delay networks  44, waveguide meshes 74, finite difference schemes  75, simple combination of IIR filters 78, and Impulse Responses (IR) convolution 87. Convolution reverbs have been taken to another step recently by introducing the concept of “modal reverb” 11 where convolution is carried out in the time domain instead of the frequency domain by using a bank of resonant bandpass filters taking advantage of the principle of modal decomposition  12. While similar experiments have been prototyped on GPUs 81, our proposal it to take this to the next level by providing ultra-low latency and active control of the acoustics of the room where the sound will be rendered.

Active control of room acoustics is a challenging topic with the first commercial applications dating back to the 90s 40. The objective of this theme is to vary at will the subjective and quantifiable acoustic parameters of a room (e.g., reverberation time, early reflections, loudness, etc.) in order to adapt it to the artistic performance and the venue 71. To reach this goal, several loudspeakers, microphones and DSP modules are used to create artificial reflections in a non-generative way  47. Several commercial systems are available on the market such as CSTB's CARMEN 76 or Yamaha's AFC system 2265 to name a few. Other examples can be found in 71. In such systems, the feedback loops should be controlled, as well as the system stability, and real-time DSP.

Recent advances in active control and approaches to sound field decomposition 54 allow us to partially overcome this limitation, by proposing “spatial active noise contol” algorithms 93, 43. As one can imagine, such approaches require a very large number of control channels with very low latency. While Emeraude investigates the technological solutions to implement such algorithms in a real situation, we also hope to make contributions to the field of acoustic active control in general.

Machine learning for digital signal processing

Machine learning and deep learning in particular, are playing an increasingly important role in the field of audio DSP. Researchers are revisiting the algorithmic techniques of signal synthesis and processing in the light of machine learning, for instance for speech processing 41. Recent breakthroughs such as the use of machine learning use in the context of Differentiable Digital Signal Processing (DDSP) 36 demonstrate its power. Hybrid approaches combine classical signal processing with deep learning to obtain CPU efficient implementations like in the RNNoise project.23

One of our goals is to integrate these new developments to the Emeraude ecosystem and to further explore their potential applications in the context of embedded audio processing.

The Faust language and its compiler

A large part of Emeraude's research results is visible thanks to the Faust ecosystem development. Faust has gained an international recognition, especially since it is used for teaching at Stanford University (in the context of courses on signal processing, physical interaction design, etc.) and developing new audio plugins  64. The efforts needed to keep Faust as the most efficient language for real-time audio processing involve research in: language design, compiler design, and development of DSP libraries.

One of the reason of Faust 's success is that it is both a language and an environment for audio signal processing. The Faust compiler typically generates high-level codes (in languages such as C, C++, etc.), following every compiler's goal: providing better code than manually written code. For that, it has to stick to the most recent compiler technologies and processors evolutions  58. For instance, a back-end for WebAssembly was recently added to the Faust compiler  57. An important deployment step was the embedding of the Faust compiler in a web browser  56 which makes it easily accessible on all computers.

Faust language design research in Emeraude

The current design of Faust , inspired by lambda-calculus, combinatory logic and John Backus’ functional programming, has to be extended to face new challenges, in particular multi-dimensional and multi-rate signals and linear algebra.

Faust allows for the description of synchronous mono-rate scalar DSP computations. This is sufficient to implement most time-domain algorithms such as filters, oscillators, waveguides, etc. However, this makes the implementation of frequency-domain algorithms (e.g., FFT, convolution, etc.) very inefficient, not to say impossible. One of our goals is to extend the language to enable multi-rate as well as vector computations. While we already have a working prototype for this, some challenges have yet to be overcome.

Along the lines of the previous point, Faust currently doesn't provide any support for efficient matrix operations and more generally linear algebra. This prevents the implementation of some classes of DSP algorithms such as Finite-Difference Time-Domain (FDTD) method for physical modeling. The skills of former Socrate members on seminal Alpha language  33 and polyhedral optimization are very useful here.

Support for the main target programming languages in Faust is essential. Recently added languages (WebAssembly, Rust, and SOUL) have opened many new opportunities. The FPGA target, studied in §3.1, introduces new challenges such as the ability to use fixed-point arithmetic or the use of HLS for targeting hardware platforms (e.g., VHDL, Verilog, etc.). Other “exotic” architectures such as GPUs or MPSoCs should be studied for compute-bound algorithms.

Musicians have to deal with a large variety of operating systems, software environments and hardware architectures. Faust is designed to favor an easy deployment of Faust programs on all these targets by making a clear separation between computation itself, as described by the program code, and how this computation should be related to the external world. This relation (with audio drivers, GUIs, sensors, etc.) is described in specific architecture files38. Architecture files concern both hardware (i.e., audio interfaces/sound cards) as well as software control interfaces (e.g., GUI, OSC,24 MIDI), new luthieries (e.g., SmartFaust, Gramophone), Web platforms (Web audio Plugin), etc. One of the goal of the work of Emeraude on Faust is to ease the programming of these audio systems.

Faust ecosystem and DSP libraries

Faust users are very attached to its ecosystem, including native applications, online and “embedded” audio applications, Just In Time (JIT) compiler, etc. Recent developments include a JIT Faust compiler on the Web, a JIT compiler in the Max/MSP environment, tools to find the best compilation parameters and ease compilation for multiple CPUs. This is constantly evolving to answer to users' demand.

The Faust DSP libraries currently implement hundreds of functions/objects ranging from simple oscillators and filters to advanced filter architectures, physical models, and complete ready-to-use audio plugins. These libraries are at the heart of Faust 's success and international position. Julius Smith25 (Stanford professor) is one of the most respected figures in the field of audio DSP and one of the main contributors to the Faust libraries. One of the ambitions of the Emeraude team is to maintain and extend this tool to make it as exhaustive and as universal as possible. Along these lines, new developments made to the language presented above (e.g., multi-rate, linear algebra, etc.) should be ported to the libraries. Finally, dedicated libraries targeting specific hardware platforms (e.g., microcontrollers, FPGAs) should be made available too.

Embedded systems for audio processing

As Emeraude's name suggests it, the implementation of audio Digital Signal Processing on embedded hardware is at the heart of the project. We naturally rely on the Faust language for these implementations. The skills of Emeraude members in compilation and embedded systems are used to add new embedded target for audio processing, in particular FPGAs, as explained previously. This action is a mix of research and engineering work, it should be very useful for the dissemination of audio processing programming.

Haptics is a huge topic, especially in the field of New Interfaces for Musical Expression (NIME), which has been studied for many years 27, 88. It has always been tightly coupled to physical modeling because this sound synthesis technique provides natural connections to the physical world. A big part of the challenge is technological because haptics requires ultra low-latency and high sampling resolution in order to be accurate. This is at the heart of Emeraude 's goals.

Virtual and Augmented Reality (VR/AR) is not limited to immersive 3D graphics, sound also has an important role to play in that emerging field. Lots of work has been done around using VR environments as a creative tool for audio  53, 24, 22. While many VR-based musical instruments have been created in the past 79, little work has been done around implementing interfaces specifically targeting VR/AR audio environments, especially in the context of 3D sound. This is something that we plan to explore as part of Emeraude.

Finally, beside ergonomic and HCI aspects, the design of musical interfaces is impacted by various kinds of technical limitations that we plan to address as part of Emeraude. First, just like for real-time audio processing, latency plays a crucial role in this context. Similarly, the “time resolution” (e.g., the sampling rate of the interfaces) can have a huge impact, especially when targeting specific kinds of instruments such as drums. Finally, the “spatial resolution” (e.g., the number of sensor points per squared centimeters on a tabletop interface) also impacts its quality. In this context, we would like to develop an embedded, high-resolution, high-sampling-rate, multi-touch multi-dimensional (X and Y + pressure) interface/instrument leveraging the development carried out in the previous axes. This work would be followed by a user study to measure the impact of this type of advanced system on perception.

6.1.1 FloPoCo

• Name:
  Floating-Point Cores, but not only
• Keyword:
  Synthesizable VHDL generator
• Functional Description:
  The purpose of the open-source FloPoCo project is to explore the many ways in which the flexibility of the FPGA target can be exploited in the arithmetic realm.
• URL:
  http://­flopoco.­org
• Contact:
  Florent de Dinechin
• Participants:
  Florent de Dinechin, Luc Forget
• Partners:
  ENS Lyon, Insa de Lyon, Inria, Fulda University of Applied Science

6.1.2 hint

• Name:
  High-level synthesis Integer Library
• Keyword:
  High-level synthesis
• Functional Description:
  Hint is an header-only arbitrary size integer API with strong semantics for C++. Multiple backends are provided using various HLS libraries, allowing a user to write one operator and synthetize it using the main vendor tools.
• URL:
  https://­github.­com/­yuguen/­hint
• Publication:
  hal-02131798v2
• Contact:
  Luc Forget
• Participants:
  Yohann Uguen, Florent de Dinechin, Luc Forget

6.1.3 marto

• Name:
  Modern Arithmetic Tools
• Keywords:
  High-level synthesis, Arithmetic, FPGA
• Functional Description:

  Marto provides C++ headers to implement custom sized arithmetic operators such as:

  Custom sized posits and their environment (including the quire) Custom sized IEEE-754 numbers Custom sized Kulisch accumulators (and sums of products)

• URL:
  https://­gitlab.­inria.­fr/­lforget/­marto
• Publication:
  hal-02130912v4
• Contact:
  Yohann Uguen
• Participants:
  Yohann Uguen, Florent de Dinechin, Luc Forget

6.1.4 Syfala

• Name:
  Low-Latency Synthesizer on FPGA
• Keywords:
  FPGA, Compilers, High-level synthesis, Audio signal processing
• Functional Description:

  The goal of Syfala is to design an FPGA-based platform for multichannel ultra-low-latency audio Digital Signal Processing programmable at a high-level with Faust and usable for various applications ranging from sound synthesis and processing to active sound control and artificial sound field/room acoustics.

  A series of tools are currently being developed around SyFaLa. While none of them has been officially released yet, you can follow their development/evolution on the project Git repository: https://gitlab.inria.fr/risset/syfala

• URL:
  https://­faust.­grame.­fr/­syfala/
• Contact:
  Tanguy Risset

6.1.5 FAUST

• Name:
  Functional Audio Stream)
• Keywords:
  Audio, Functional programming
• Functional Description:

  The core component of Faust is its compiler. It allows to "translate" any Faust digital signal processing (DSP) specification to a wide range of non-domain specific languages such as C++, C, LLVM bit code, WebAssembly, Rust, etc. In this regard, Faust can be seen as an alternative to C++ but is much simpler and intuitive to learn.

  Thanks to a wrapping system called "architectures," codes generated by Faust can be easily compiled into a wide variety of objects ranging from audio plug-ins to standalone applications or smartphone and web apps, etc.

• URL:
  https://­faust.­grame.­fr/
• Contact:
  Yann Orlarey
• Partners:
  GRAME, Insa de Lyon, Inria

7.1.1 C++ optimizations in the context of High-Level Synthesis

Participants: Maxime Popoff, Tanguy Risset, Romain Michon, Pierre Cochard.

When compiling Faust programs to FPGA, Syfala relies on the High Level Synthesis (HLS) tool provided by Xilinx, which takes a C++ program as an input. Hence, Faust generates C++ code from a Faust program and Syfala feeds it to HLS. The topology of the C++ code provided to HLS has a huge impact on the performances of the generated Intellectual Property (IP). In 2023, we conducted a study aiming at understanding the kind of optimizations that can be carried out on C++ code in the context of the high-level synthesis of real-time audio DSP programs. Thanks to this work, we managed to significantly optimize the applications generated by Syfala allowing for much more complex audio DSP algorithms to be run on the FPGA. While these findings haven't been integrated to the Faust Syfala backend, they can be used with the new Syfala C++ support. Indeed, we recently added a new mode in Syfala allowing for C++ code to be used as a substitute for Faust . This, combined with an exhaustive public documentation of the aforementionned optimizations will help increasing the attractivity of Syfala.

This work will be published in 2024.

7.1.2 Linux support for Syfala

Participants: Pierre Cochard, Maxime Popoff, Antoine Fraboulet, Tanguy Risset, Stephane Letz, Romain Michon.

Up to now, the CPU portion of applications generated by Syfala was implemented as a bare-metal kernel. In 2023, we added the possibility to run Alpine Linux on the CPU of the Zybo board while carrying out audio DSP operations on the FPGA, taking a hardware accelerator approach. This enables the compilation of complete audio applications involving various control protocols and approaches such as OSC (Open Sound Control) through Ethernet or Wi-Fi, MIDI, web interfaces running on an HTTPD server, etc. It also opens the door to the integration of hardware accelerators in high-level computer music programming environments such as Pure Data, SuperCollider, etc.

This work led to a publication at the 2023 Sound and Music Computing conference (SMC-23)  1.

7.1.3 Syfala PipeWire support

Participants: Jurek Weber, Pierre Cochard, Tanguy Risset, Romain Michon.

As we worked on applications for Syfala requiring the handling of a large number of audio channels in parallel for spatial audio, we realized that we needed a way to send and receive audio streams in parallel between a laptop computer and our FPGA board. For this, we opted for an open standard named PipeWire which allows for the transmission of digital audio streams in real-time over an ethernet connection. PipeWire was implemented in the Linux layer of Syfala (see §7.1.2) and is now perfectly integrated to the toolchain. It will allow to significantly expand the scope of the various spatial audio systems that we've been working on in the context of PLASMA (see §7.2)

This work will be published in 2024.

7.1.4 Multichannel audio boards for FPGA

Participants: Maxime Popoff, Romain Michon, Tanguy Risset.

We developed two audio FPGA sister boards aiming various kinds of spatial audio applications.

One targets the Digilent Zybo Z7 (10 or 20) board and is designed to be cost-efficient, accessible, and easily reproducible (see Figure 6). It provides 32 amplified (3W) audio outputs to which small speakers can be directly connected. Its goal is to provide an affordable way to deal with a large number of audio outputs in the context of spatial audio. It was used at the heart of the frugal spatial audio system described in §7.2.2.

Affordable sister board.

The other board that we developed is meant to be connected to a Digilent Genesys board and targets high-end spatial audio applications with a strong focus on active control (see Figure 7). It provides 32 ultra-low latency (10$\mu s$) balanced inputs and outputs, leveraging the work presented in   72. It is currently used as part of the FAST ANR project for implementing FxLMS algorithms for active control.

Sister board with 32 inputs and outputs.

This work will be published in 2024.

7.1.5 Faust to VHDL backend

Participants: Jessica Zaki Sewa, Alois Rautureau, Pierre Cochard, Tanguy Risset, Yann Orlarey, Stéphane Letz.

Syfala uses HLS for compiling C++ code downto VHDL, the C++ code being itself generated from Faust . However, Faust , as a functionnal langage, exhibits all the parallelism of the audio application. The code is sequentialized in the C++ code and then re-parallelized by the viti_hls tool for the FPGA.

An interesting alternative is to translate directly Faust downto VHDL. Faust programs can be represented as audio circuits connected together and hence provides a natural equivalence with VHDL structural representation of such circuits. The VHDL program is just a translation of the data-flow graph of the audio application.

However, for an efficient implementation on FPGA, this data-flow graph must be retimed. Retiming  55 is an old classical transformation that adds registers in a digital circuit without changing its functionnal behaviour but allowing for a much faster clock rate.

A first Faust2VHDL translator prototype was issued in 2022 generating a fully combinatorial datapath on the FPGA. In 2023 we have released the first real Faust2FPGA compiler which includes retiming and fixed point computations.

This work will be published in 2024. Preliminary results shows that the IP generated by our Faust2FPGA compiler are twice smaller than the IP generated by viti_hls. However, the use of HLS is still prefered because many features are not included in the Faust2FPGA compiler (i.e., control from the ARM processeur or use of the external RAM).

7.2.1 Distributed spatial audio system

Participants: Thomas Rushton, Romain Michon, Tanguy Risset, Stéphane Letz.

We developed a distributed systems for spatial audio DSP based on a network of microcontrollers (see Fig. 8). We chose this type of platform because they are cost-effective, very lightweight, and OS-free. We used PJRC's Teensys 4.127 as they host a powerful Cortex M7 clocked at 600 MHz as well as built-in ethernet support. PJRC also provides an “audio shield” which is just a breakout board for a stereo audio codec (SGTL-5000) compatible with the Teensy 4.1.

A preliminary task was to send audio streams over the Ethernet from a laptop to the Teensy. For that, we decided to use the JackTrip protocol which is open source and used a lot in the audio/music tech community  23. Implementing a JackTip client on the Teensy was fairly straightforward. We then implemented our own protocol which can be easily accessed through an audio plugin directly runnable in a Digital Audio Workstation (DAW).

Audio DSP is carried out directly on the Teensys which are programmed with the Faust programming language thanks to the faust2teensy  62 tool developed by the Emeraude team. A laptop is used to transmit audio streams to the Teensys which are controlled using the Open Sound Control (OSC) standard. OSC messages are multicast/broadcast to save bandwidth. The same audio streams are sent to all the Teensys in the network (all audio processing is carried out on the Teensys, not on the laptop computer).

This work was presented at the SMC-23 conference  6.

Wave Field Synthesis system based on a network of microcontrollers.

7.2.2 Frugal FPGA-based spatial audio system

Participants: Romain Michon, Joseph Bizien, Pierre Cochard, Tanguy Risset.

As a first step towards implementing a centralized system for spatial audio, we worked on a low-cost Wave Field Synthesis (WFS)  15 (see Figure 9) system based on an affordable sister board for the Digilent Zybo Z7 FPGA board that we developed (see §7.1.4).

We successfully ran a standard WFS algorithm implemented in the Faust programming language on this system where multiple sources can be moved in space. Individual sources are sent to the speaker array thanks to the Syfala PipeWire support (see §7.1.3). Their position can be controlled using a web interface accessible through an HTTPD server or OSC.

Thanks to the use of the FPGA, the cost of each new channel in the system is very low. Implementing a WFS system with a more traditional architecture would be way more expensive that what we achieved which is very promising!

This work was presented at the NIME-23 conference  8.

The PLASMA FPGA-based WFS system

7.3.1 Extension to the Faust language with widget modulation

Faust favors the reuse of existing code via several mechanisms, particularly through the component() expression. Utilizing component("freeverb.dsp"), for instance, facilitates the integration of a comprehensive reverb program, as specified in the freeverb.dsp file, along with its user interface, into a programmer's own application. However, a challenge arises when there is a need to modify the user interface of such a component. Consider scenarios like replacing a slider with a constant value, or employing an audio signal to modulate a slider's value. Previously, such modifications necessitated alterations to the source code of freeverb.dsp.

The conventional approach to this limitation was, for the developer of freeverb.dsp, to structure the code such that the audio processing component is distinct from its user interface. This practice is advantageous as it allows for the audio processing component to be reused in various contexts or with different user interfaces. Nonetheless, it does not ease the reuse of any part of the existing user interface.

The introduction of a new feature in Faust version 2.69.0, termed widget modulation, addresses this issue. This feature enables modifications to an existing component, such as altering a slider's functionality, replacing a slider, or setting a slider to a fixed value, all without requiring changes to the component's original source code. Programmers specify the desired modifications to the user interface, and the compiler generates a modified version of the code that implements these changes.

7.3.2 Fixed-point extension in the Faust programming language

In this axis of the FAST project, we are developing a fixed-point extension to Faust to optimize the performance of Faust programs on FPGAs.

The already existing interval library, which makes it possible to annotate signals with their range (minimum and maximum value they can attain) has been extended with a precision property. The precision denotes the number of bits that needs to be used to represent a signal, with the implicit goal of balancing signal quality (using enough bits to preserve information) and resource economy (using no more bits than what is needed).

The strategy to determine this precision is theoretically founded on the novel notion of pseudo-injectivity, introduced in the context of that work. This notion is attached to a function and its definition interval, and links the input and output precisions. More precisely, a function defined on a given interval and equipped with input and output precisions is said to be pseudo-injective if all images of representable inputs, in the given input format, are distinctly representable in the output format. We established a closed-form formula to determine the output format from an input format and vice-versa (for a given function and definition interval).

This information can be forwardly propagated into the signal graph, in the sense that each node receives the precision and interval information of its inputs, and uses them to infer the output interval and precision of the function. The global propagation is done as a compiler pass, in the same way that other properties are propagated: the operation starts from the inputs of the program (for which the precision is typically set by external factors) and traverses the program graph until it reaches the outputs.

Preliminary experiments towards backwards propagation of these signals have been conducted, without being implemented into the compiler for now. This backwards propagation consists in traversing the program graph from the outputs to the inputs, inferring the input precision of each node from its output precision in a way that preserves pseudo-injectivity. Early results seem to indicate that performing a step of backward propagation results in smaller precisions than forward propagation alone, without sacrificing the overall quality of the signal.

Once the precision and interval has been inferred for every signal, it is then used to generate C++ code, using the ap_fixed library from Xilinx for fixed-point formats. The position of the most significant bit of the format is determined by the amplitude of the interval, and the position of the least significant bit is given by the precision.

This work will be published in 2024.

7.3.3 faust2rnbo project

RNBO is a library and toolchain that can take Max-like patches, export them as portable code, and directly compile that code to targets like a VST, a Max External, or a Raspberry Pi. DSP programs can be compiled to the internal codebox  sample level scripting language. Compiling Faust DSP to codebox  code allows us to take profit of hundreds of DSP building blocks implemented in the Faust libraries, ready to use examples, any DSP program developed in more than 200 projects listed in the Powered By Faust page, or Faust DSP programs found on the net.

A new backend to produce the codebox language has been added to the compiler. The codebox code can be generated using the following line (note the use of -double option, the default sample format in RNBO/codebox ):

faust -lang codebox -double foo.dsp -o foo.codebox

7.3.4 Other developments around Faust

7.4.1 Hardware-optimal digital FIR filters

Design and implementation of a hardware FIR filter from frequency constraints

The article 7 addresses the implementation of Finite Impulse Response filters as digital hardware circuits (Figure 10). It formalizes, as a mathematical model, the problem of finding the optimal circuit for a given frequency specification and given input/output fixed-point formats. This model captures at the bit level a wide class of implementations (transposed-form circuits based on truncated shift-and-add adder graphs). It also captures formally the constraints due to the frequency specification, as well as those due to rounding to the output format. This model can be expressed as an Integer Linear Programming (ILP) problem, such that the optimal circuit (in terms of bit-level adders and registers) can be found by standard ILP solvers. This approach allows for a completely automatic tool from a frequency specification to a circuit with user-specified input and output formats. This tool is evaluated (with cost functions modeling FPGAs) on several benchmarks.

7.4.2 High-performance floating-point hardware and their applications

The work in this section took place in the framework of the CIFRE thesis of Oregane Desrentes with Kalray.

Low bit-width floating-point formats appear as the main alternative to 8-bit integers for quantized deep learning applications. The article 3 proposes an architecture for exact dot product accumulate operators and compares its implementation costs for different 8-bit floating-point formats: FP8 with five exponent bits and two fraction bits (E5M2), FP8 with four exponent bits and three fraction bits (E4M3), and Posit8 formats with different exponent sizes. The front-ends of these exact dot product accumulate operators take 8-bit multiplicands, expand their fullprecision products to fixed-point, and sum terms into wide accumulators. The back-ends of these operators round down the wide accumulators contents first to FP32 and then to one of the 8-bit floating-point formats. The proposed 8-bit floating-point exact dot product accumulate operators are synthesized targeting the TSMC 16FFC node in order to compare their area and power to a baseline of operators with FP16 and INT8 multiplicands.

For larger precisions, the article 2 explores architectures of exact (correctly rounded) fused dot product and add operators suitable for the FP32 and FP64 binary floating-point representations with subnormal support, and other representations with a wide dynamic range such as bfloat16. The exact summation of terms before rounding requires a full-size accumulator, and this work discusses techniques to compress the identical bits of this accumulator. This requires the computation of the relative shift amounts of the terms, which is formulated as a parallel prefix algorithm, allowing for a low-latency implementation. Architectural options for the exact fused dot product and add operators with up to 16 products for FP32, FP64 and mixed-precision BF16 to FP32 are evaluated using the TSMC 16FFC technology node.

The article 4 revisits 1D Fast Fourier Transforms (FFT) implementation approaches in the context of compute units composed of multiple cores with SIMD ISA extensions and sharing a multi-banked local memory. A main constraint is to spare use of local memory, which motivates us to use in-place FFT implementations and to generate the twiddle factors with trigonometric recurrences. A key objective is to maximize bandwidth of the multi-banked local memory system by ensuring that cores issue maximum-width aligned non-temporal SIMD accesses. We propose combining the SIMD lane-slicing and sample partitioning techniques to derive multicore FFT implementations that do not require matrix transpositions and only involve one stage of bit-reverse unscrambling. This approach is demonstrated on the Kalray MPPA3 processor compute unit, where it outperforms the classic six-step algorithm for multicore FFT implementation.

9.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

• Title:
  Pushing the Limits of Audio Spatialization with eMerging Architectures (PLASMA)
• Partner Institution(s):
  • Stanford University Stanford (USA)
• Date/Duration:
  2022 -> 2025
• Additionnal info/keywords:
  PLASMA is an associate research team gathering the strength of Emeraude and of the Center for Computer Research in Music and Acoustics (CCRMA) at Stanford University. The two main objectives of Plasma are: (i) Exploring various approaches based on embedded systems towards the implementation of modular audio signal processing systems involving a large number of output channels (and hence speakers) in the context of spatial audio; (ii) Making these systems easily programmable: we want to create an open and accessible system for spatial audio where the number of output channels is not an issue anymore. Two approaches are being considered in parallel: (i) Distributed using cheap simple embedded audio systems (i.e., Teensy, etc.); (ii) Centralized using an FPGA-based (Field-Programmable Gate Array) solution.

9.2.1 Visits of international scientists

9.2.2 Visits to international teams

9.3.1 ANR FAST

Embedded systems for audio and multimedia are increasingly used in the arts and culture (e.g., interactive systems, musical instruments, virtual and augmented reality, artistic creation tools, etc.). They are typically based on a CPU (Central Processing Unit) which limits their computational power and induces some latency. FPGAs (Field Programmable Gate Arrays) can be seen as a solution to these problems. However, these types of chips are extremely complex to program, making them largely inaccessible to musicians, digital artists and makers communities.

The goal of the FAST ANR project is to enable high-level programming of FPGA-based platforms for multichannel ultra-low-latency audio processing using the Faust programming language (a standard in the field of computer music). We plan to use this system for various applications ranging from sound synthesis and processing to active sound control and artificial sound field/room acoustics.

FAST officially started in March 2021. It gathers the strength of GRAME-CNCM, CITI Lab (INRIA/INSA Lyon), and LMFA (École Centrale Lyon).

10.1.1 Scientific events: organisation

• Romain Michon and Stephane Letz organized the 2023 Programmable Audio Workshop (PAW-23: paw.grame.fr) which is a one day workshop on emerging programmable audio technologies (see §5.1). The theme of this event this year was “Artificial Intelligence and Audio Programming Languages” It took place at INSA Lyon on December 2, 2023.
• The Syfala Team (Tanguy Risset, Romain Michon, Maxime Popov, Pierre Cochard, Yann Orlarey) with the help of Matthieu Imbert have organized the first "Syfala Workshop" using Grid5000 facility. 10 persons were assisting, 6 from abroad (Germany, Switzerland, Poland, Stanford). We show how to use the open-source Syfala compiler to compile Faust programs.
• Anastasia Volkova organized a one-day workshop on Optimization for Hardware Arithmetic Architectures on October 26th in Nantes, France. 15 persons from University of Lincoping (Sweden), CEA Paris, CNRS, Inria Rennes, Inria Lyon and INSA Lyon participated. Potential collaborations in the fields of machine learning and digital signal processing acceleration were discussed and planned.

10.1.2 Scientific events: selection

• Romain Michon was a member of the conference program committee of NIME-23 (www.nime2023.org/).
• Romain Michon was a member of the conference program committee of DAFx-23 (dafx23.create.aau.dk/).
• Romain Michon was a member of the conference program committee of SMC-23 (smcnetwork.org/smc2023/).
• Tanguy Risset was a member of the conference program committee of SMC-23 (smcnetwork.org/smc2023/) and DATE 2023 (Design Automation and Test in Europe), on track " Architectural and Microarchitectural Design".
• Florent de Dinechin was a member of the conference program committee of the conferences Arith (arith2023.arithsymposium.org/), FCCM (www.fccm.org/past/2023/), and FPL (2023.fpl.org/).

10.1.3 Invited talks

• Romain Michon gave an invited talk at the Rhode Island School of Design in April 2023.
• Romain Michon gave an invited talk at the Fédération Informatique de Lyon in July 2023.
• Romain Michon gave an invited talk at the École Normale Supérieure de Lyon in October 2023.
• Romain Michon gave an invited talk at the “Fabrique des sons” symposium at the École des arts de la Sorbonne in Paris in October 2023.
• Romain Michon gave an invited talk at the CCRMA colloquium at Stanford University in October 2023 on the work carried out around the Plasma team.
• Tanguy Risset gave an invited talk at CCRMA in the DSP Seminar of Julius Smith at Stanford University in October 2023 on the recent advances of the Syfala project.
• Tanguy Risset gave an invited talk at ENS-Rennes in January presenting the research of the Emeraude Team and more specifically the Syfala project.
• Tanguy Risset gave an invited talk for the GDR SoC in Lyon in June presenting the research of the Emeraude Team.
• Florent de Dinechin gave an invited online talk to the FPBench community meeting (fpbench.org/community-meetings.html) on the upcoming book Application-Specific Arithmetic.
• Florent de Dinechin gave two invited lectures at the Joint ICTP-IAEA School on Systems-on-Chip Based on FPGA for Scientific Instrumentation and Reconfigurable Computing (indico.ictp.it/event/10225/).

10.1.4 Scientific expertise

Florent de Dinechin is a member of the scientific committee of the DeepGreen platform (deepgreen.ai/).

10.2.1 Teaching

• Tanguy Risset is professor at the Telecommunications Department of Insa Lyon.
• Florent de Dinechin is a professor at the Computer Science Department of Insa Lyon. He also teaches computer architecture at ENS-Lyon.
• Romain Michon is a part-time lecturer at Stanford University.
• Romain Michon is a part-time associate professor at the Telecommunications Department of Insa Lyon.
• Romain Michon teaches 2 courses as part of the RIM/RAN Masters Program at the université of Saint-Étienne.
• Romain Michon teaches 2 one week workshops at Aalborg University in Copenhagen every year.
• Stephane Letz teaches 1 course as part of the RIM/RAN Masters Program at the université of Saint-Étienne.

10.2.2 Supervision

• PhD starting: Thomas Rushton: Distributed spatial audio
• PhD in progress: Orégane Desrentes: Hardware arithmetic: fused operators and applications
• PhD in progress: Maxime Popoff: Compilation of Audio Program on FPGA
• PostDoc in progress: Agathe Herrou: Fixed-point extention for the Faust programming language
• PhD in progress: Lucas Chaloyard: Cross-assemblage d’un système d’exploitation frugal

10.2.3 Juries

Tanguy Risset was a member of the jury of the following theses:

• Martin Fouilleul (reviewer), Sorbonne U.

Florent de Dinechin was a member of the jury for the following defenses:

• PhD of Mak Nazecic-Andrlon (reviewer), U. Melbourne
• PhD of Van-Phu Ha (reviewer), U. Rennes 1
• PhD of Ilias Bournias (reviewer), Sorbonne U.
• HDR of Guillaume Revy (reviewer), U. Perpignan Via Domitia

Florent de Dinechin served as External Assessor for the tenure appointment of Dr Hayden Kwok Hay So (Hong Kong U.).

International peer-reviewed conferences

• 1 inproceedingsP.Pierre Cochard, M.Maxime Popoff, A.Antoine Fraboulet, T.Tanguy Risset, S.Stéphane Letz and R.Romain Michon. A Programmable Linux-Based FPGA Platform for Audio DSP.Proceedings of the 20th Sound and Music ComputingSound and Music Computing ConferenceStockholm, SwedenBresin, R., & Falkenberg, K.2023, 110-116HALback to text
• 2 inproceedingsO.Orégane Desrentes, B.Benoît Dupont de Dinechin and F.Florent de Dinechin. Exact Fused Dot Product Add Operators.2023 ARITH - 30th IEEE International Symposium on Computer ArithmeticPortland, OR, United StatesSeptember 2023HALback to text
• 3 inproceedingsO.Orégane Desrentes, B.Benoît Dupont de Dinechin and J.Julien Le Maire. Exact Dot Product Accumulate Operators for 8-bit Floating-Point Deep Learning.DSD/SEAA 2023 - 26th Euromicro Conference Series on Digital System DesignDurres, AlbaniaSeptember 2023HALback to text
• 4 inproceedingsB.Benoît Dupont de Dinechin, J.Julien Hascoet and O.Orégane Desrentes. In-Place Multicore SIMD Fast Fourier Transforms.HPEC 2023 - 27th Annual IEEE High Performance Extreme Computing Virtual ConferenceVirtual conference, United StatesSeptember 2023HALback to text
• 5 inproceedingsR.Romain Michon, J.Julien Sourice, V.Victor Lazzarini, J.Joseph Timoney and T.Tanguy Risset. Towards High Sampling Rate Sound Synthesis On FPGA.Proceedings of the 2023 Digital Audio Effects Conference (DAFx23)Copenhagen, DenmarkSeptember 2023HAL
• 6 inproceedingsT. A.Thomas Albert Rushton, R.Romain Michon and S.Stéphane Letz. A Microcontroller-Based Network Client Towards Distributed Spatial Audio.Proceedings of the 2023 Sound and Music Computing Conference (SMC-23)Stockholm, SwedenJune 2023HALback to text
• 7 inproceedingsA.Anastasia Volkova, F.Florent de Dinechin and M.Martin Kumm. Hardware-optimal digital FIR filters: one ILP to rule them all and in faithfulness bind them.Proceedings of the Asilomar conference2023 Asilomar Conference on Signals, Systems, and ComputersAsilomar, United StatesMarch 2024HALback to text

Conferences without proceedings

• 8 inproceedingsR.Romain Michon, J.Joseph Bizien, M.Maxime Popoff and T.Tanguy Risset. Making Frugal Spatial Audio Systems Using Field-Programmable Gate Arrays.Proceedings of the 2023 New Interfaces for Musical Expression ConferenceMexico City, MexicoMay 2023HALback to text

Doctoral dissertations and habilitation theses

• 9 thesisL.Luc Forget. Description and compilation of ad-hoc arithmetic operators in the context of High-Level Synthesis.INSA de LyonJune 2023HAL

Reports & preprints

• 10 reportM.Maxime Popoff, R.Romain Michon, T.Tanguy Risset, P.Pierre Cochard, S.Stephane Letz, Y.Yann Orlarey and F.Florent de Dinechin. Audio DSP to FPGA Compilation: The Syfala Toolchain Approach.RR-9507Univ Lyon, INSA Lyon, Inria, CITI, Grame, EmeraudeMay 2023HAL