Audio-to-fixed easier than float-to-fixed

In audio processing, we know that the inputs and outputs are fixed-point data, and we also have a lot of domain knowledge about audio physics. This gives serious hope that Faust audio can be compiled directly to fixed-point. This is a requirement for FPGAs, but it will also reduce the latency and power consumption on software targets if we can use their integer units. It will also enable the compilation of Faust to ultra-low-power microcontrollers without floating-point hardware.

“Domain-specific” is the key word here making us confident that a problem that is generally intractable (float-to-fixed conversion) can be addressed with little or no modification to a Faust program. The challenge here is to keep this additional work so high-level and sound-related that it is not a burden for a musician or a sound engineer. A central objective is that Faust programmers should not need to become fixed-point experts. They should actually not be anymore aware of the underlying arithmetic than they currently are with floating-point. Being high-level is a key reason for the success of Faust .

Automated error analysis for hardware computing just right

The main issue is to understand how arithmetic errors propagate, are amplified, are accumulated, etc. in a computation and in a circuit. This is called error analysis. Then a general technique 33 is to add enough bits to the right of internal fixed-point formats so that errors accumulate in these bits and the overall error accumulation does not hinder the final quality of the result. Error analysis is also managed by a worst-case combination, but here there is nothing implicit or hidden. This is therefore a comparatively well understood problem, and there is no reason to believe it cannot be fully automated in a compiler that is already able to derive the format information, building on the experience accumulated when designing complex FloPoCo operators 32, 31, 22, 84, 89.

Digital filters as arithmetic objects

Digital filters are essential components of everyday electronics like radios, mobile phones, etc., but also in audio systems of course. Their design is a core topic in digital signal processing and control theory, one that has received significant research interest for the better part of the last half century. A lot of effort has gone into constructing flexible filter design methods. For designing software-based digital filters with floating-point coefficients, there are many powerful approaches that are relatively easy to use by the filter designer (all the more as they rely on over-dimensioned floating-point operators). When designing hardware, things are not that simple for several reasons:

• algorithms developed for software-implemented filters cannot be transferred directly to hardware: what is a constraint in software (e.g., “use a 32-bit fixed-point format”) becomes a degree of freedom in hardware design (“What is the smallest fixed-point format that can be used?”);
• another degree of freedom comes from different available realization techniques to implement the arithmetic itself, for instance the construction of multipliers by constants.

A consequence is that popular tools, such as the popular fdatool (filter design and analysis tool) from Matlab's Signal Processing toolbox, offer a complex interface, requiring a tedious hand-tuning process, and expect some domain expertise. Such tools input a frequency response, and decompose the filter implementation problem in three steps: 1/ the filter design (FD) step consists in finding a filter with ideal (high precision) coefficients that adheres to the frequency response; 2/ the quantization (Q) step converts the obtained coefficients to hardware-friendly fixed-point values ; 3/ the implementation (I) step generates a valid hardware description (e.g., a VHDL or Verilog description) using the quantized coefficients.

The objective of this research axis is to offer an optimal solution to the global FD + Q + I problem. Optimal techniques exist for each of the FD, Q and I steps in isolation. The combination of the FD & Q steps have been studied since the 1960's   49, and can even be regarded as solved for certain practical instances of fixed-point Finite Impulse Response (FIR) design 50. A large body of work also exists for the I step, with recent optimal approaches  14, 51, 52. However, these approaches are only optimal for a given set of coefficients, and therefore strongly depend on the FD and Q steps.

Arithmetic-oriented scheduling and tiling for low-latency audio

Finally, we also want to formally insert arithmetic considerations in the global problem of distributing a very heavy computation between space (we have up to several thousands multipliers in an FPGA, and many more if we multiply by a constant) and time (we have thousands of FPGA cycles within one audio cycle). These are well-researched compilation issues, called the scheduling and tiling problems. There is local expertise in Lyon (in particular in the CASH team and its spin-off XtremLogic21) who have worked on these problems for FPGAs. However, scheduling and tiling techniques so far consider each operation as having a standard, constant cost (e.g., multiplication costs $c_m$ and has latency $t_m$, addition costs $c_a$ in space and $t_a$ in time). This is a very crude simplification if one attempts to optimize each operator, think for multiplications by constant for instance. The availability of many audio-related case studies in Emeraude will allow us (hopefully in collaboration with CASH) to develop arithmetic-aware scheduling and tiling techniques that will eventually prove useful well beyond the world of digital audio.

Physical modeling

One of the most active areas of research in audio DSP is physical modeling of musical instruments, mechanic vibrations and analog electronic circuits (also called “virtual analog”). Because these techniques have a direct link with the real acoustical-world, they make a lot of sense in the context of Emeraude which develops skills in embedded systems, hardware and electronics.

While waveguide 81, mass-interaction 26, and modal models 13 are relatively well understood and can be easily implemented in Faust , we would like to focus on algorithms using the Finite-Difference Time-Domain (FDTD) method 19, 18. Similarly, we would like to work on Wave Digital Filter (WDF) algorithms 38 in the context of the modeling of analog electronic circuits for audio processing and synthesis 91. This is a very hot topic 90.

Virtual acoustics, acoustic active control, and spatial audio

A large portion of the tools developed as part of Emeraude target real-time ultra-low-latency audio (i.e., FPGAs, etc.). Some of the application areas for this type of systems are virtual room acoustics (e.g., artificial reverberation, etc.), acoustic active control, and spatial audio (i.e., Ambisonics  40 and binaural  66).

Artificial reverberation can be implemented using a wide range of techniques such as feedback delay networks  45, waveguide meshes 73, finite difference schemes  74, simple combination of IIR filters 77, and Impulse Responses (IR) convolution 86. Convolution reverbs have been taken to another step recently by introducing the concept of “modal reverb” 12 where convolution is carried out in the time domain instead of the frequency domain by using a bank of resonant bandpass filters taking advantage of the principle of modal decomposition  13. While similar experiments have been prototyped on GPUs 80, our proposal it to take this to the next level by providing ultra-low latency and active control of the acoustics of the room where the sound will be rendered.

Active control of room acoustics is a challenging topic with the first commercial applications dating back to the 90s 41. The objective of this theme is to vary at will the subjective and quantifiable acoustic parameters of a room (e.g., reverberation time, early reflections, loudness, etc.) in order to adapt it to the artistic performance and the venue 71. To reach this goal, several loudspeakers, microphones and DSP modules are used to create artificial reflections in a non-generative way  48. Several commercial systems are available on the market such as CSTB's CARMEN 75 or Yamaha's AFC system 2265 to name a few. Other examples can be found in 71. In such systems, the feedback loops should be controlled, as well as the system stability, and real-time DSP.

Recent advances in active control and approaches to sound field decomposition 55 allow us to partially overcome this limitation, by proposing “spatial active noise contol” algorithms 92, 44. As one can imagine, such approaches require a very large number of control channels with very low latency. While Emeraude investigates the technological solutions to implement such algorithms in a real situation, we also hope to make contributions to the field of acoustic active control in general.

Machine learning for digital signal processing

Machine learning and deep learning in particular, are playing an increasingly important role in the field of audio DSP. Researchers are revisiting the algorithmic techniques of signal synthesis and processing in the light of machine learning, for instance for speech processing 42. Recent breakthroughs such as the use of machine learning use in the context of Differentiable Digital Signal Processing (DDSP) 37 demonstrate its power. Hybrid approaches combine classical signal processing with deep learning to obtain CPU efficient implementations like in the RNNoise project.23

One of our goals is to integrate these new developments to the Emeraude ecosystem and to further explore their potential applications in the context of embedded audio processing.

The Faust language and its compiler

A large part of Emeraude's research results is visible thanks to the Faust ecosystem development. Faust has gained an international recognition, especially since it is used for teaching at Stanford University (in the context of courses on signal processing, physical interaction design, etc.) and developing new audio plugins  64. The efforts needed to keep Faust as the most efficient language for real-time audio processing involve research in: language design, compiler design, and development of DSP libraries.

One of the reason of Faust 's success is that it is both a language and an environment for audio signal processing. The Faust compiler typically generates high-level codes (in languages such as C, C++, etc.), following every compiler's goal: providing better code than manually written code. For that, it has to stick to the most recent compiler technologies and processors evolutions  58. For instance, a back-end for WebAssembly was recently added to the Faust compiler  57. An important deployment step was the embedding of the Faust compiler in a web browser  56 which makes it easily accessible on all computers.

Faust language design research in Emeraude

The current design of Faust , inspired by lambda-calculus, combinatory logic and John Backus’ functional programming, has to be extended to face new challenges, in particular multi-dimensional and multi-rate signals and linear algebra.

Faust allows for the description of synchronous mono-rate scalar DSP computations. This is sufficient to implement most time-domain algorithms such as filters, oscillators, waveguides, etc. However, this makes the implementation of frequency-domain algorithms (e.g., FFT, convolution, etc.) very inefficient, not to say impossible. One of our goals is to extend the language to enable multi-rate as well as vector computations. While we already have a working prototype for this, some challenges have yet to be overcome.

Along the lines of the previous point, Faust currently doesn't provide any support for efficient matrix operations and more generally linear algebra. This prevents the implementation of some classes of DSP algorithms such as Finite-Difference Time-Domain (FDTD) method for physical modeling. The skills of former Socrate members on seminal Alpha language  34 and polyhedral optimization are very useful here.

Support for the main target programming languages in Faust is essential. Recently added languages (WebAssembly, Rust, and SOUL) have opened many new opportunities. The FPGA target, studied in §3.1, introduces new challenges such as the ability to use fixed-point arithmetic or the use of HLS for targeting hardware platforms (e.g., VHDL, Verilog, etc.). Other “exotic” architectures such as GPUs or MPSoCs should be studied for compute-bound algorithms.

Musicians have to deal with a large variety of operating systems, software environments and hardware architectures. Faust is designed to favor an easy deployment of Faust programs on all these targets by making a clear separation between computation itself, as described by the program code, and how this computation should be related to the external world. This relation (with audio drivers, GUIs, sensors, etc.) is described in specific architecture files39. Architecture files concern both hardware (i.e., audio interfaces/sound cards) as well as software control interfaces (e.g., GUI, OSC,24 MIDI), new luthieries (e.g., SmartFaust, Gramophone), Web platforms (Web audio Plugin), etc. One of the goal of the work of Emeraude on Faust is to ease the programming of these audio systems.

Faust ecosystem and DSP libraries

Faust users are very attached to its ecosystem, including native applications, online and “embedded” audio applications, Just In Time (JIT) compiler, etc. Recent developments include a JIT Faust compiler on the Web, a JIT compiler in the Max/MSP environment, tools to find the best compilation parameters and ease compilation for multiple CPUs. This is constantly evolving to answer to users' demand.

The Faust DSP libraries currently implement hundreds of functions/objects ranging from simple oscillators and filters to advanced filter architectures, physical models, and complete ready-to-use audio plugins. These libraries are at the heart of Faust 's success and international position. Julius Smith25 (Stanford professor) is one of the most respected figures in the field of audio DSP and one of the main contributors to the Faust libraries. One of the ambitions of the Emeraude team is to maintain and extend this tool to make it as exhaustive and as universal as possible. Along these lines, new developments made to the language presented above (e.g., multi-rate, linear algebra, etc.) should be ported to the libraries. Finally, dedicated libraries targeting specific hardware platforms (e.g., microcontrollers, FPGAs) should be made available too.

Embedded systems for audio processing

As Emeraude's name suggests it, the implementation of audio Digital Signal Processing on embedded hardware is at the heart of the project. We naturally rely on the Faust language for these implementations. The skills of Emeraude members in compilation and embedded systems are used to add new embedded target for audio processing, in particular FPGAs, as explained previously. This action is a mix of research and engineering work, it should be very useful for the dissemination of audio processing programming.

Haptics is a huge topic, especially in the field of New Interfaces for Musical Expression (NIME), which has been studied for many years 28, 87. It has always been tightly coupled to physical modeling because this sound synthesis technique provides natural connections to the physical world. A big part of the challenge is technological because haptics requires ultra low-latency and high sampling resolution in order to be accurate. This is at the heart of Emeraude 's goals.

Virtual and Augmented Reality (VR/AR) is not limited to immersive 3D graphics, sound also has an important role to play in that emerging field. Lots of work has been done around using VR environments as a creative tool for audio  54, 25, 23. While many VR-based musical instruments have been created in the past 78, little work has been done around implementing interfaces specifically targeting VR/AR audio environments, especially in the context of 3D sound. This is something that we plan to explore as part of Emeraude.

Finally, beside ergonomic and HCI aspects, the design of musical interfaces is impacted by various kinds of technical limitations that we plan to address as part of Emeraude. First, just like for real-time audio processing, latency plays a crucial role in this context. Similarly, the “time resolution” (e.g., the sampling rate of the interfaces) can have a huge impact, especially when targeting specific kinds of instruments such as drums. Finally, the “spatial resolution” (e.g., the number of sensor points per squared centimeters on a tabletop interface) also impacts its quality. In this context, we would like to develop an embedded, high-resolution, high-sampling-rate, multi-touch multi-dimensional (X and Y + pressure) interface/instrument leveraging the development carried out in the previous axes. This work would be followed by a user study to measure the impact of this type of advanced system on perception.

5.2.1 2022 Sound and Music Computing

Romain Michon was the general chair and main organizer of the 2022 Sound and Music Computing conference27 (SMC-22) which took place in Saint-Étienne (and on the Web) on June 5-12, 2022. SMC-22 (see Fig. 5) combined multiple events: a summer school, a festival with 8 concerts, the Sound and Music Computing conference, the International Faust Conference (IFC-22), and the Journée de l'Informatique Musicale (JIM-22).

5.2.2 2022 Programmable Audio Workshop

Emeraude organized the 2022 Programmable Audio Workshop28 (PAW-22). This event took place at INSA Lyon on December 3d, 2022. PAW is a yearly one day event gathering members of the programmable audio community around scientific talks and hands-on workshops. The theme of PAW this year was “Networked and Embedded Audio Systems,” with a strong focus on spatial audio and Field-Programmable Gate Arrays (FPGAs).

5.2.3 ARITH 2022

Florent de Dinechin was the general chair of ARITH 2022291. Since 1969, ARITH has served as the premier conference for presenting the latest research in computer arithmetic. Due to the uncertainty of the world health situation and travel restrictions, the 29th edition of the symposium, ARITH 2022, has been a virtual conference with live presentation of research results, keynote talks, and panels.

6.1.1 FloPoCo

• Name:
  Floating-Point Cores, but not only
• Keyword:
  Synthesizable VHDL generator
• Functional Description:
  The purpose of the open-source FloPoCo project is to explore the many ways in which the flexibility of the FPGA target can be exploited in the arithmetic realm.
• URL:
  http://­flopoco.­org
• Contact:
  Florent de Dinechin
• Participants:
  Florent de Dinechin, Luc Forget
• Partners:
  ENS Lyon, Insa de Lyon, Inria, Fulda University of Applied Science

6.1.2 hint

• Name:
  High-level synthesis Integer Library
• Keyword:
  High-level synthesis
• Functional Description:
  Hint is an header-only arbitrary size integer API with strong semantics for C++. Multiple backends are provided using various HLS libraries, allowing a user to write one operator and synthetize it using the main vendor tools.
• URL:
  https://­github.­com/­yuguen/­hint
• Publication:
  hal-02131798v2
• Contact:
  Luc Forget
• Participants:
  Yohann Uguen, Florent de Dinechin, Luc Forget

6.1.3 marto

• Name:
  Modern Arithmetic Tools
• Keywords:
  High-level synthesis, Arithmetic, FPGA
• Functional Description:

  Marto provides C++ headers to implement custom sized arithmetic operators such as:

  Custom sized posits and their environment (including the quire) Custom sized IEEE-754 numbers Custom sized Kulisch accumulators (and sums of products)

• URL:
  https://­gitlab.­inria.­fr/­lforget/­marto
• Publication:
  hal-02130912v4
• Contact:
  Yohann Uguen
• Participants:
  Yohann Uguen, Florent de Dinechin, Luc Forget

6.1.4 Syfala

• Name:
  Low-Latency Synthesizer on FPGA
• Keywords:
  FPGA, Compilers, High-level synthesis, Audio signal processing
• Functional Description:

  The goal of Syfala is to design an FPGA-based platform for multichannel ultra-low-latency audio Digital Signal Processing programmable at a high-level with Faust and usable for various applications ranging from sound synthesis and processing to active sound control and artificial sound field/room acoustics.

  A series of tools are currently being developed around SyFaLa. While none of them has been officially released yet, you can follow their development/evolution on the project Git repository: https://gitlab.inria.fr/risset/syfala

• URL:
  https://­faust.­grame.­fr/­syfala/
• Contact:
  Tanguy Risset

6.1.5 FAUST

• Name:
  Functional Audio Stream)
• Keywords:
  Audio, Functional programming
• Functional Description:

  The core component of Faust is its compiler. It allows to "translate" any Faust digital signal processing (DSP) specification to a wide range of non-domain specific languages such as C++, C, LLVM bit code, WebAssembly, Rust, etc. In this regard, Faust can be seen as an alternative to C++ but is much simpler and intuitive to learn.

  Thanks to a wrapping system called "architectures," codes generated by Faust can be easily compiled into a wide variety of objects ranging from audio plug-ins to standalone applications or smartphone and web apps, etc.

• URL:
  https://­faust.­grame.­fr/
• Contact:
  Yann Orlarey
• Partners:
  GRAME, Insa de Lyon, Inria

7.1.1 Syfala compiler now available

An operational version of this compiler was presented at SMC2022  8. The major innovation presented in this paper is the very low analog to analog obtained latency: 11$\mu s$ while previous state of the art only announced a 100$\mu s$ latency. The compiler was also demonstrated with many application: noise cancellation, virtual analog, minimoog emulation etc. In 2022, the Syfala compiler includes many important improvements:

• Hardware/software automatic partitioning: the ARM processor being responsible of initialization and control-rate modifications, the FPGA;
• Access to DDR memory from FPGA and ARM processor, necessary to hold long delay lines induced by echos;
• Target (Xilinx) FPGA and audio codec can be configured (Zybo board and Genesys ZU board currently available).

7.1.2 Syfala github release

• A public github release of Syfala has been published on github.com/inria-emeraude/syfala. The distribution has already been successefully used outside of Emeraude (in Maynooth, Ireland, see §7.3).
• A complete re-writing of the compilation scheme using TCL-scripts allowing for a much more parameterizable toolchain than what was achievable with Makefiles.

7.1.3 I2S improvements

The FPGA design used in Syfala uses an IP (currently generated by Xilinx HLS tools) but also a manually written VHDL IP implementing the I2S serial protocol used to interface audio Codecs. It rapidely occured to us that this component was very important, especially if a large number of I/O audio channel are used which is the case in many applications targeted by Emeraude (see the WFS prototype in section 7.2 for instance).

In 2022 two improvements were implemented in the I2S IP:

• A script is now able to generate as many I2S transceiver as needed by the Faust IP. Indeed, when compiling a Faust program with many I/O channel, the complete Xilinx block design had to be manually changed and the compilation process was slowed down by this manual intervention. These changes in the block design are now scripted and hence automated in the compilation process.
• A TDM version of the I2S transceiver was developped allowing us to target up to 8 audio codecs with one I2S transceiver. Thanks to this, we can now target very large number of I/O channels (e.g., up to 128 I/O channel on the Zybo board).

7.1.4 Faust to VHDL compilation

Syfala currently uses HLS tools proposed by Xilinx for generating VHDL. But there are other ways for getting VHDL from Faust : generating directly VHDL code from the Faust compiler internal representation.

Generating directly VHDL code has some advantages (e.g., using fixed point flopoco operators, simplify the control infrastucture, as Faust signal graph internal representation is similar to synchronous data flow, etc.), and some drawbacks (e.g., difficulties to access the memory using the complex AXI bus protocol, difficulties in communicating with the ARM processor, complex scheduling of the graph, possibly reusing operators, etc.).

A first very simple Faust2VHDL translator has been written within the Faust compiler. For now, it only computes the whole Faust graph in one FPGA cycle and do not use any controllers of memory accesses. It should soon be improved by providing a way to evaluate the Faust IP complexity overhead induced by the use of HLS.

7.2.1 Centralized system for spatial audio

As a first step towards implementing a centralized system for spatial audio, we worked on a low-cost Wave Field Synthesis (WFS) 16 system based on MAX98357A digital amplifiers which are compatible with Time Division Multiplexing (TDM) up to 8 channels and only cost 5 euros a piece! We made custom speakers with laser cut wood which directly connect to the amplifiers. We also implemented a specific TDM-compatible version of the Syfala toolchain targeting this system and accessible through the –tdm option in the Syfala tool.

The current version of the WFS system has 32 speakers and only uses 6 GPIOS (2 + 1 + 1 + 1 +1) of a Zybo Z7-20 board (see Fig. 8). We successfully ran a standard WFS algorithm implemented in the Faust programming language on this system where 2 sources can be moved in space. Individual sources are sent to the speaker array though analog audio using the built-in audio inputs of the Zybo Z7-20. Their position can be controlled using a web interface accessible through an HTTPD server.

Thanks to the use of the FPGA, the cost of each new channel in the system is very low. Implementing a WFS system with a more traditional architecture would be way more expensive that what we achieved which is very promising!

We're planning on presenting this work at the NIME-23 conference this year.

7.2.2 Distributed system for spatial audio

We developed a distributed systems for spatial audio DSP based on a network of microcontrollers (see Fig. 9). We chose this type of platform because they are cost-effective, very lightweight, and OS-free. We used PJRC's Teensys 4.130 as they host a powerful Cortex M7 clocked at 600 MHz as well as built-in ethernet support. PJRC also provides an “audio shield” which is just a breakout board for a stereo audio codec (SGTL-5000) compatible with the Teensy 4.1.

A preliminary task was to send audio streams over the Ethernet from a laptop to the Teensy. For that, we decided to use the JackTrip protocol which is open source and used a lot in the audio/music tech community 24. Implementing a JackTip client on the Teensy was fairly straightforward. We do believe that this also has applications for network audio performances.

Audio DSP is carried out directly on the Teensys which are programmed with the Faust programming language thanks to the faust2teensy62 tool developed by the Emeraude team. A laptop is used to transmit audio streams to the Teensys which are controlled using the Open Sound Control (OSC) standard. OSC messages are multicast/broadcast to save bandwidth. The same audio streams are sent to all the Teensys in the network (all audio processing is carried out on the Teensys, not on the laptop computer).

We're planning on presenting this work at the SMC-23 conference this year.

7.5.1 Faust compiler

The Faust compiler now follows a “four official release per year” model, adding new features, fixing bugs, embedding the updated Faust libraries with new contributions, improving the architecture files ecosystem, and continuously improving the documentation (by adding new tutorials, tools description, etc.).

7.5.2 New -os2 compiler option

A new -os2 compilation option has been specifically added to the Faust compiler for the C and C++ backends to be used by the Syfala toolchain  8.

It implements one-sample computation and allows for a better separation between control rate and sample rate computations as well as for a better control of the DSP memory layout. The DSP memory is divided between a section kept in the DSP class/structure, and a separated section typically allocated on the DDR, like for long delay lines.

Two specialized architecture files have been developed to deploy the generated code on the ARM on one side and on the FPGA on the other side.

7.5.3 New backends

Two new backends have been added in the compiler:

• For the Cmajor language31 a C-family language designed specifically for writing DSP signal processing code recently announced by the SoundSlacks company.32 The backend directly produces Cmajor source code. The faust2cmajor tool compiles a Faust DSP program in a folder containing the Cmajor source code and Cmajor patch. The result can be a monophonic DSP or a MIDI controllable polyphonic one (when the DSP describes an instrument, following the freq, gain, gate parameter naming convention). The resulting Cmajor code can be played using the cmaj runtime.
• For JAX,33 a python framework that can automatically differentiate programs, thus allowing Faust DSP code to support differentiability and learnable parameters, which is a huge research opportunity in the Machine Learning domain. This contribution has been done by David Braun, a former CCRMA Stanford student, with the help of GRAME.

7.5.4 DSP code debugging tool

Debugging and tracing tools have been greatly improved. The interp-tracer tool runs and instruments the compiled program using the Interpreter backend, running bytecode in a virtual execution environment. Various statistics on the code are collected and displayed while running and/or when closing the application, typically FP_SUBNORMAL, FP_INFINITE and FP_NAN values, or INTEGER_OVERFLOW, CAST_INT_OVERFLOW and DIV_BY_ZERO operations, or LOAD/STORE errors.

Debugging out-of-domain computations and checking rdtable and rwtable primitives have been added in the tool.

7.5.5 New Box and Signal API

The Faust compiler is composed of several steps: starting from the DSP source code, the Semantic Phase produces signals as conceptually infinite streams of samples or control values. Those signals are then compiled in imperative code (e.g., C/C++, LLVM IR, WebAssembly, etc.) in the Code Generation Phase.

The new exposed box and signal API opens intermediate access inside the Faust compilation chain. They allow new audio DSP languages (textual or graphical) to be built on top of the box or signal API, and take advantage of parts of the Faust compiler infrastructure.

The DawDreamer34 python based project developed by David Braun, offers an easy access to Faust infrastructure; the libfaust embeddable compiler, with access to the new box or signal API as well as the regular DSP compilation and execution APIs.

7.5.6 Interval computation

Interval computation is a technique used by the Faust compiler to estimate the minimum and maximum values that a signal can take during the execution of a program. This estimation allows to check the causality of the calculation in the case of the delay operation and to determine the memory needed for the delay lines. It also helps to avoid problems such as division by zero, table index overflows and generally ensures that the program runs correctly. Interval computation is therefore a key element in ensuring the stability and reliability of the program generated by the Faust compiler.

In order to further improve the performances of this technique, we have undertaken the development of a new implementation of the interval library. This new implementation includes, among other things, interval computation for bitwise operations and improved fixed point searches for signals from IIR filters. In addition, the intervals are extended to include an estimate of the precision required for a signal when calculations are performed in fixed point. This is particularly important for the FAST project and FPGA programming since fixed point operations are generally more resource efficient than the same floating point operations.

7.5.7 Ondemand computations

Until now, the computational model of Faust has been limited to a single rate, the audio rate. All computations are performed every audio sample, with the exception of control signals which can be computed at a lower rate. This model has the advantage of simplicity and guarantees bounded CPU and memory footprints, a property well suited for real-time constraints and embedded systems. However, it is not well suited for spectral domain applications, where you don't want to compute an FFT every sample, or for energy efficient applications where you only want to compute when needed.

To address these limitations, we introduce the concept of on-demand computations. In this model, computations are not performed on a sample-by-sample basis, but only on demand. Conceptually, these requests are represented by an additional input signal, a kind of clock. When the value of the clock is 0, no computation is performed, when it is 1, a computation is triggered. The challenge is to introduce on-demand computations while maintaining the simple, well-defined, and preservable semantics of the Faust signal processor. The key observation here is that an on-demand computation is simply a regular computation on downsampled signals, whose results are then upsampled. Downsampling and upsampling are done according to the clock signal.

The extension to the language is minimal, with the introduction of a single new construction: ondemand(C)$\to$ C' which transforms a circuit C' into an on-demand version C with an additional clock input. A complete specification of on-demand computations has been developed and presented at IFC 22, along with a prototype implementation.

ANR - Imprenum

The objective of this project (INSA-Lyon, École Normale Supérieure de Lyon, CEA LETI) is to promote accuracy as a first class concern in all the levels of a computing system:

• at the hardware level, with better support for lower-than-standard and higher-than-standard precisions;
• at the level of run-time support software, in particular answering the memory management challenges entailed by adaptive precision;
• at the lower level of mathematical libraries (kernel level), for instance BLAS for linear algebra, enhancing well established libraries with precision and accuracy control;
• at the higher level of mathematical libraries (solver level, including algebraic linear solvers such as LAPACK, ad hoc steppers for Ordinary Differential Equation, eigenvalues kernels, triangularization problems for computational geometry, etc.) Here, accuracy and precision control of the lower levels should enable higher-level properties such as convergence and stability;
• at the compiler level, enhancing optimising compilers with novel optimisations related to precision and accuracy;
• at the language level, embedding accuracy specification and control in existing languages, and possibly defining domain-specific languages with accuracy-aware semantics for some classes of applications.

ANR FAST

Embedded systems for audio and multimedia are increasingly used in the arts and culture (e.g., interactive systems, musical instruments, virtual and augmented reality, artistic creation tools, etc.). They are typically based on a CPU (Central Processing Unit) which limits their computational power and induces some latency. FPGAs (Field Programmable Gate Arrays) can be seen as a solution to these problems. However, these types of chips are extremely complex to program, making them largely inaccessible to musicians, digital artists and makers communities.

The goal of the FAST ANR project is to enable high-level programming of FPGA-based platforms for multichannel ultra-low-latency audio processing using the Faust programming language (a standard in the field of computer music). We plan to use this system for various applications ranging from sound synthesis and processing to active sound control and artificial sound field/room acoustics.

FAST officially started in March 2021. It gathers the strength of GRAME-CNCM, CITI Lab (INRIA/INSA Lyon), and LMFA (École Centrale Lyon).

ADT FFF

The objective of this ADT is to continue the engineering support for the development of the Syfala tools for the compilation from high level languages (C, Faust ) to FPGA for audio signal processing applications.

This engineer constitutes a very important support for tool development in the team (Syfala, of course but also for FloPoCo or Faust).

9.2.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

9.2.2 Other International Collaborations

10.1.1 Scientific events: organisation

• Romain Michon was the general chair of Sound and Music Computing 2022 (smc22.grame.fr) which combined three conferences (SMC-22, IFC-22, and JIM-22) as well as a festival around music and new technologies and a summer school for grad students. This event was organized in partnership with the Université Jean Monnet of Saint-Étienne and GRAME-CNCM and it took place in Saint-Étienne on June 5-12, 2022.
• Romain Michon and Stephane Letz organized the 2022 Programmable Audio Workshop (PAW-22: paw.grame.fr) which is a one day workshop on emerging programmable audio technologies. It took place at CITI Lab (INSA Lyon) on December 3, 2022.

10.1.2 Scientific events: selection

10.1.3 Invited talks

• Romain Michon gave an invited talk and workshop at the University of Nebraska in Omaha in October 2022 on the work carried out around the FAST project.
• Romain Michon gave an invited talk at the CCRMA open house at Stanford University in October 2022 on the work carried out around the Plasma team.
• Romain Michon gave an invited talk and workshop at the University of Michigan in October 2022 on the work carried out around the FAST project.
• Florent de Dinechin gave an invited lecture at the Joint ICTP-IAEA School on FPGA-based SoC and its Applications to Nuclear and Scientific Instrumentation (indico.ictp.it/event/9933/). He gave an invited talk in Saclay at the national event “Scientific computing accelerated on FPGAs.”

10.2.1 Teaching

• Tanguy Risset is professor at the Telecommunications Department of Insa Lyon.
• Florent de Dinechin is a professor at the Computer Science Department of Insa Lyon. He also teaches computer architecture at ENS-Lyon.
• Romain Michon is a part-time associate professor at the Telecommunications Department of Insa Lyon.
• Romain Michon is a part-time lecturer at Stanford University
• Romain Michon teaches 2 courses as part of the RIM/RAN Masters Program at the université of Saint-Étienne.
• Romain Michon teaches 2 one week workshops at Aalborg University in Copenhagen every year.

10.2.2 Supervision

• PhD in progress : Luc Forget : Algèbre linéaire calculant au plus juste, ANR Imprenum, since 10/2018.
• PhD in progress : Maxime Christ : Learning in Very Low Precision since 10/2018
• PhD starting : Orégane Desrentes : Hardware arithmetic: fused operators and applications
• PhD starting : Maxime Popoff : Compilation of Audio Program on FPGA since 10/2020

10.2.3 Juries

Florent de Dinechin was a member of the jury of the following theses:

• Rémi Parrot (reviewer, U. Nantes)
• Daouda Diakite (reviewer, U. Paris-Saclay

Tanguy Risset was a member of the jury of the following theses:

• David Pala (reviewer, U. Rennes)
• Erwan Lenormand (reviewer, U. Paris-Saclay )
• Martin Fouilleul (reviewer, Sorbonne Université)
• Corentin Lavaud (reviewer, U. Rennes)
• Louis Bonicel (reviewer, U. Grenoble Alpes )

International journals

• 1 articleP.Paolo Montuschi, J.-M.Jean-Michel Muller and F.Florent de Dinechin. Computer Arithmetic: Continuing a Long and Steady Emergence.Computer5510October 2022, 4-6
  HALDOIback to text

International peer-reviewed conferences

• 2 inproceedingsA.Andreas Böttcher, M.Martin Kumm and F.Florent de Dinechin. Resource Optimal Squarers for FPGAs.International Conference on Field-Programmable Logic and Applications (FPL)Belfast, United KingdomIEEEAugust 2022
  HALDOIback to text
• 3 inproceedingsM.Maxime Christ, F.Florent de Dinechin and F.Frédéric Pétrot. Low-precision logarithmic arithmetic for neural network accelerators.ASAP 2022 - 33rd IEEE International Conference on Application-specific Systems, Architectures and ProcessorsGothenburg, SwedenJuly 2022
  HALback to text
• 4 inproceedingsO.Orégane Desrentes and F.Florent de Dinechin. Using integer linear programming for correctly rounded multipartite architectures.FPT 2022 - International Conference on Field Programmable TechnologyHong Kong, ChinaDecember 2022
  HALback to text
• 5 inproceedingsR.Raphaël Dingé and S.Stéphane Letz. Automatically generating eurorack hardware running faust programs using eurorack-blocks.IFC 22 - International Faust ConferenceSaint-Etienne, FranceJune 2022
  HAL
• 6 inproceedingsL.Luc Forget, G.Gauthier Harnisch, R.Ronan Keryell and F.Florent de Dinechin. A single-source C++20 HLS flow for function evaluation on FPGA and beyond.HEART 2022 - 12th International Symposium on Highly Efficient Accelerators and Reconfigurable TechnologiesTsukuba, JapanJune 2022
  HALback to text
• 7 inproceedings S.Stéphane Letz, R.Romain Michon and Y.Yann Orlarey. What's new in the faust ecosystem and community? IFC 22 - International Faust Conference Saint-Etienne, France June 2022
  HAL
• 8 inproceedingsM.Maxime Popoff, R.Romain Michon, T.Tanguy Risset, Y.Yann Orlarey and S.Stéphane Letz. Towards an FPGA-Based Compilation Flow for Ultra-Low Latency Audio Signal Processing.SMC-22 - Sound and Music ComputingSaint-Étienne, FranceJune 2022
  HALback to textback to textback to textback to text
• 9 inproceedingsS.Shihong Ren, S.Stéphane Letz, Y.Yann Orlarey, D.Dominique Fober, R.Romain Michon, M.Michel Buffa and L.Laurent Pottier. Modernized Toolchains to Create JSPatcher Objects and WebAudioModules from Faust Code.WAC 2022 - 7th International Web Audio ConferenceCannes, FranceJuly 2022
  HALDOI

Edition (books, proceedings, special issue of a journal)

• 10 proceedingsR.Romain MichonL.Laurent PottierY.Yann OrlareyProceedings of the 19th Sound and Music Computing Conference.SMC NetworkJuly 2022
  HAL

Reports & preprints

• 11 reportP.-E.Pierre-Evariste Dagand, G.Gautier Berthou, D.Delphine Demange and T.Tanguy Risset. A Formal Model of Interrupt-based Checkpointing with Peripherals.IRIF; IRISA; INSA RENNESFebruary 2022, 1-36
  HAL