2025Activity‌ reportProject-TeamROBOTLEARN

3.1​‌ Deep probabilistic models

A​​ large number of perception​​​‌ and interaction processes require​ temporal modeling. Consider for​‌ example the task of​​ extracting a clean speech​​​‌ signal from visual and​ audio data. Both modalities​‌ live in high-dimensional observation​​ spaces and one challenge​​​‌ is to extract low-dimensional​ embeddings that encode information​‌ in a compact way​​ and to update it​​​‌ over time. These high-dimensional​ to low-dimensional mappings are​‌ nonlinear in the general​​ case. Moreover, audio and​​​‌ visual data are corrupted​ by various perturbations, e.g.​‌ by the presence of​​ background noise which is​​​‌ mixed up with the​ speech signal uttered by​‌ a person of interest,​​ or by head movements​​​‌ that overlap with lip​ movements. Finally, for robotics​‌ applications, the available data​​ is scarce, and datasets​​​‌ captured in other settings​ can only serve as​‌ proxies, thus requiring either​​ adaptation 62 or the​​ use of unsupervised models​​​‌ 48. Therefore, the‌ problem is manyfold: to‌​‌ extract low-dimensional compact representations​​ from high-dimensional inputs, to​​​‌ disregard useless data in‌ order to retain information‌​‌ that is relevant for​​ the task at hand,​​​‌ to update and maintain‌ reliable information over time,‌​‌ and to do so​​ in without (or with​​​‌ very few) annotated data‌ from the robot.

This‌​‌ class of problems can​​ be addressed in the​​​‌ framework of state-space models‌ (SSMs). In their most‌​‌ general form, SSMs are​​ stochastic nonlinear systems with​​​‌ latent variables. Such a‌ system is composed of‌​‌ a state equation, that​​ describes the dynamics of​​​‌ the latent (or state)‌ variables, and $M$ observation‌​‌ equations (an observation equation​​ for each sensorial modality​​​‌ $m$) that predict‌ observations from the state‌​‌ of the system, namely:​​

where the latent vector​​ $𝐱\in {ℝ}^{\mathrm{L‌}}$ evolves according to a‌ nonlinear stationary Markov dynamic‌​‌ model driven by the​​ observed control variable $\mathrm{𝐮‌}$ and corrupted by the‌ noise $𝐯$. Similarly,‌​‌ the observed vectors ${\mathrm{𝐲}}^m\in {ℝ}^{{\mathrm{D‌}}_m}$ are modeled with‌ nonlinear stationary functions of‌​‌ the current state and​​ current input, affected by​​​‌ noise ${𝐰}^m$.‌ Models of this kind‌​‌ have been examined for​​ decades and their complexity​​​‌ increases from linear-Gaussian models‌ to nonlinear and non-Gaussian‌​‌ ones. Interestingly, they can​​ also be viewed in​​​‌ the framework of probabilistic‌ graphical models to represent‌​‌ the conditional dependencies between​​ the variables. The objective​​​‌ of an SSM is‌ to infer the sequence‌​‌ of latent variables by​​ computing the posterior distribution​​​‌ of the latent variable,‌ conditioned by the sequence‌​‌ of observations, $p({𝐱}_t|{\mathrm{𝐲‌}}_{1:t})‌$.

When the two‌​‌ functions are linear, the​​ model boils down to​​​‌ a linear dynamical system,‌ that can be learned‌​‌ with an exact Expectation-Maximization​​ (EM) algorithm. Beyond this​​​‌ simple case, non-linearity can‌ be achieved via mixtures‌​‌ of $K$ linear models​​ or more general non-linear​​​‌ (e.g. deep neural) functions.‌ Either case, learning and‌​‌ inference cannot be exact​​ and must be approximated,​​​‌ either by using variational‌ EM algorithms 46,‌​‌ 56, 49,​​ 3, amortized variational​​​‌ inference 55, 47‌ or a combination of‌​‌ both techniques 57,​​ 18.

We name​​​‌ the larger family of‌ all these methods as‌​‌ Deep Probabilistic Models (DPMs),​​ which form a backbone​​​‌ among the methodological foundations‌ of RobotLearn. Learning‌​‌ DPMs is challenging from​​ the theoretical, methodological and​​​‌ computational points of view.‌ Indeed, the problem of‌​‌ learning, for instance, deep​​ generative Bayesian filters in​​​‌ the framework of nonlinear‌ and non-Gaussian SSMs remains‌​‌ intractable and approximate solutions,​​ that are both optimal​​​‌ from a theoretical point‌ of view and efficient‌​‌ from a computational point​​​‌ of view, remain to​ be proposed. We plan​‌ to investigate both discriminative​​ and generative deep recurrent​​​‌ Bayesian networks and to​ apply them to audio,​‌ visual and audio-visual processing​​ tasks.

Exemplar application: deep​​​‌ probabilistic sequential modeling

We​ have investigated a latent-variable​‌ generative model called mixture​​ of dynamical variational autoencoders​​​‌ (MixDVAE) to model the​ dynamics of a system​‌ composed of multiple moving​​ sources. A DVAE model​​​‌ is pre-trained on a​ single-source dataset to capture​‌ the source dynamics. Then,​​ multiple instances of the​​​‌ pre-trained DVAE model are​ integrated into a multi-source​‌ mixture model with a​​ discrete observation-to-source assignment latent​​​‌ variable. The posterior distributions​ of both the discrete​‌ observation-to-source assignment variable and​​ the continuous DVAE variables​​​‌ representing the sources content/position​ are estimated using the​‌ variational expectation-maximization algorithm, leading​​ to multi-source trajectories estimation.​​​‌ We illustrated the versatility​ of the proposed MixDVAE​‌ model on two tasks:​​ a computer vision task,​​​‌ namely multi-object tracking, and​ an audio processing task,​‌ namely single-channel audio source​​ separation. Consequently, this mixture​​​‌ models allows to mix​ different non-linear source models​‌ within the maximum likelihood​​ umbrella and combine the​​​‌ model with other probabilistic​ models as well.

MixDVAE​‌ overall diagram.

3.2 Human behavior understanding​

Interactions between a robot​‌ and a group of​​ people require human behavior​​​‌ understanding (HBU) methods. Consider​ for example the tasks​‌ of detecting eye-gaze and​​ head-gaze and of tracking​​​‌ the gaze directions associated​ with a group of​‌ participants. This means that,​​ in addition to gaze​​​‌ detection and gaze tracking,​ it is important to​‌ detect persons and to​​ track them as well.​​​‌ Additionally, it is important​ to extract segments of​‌ speech, to associate these​​ segments with persons and​​​‌ hence to be able​ to determine over time​‌ who looks to whom​​ and who is the​​​‌ speaker and who are​ the listeners. The temporal​‌ and spatial fusion of​​ visual and audio cues​​​‌ stands at the basis​ of understanding social roles​‌ and of building a​​ multimodal conversational model.

Performing​​​‌ HBU tasks in complex,​ cluttered and noisy environments​‌ is challenging for several​​ reasons: participants come in​​​‌ an out of the​ camera field of view,​‌ their photometric features, e.g.​​ facial texture, clothing, orientation​​​‌ with respect to the​ camera, etc., vary drastically,​‌ even over short periods​​ of time, people look​​​‌ at an object of​ interest (a person entering​‌ the room, a speaking​​ person, a TV/computer screen,​​​‌ a wall painting, etc.)​ by turning their heads​‌ away from the camera,​​ hence facial image analysis​​​‌ is difficult, small head​ movements are often associated​‌ with speech which perturbs​​ both lip reading and​​​‌ head-gaze tracking, etc. Clearly,​ understanding multi-person human-robot interaction​‌ is complex because the​​ person-to-person and person-to-object, in​​​‌ addition to person-to-robot, interactions​ must explicitly be taken​‌ into account.

We propose​​ to perform audio-visual HBU​​​‌ by taking explicitly into​ account the complementary nature​‌ of these two modalities.​​ Differently from one current​​​‌ trend in AV learning​ 45, 52,​‌ 54, we opt​​ for unsupervised probabilitic methods​​ that can (i) assign​​​‌ observations to persons without‌ supervision, (ii) be combined‌​‌ with various probabilistic noise​​ models and (iii) and​​​‌ fuse various cues depending‌ on their availability in‌​‌ time (i.e. handle missing​​ data). Indeed, in face-to-face​​​‌ communication, the robot must‌ choose with who it‌​‌ should engage dialog, e.g.​​ based on proximity, eye​​​‌ gaze, head movements, lip‌ movements, facial expressions, etc.,‌​‌ in addition to speech.​​ Unlike in the single-user​​​‌ human-robot interaction case, it‌ is crucial to associate‌​‌ temporal segments of speech​​ to participants, referred to​​​‌ as speech diarization. Under‌ such scenarios, speech signals‌​‌ are perturbed by noise,​​ reverberation and competing audio​​​‌ sources, hence speech localization‌ and speech enhancement methods‌​‌ must be used in​​ conjunction with speech recognition.​​​‌

It is also necessary‌ to perform some kind‌​‌ of adaptation to the​​ distribution of the particular​​​‌ data at hand, e.g.‌ collected with robot sensors.‌​‌ If these data are​​ available in advance, off-line​​​‌ adaptation can be done,‌ otherwise the adaptation needs‌​‌ to be performed on-line​​ or at run time.​​​‌ Such strategies will be‌ useful given the particular‌​‌ experimental conditions of practical​​ human-robot interaction scenarios. Either​​​‌ way we will need‌ some sort of on-line‌​‌ learning to perform final​​ adaptation. On-line learning based​​​‌ on deep neural networks‌ is far from being‌​‌ well understood. We plan​​ to thoroughly study the​​​‌ incorporation of on-line learning‌ into both Bayesian and‌​‌ discriminative deep networks. In​​ the practical case of​​​‌ interaction, real-time processing is‌ crucial. Therefore, a compromise‌​‌ must be found between​​ the size of the​​​‌ network, its discriminative power‌ and the computational cost‌​‌ of the learning and​​ prediction algorithms. Clearly, there​​​‌ is no single solution‌ given the large variety‌​‌ of problems and scenarios​​ that are encountered in​​​‌ practice.

Exemplar application: expression-preserving‌ face frontalization

Face frontalization‌​‌ consists of synthesizing a​​ frontally-viewed face from an​​​‌ arbitrarily-viewed one. We proposed‌ a frontalization methodology that‌​‌ preserves non-rigid facial deformations​​ in order to boost​​​‌ the performance of visually‌ assisted speech communication. The‌​‌ method alternates between the​​ estimation of (i) the​​​‌ rigid transformation (scale, rotation,‌ and translation) and (ii)‌​‌ the non-rigid deformation between​​ an arbitrarily-viewed face and​​​‌ a face model. The‌ method has two important‌​‌ merits: it can deal​​ with non-Gaussian errors in​​​‌ the data and it‌ incorporates a dynamical face‌​‌ deformation model. For that​​ purpose, we used the​​​‌ generalized Student t-distribution in‌ combination with a linear‌​‌ dynamic system in order​​ to account for both​​​‌ rigid head motions and‌ time-varying facial deformations caused‌​‌ by speech production. We​​ proposed to use the​​​‌ zero-mean normalized cross-correlation (ZNCC)‌ score to evaluate the‌​‌ ability of the method​​ to preserve facial expressions.​​​‌ We showed that the‌ method, when incorporated into‌​‌ deep learning pipelines, namely​​ lip reading and speech​​​‌ enhancement, improves word recognition‌ and speech intelligibility scores‌​‌ by a considerable margin.​​

Some results of the​​​‌ proposed expression-preserving face frontalization‌ method.

3.3 Learning and‌ control for social robots‌​‌

Traditionally, research on human-robot​​​‌ interaction focused on single-person​ scenarios also called dyadic​‌ interactions. However, over the​​ past decade several studies​​​‌ were devoted to various​ aspects of multi-party interactions,​‌ meaning situations in which​​ a robot interacts with​​​‌ a group of two​ or more people 59​‌. This line of​​ research is much more​​​‌ challenging because of two​ main reasons. First, the​‌ behavioral cues of each​​ individual and of the​​​‌ group need to be​ faithfully extracted (and assigned​‌ to each individual). Second,​​ the behavioral dynamics of​​​‌ groups of people can​ be pushed by the​‌ presence of the robot​​ towards competition 51 or​​​‌ even bullying 50.​ This is why some​‌ studies restrict the experimental​​ conditions to very controlled​​​‌ collaborative scenarios, often lead​ by the robot, such​‌ as quiz-like game playing​​ 61 or very specific​​​‌ robot roles 53.​ Intuitively, constraining the scenario​‌ also reduces the gesture​​ variabilty and the overall​​​‌ interaction dynamics, leading to​ methods and algorithms with​‌ questionable generalisation to free​​ and natural social multi-party​​​‌ interactions.

Whenever a robot​ participates in such multi-party​‌ interactions, it must perform​​ social actions. Such​​​‌ robot social actions are​ typically associated with the​‌ need to perceive a​​ person or a group​​​‌ of persons in an​ optimal way as well​‌ as to take appropriate​​ decisions such as to​​​‌ safely move towards a​ selected group, to pop​‌ into a conversation or​​ to answer a question.​​​‌ Therefore, one can distinguish​ between two types of​‌ robot social actions: (i)​​ physical actions which correspond​​​‌ to synthesizing appropriate motions​ using the robot actuators​‌ (motors), possibly within a​​ sensorimotor loop, so as​​​‌ to enhance perception and​ maintain a natural interaction​‌ and (ii) spoken actions​​ which correspond to synthesizing​​​‌ appropriate speech utterances by​ a spoken dialog system.​‌ In RobotLearn we will​​ focus on the former,​​​‌ and integrate the latter​ via collaborations with research​‌ groups having with established​​ expertise in speech technologies.​​​‌

In this regard we​ face three problems. First,​‌ given the complexity of​​ the environment and the​​​‌ inherent limitations of the​ robot's perception capabilities, e.g.​‌ limited camera field of​​ view, cluttered spaces, complex​​​‌ acoustic conditions, etc., the​ robot will only have​‌ access to a partial​​ representation of the environment,​​​‌ and up to a​ certain degree of accuracy.​‌ Second, for learning purposes,​​ there is no easy​​​‌ way to annotate which​ are the best actions​‌ the robot must choose​​ given a situation: supervised​​​‌ methods are therefore not​ an option. Third, since​‌ the robot cannot learn​​ from scratch by random​​​‌ exploration in a new​ environment, standard model-free RL​‌ approaches cannot be used.​​ Some sort of previous​​​‌ knowledge on the environment​ or a similar one​‌ should be exploited. Finally,​​ given that the robot​​​‌ moves within a populated​ environment, it is desirable​‌ to have the capability​​ to enforce certain constrains,​​​‌ thus limiting the range​ of possible robot actions.​‌

Building algorithms to endow​​ robots with autonomous decision​​​‌ taking is not straightforward.​ Two relatively distinct paradigms​‌ are available the literature.​​ First, one can devise​​ customized strategies based on​​​‌ techniques such as robot‌ motion planning combined with‌​‌ sensor-based robot control.​​ These techniques lack generalization,​​​‌ in particular when the‌ robot acts in complex,‌​‌ dynamic and unconstrained environments.​​ Second, one can let​​​‌ the robot devise its‌ own strategies based on‌​‌ reinforcement learning (RL) –​​ a machine learning paradigm​​​‌ in which “agents" learn‌ by themselves by trial‌​‌ and error to achieve​​ successful strategies60.​​​‌ It is very difficult,‌ however, to enforce any‌​‌ kind of soft- or​​ hard-constraint within this framework.​​​‌ We will showcase these‌ two scientific streams with‌​‌ one group of techniques​​ for each one: model​​​‌ predictive control (MPC) and‌ Q-learning, deep Q-networks (DQNs),‌​‌ more precisely. These two​​ techniques are promising. Moreover,​​​‌ they are well documented‌ in the robotics and‌​‌ machine learning. Nevertheless, combining​​ them is extremely challenging.​​​‌

An additional challenge, independent‌ from the learning and‌​‌ control combination foreseen, is​​ the data distribution gap​​​‌ between the simulations and‌ the real-world. Meta-learning, or‌​‌ the ability to learn​​ how to learn, can​​​‌ provide partial answers to‌ this problem. Indeed, developing‌​‌ machine learning methods able​​ to understand how the​​​‌ learning is achieved can‌ be used to extend‌​‌ this learning to a​​ new task and speed​​​‌ up the learning process‌ on the new task.‌​‌ Recent developments proposed meta-learning​​ strategies specifically conceived for​​​‌ reinforcement learning, leading to‌ Meta-RL methods. One promising‌​‌ trend in Meta-RL is​​ to have a probabilistic​​​‌ formulation involving SSMs and‌ VAEs, i.e. hence sharing‌​‌ the methodology based on​​ dynamical variational autoencoders described​​​‌ before. Very importantly, we‌ are not aware of‌​‌ any studies able to​​ combine Meta-RL with MPC​​​‌ to handle the constraints,‌ and within a unified‌​‌ formulation. From a methodological​​ perspective, this is an​​​‌ important challenge we face‌ in the next few‌​‌ years.

5.1​​​‌ Impact of research results‌

Our line of research‌​‌ on developing unsupervised learning​​ methods exploiting audio-visual data​​​‌ to understand social scenes‌ and to learn to‌​‌ interact within is very​​ interesting and challenging, and​​​‌ has large economical and‌ societal impact. Economical impact‌​‌ since the auditory and​​ visual sensors are the​​​‌ most common one, and‌ we can find (many‌​‌ of) them in almost​​ every smartphone in the​​​‌ market. Beyond telephones, manufacturers‌ designing new systems meant‌​‌ for human use, should​​​‌ take into account the​ need for verbal interaction,​‌ and hence for audio-visual​​ perception. A clear example​​​‌ of this potential is​ the transfer of our​‌ technology to a real​​ robotic platform, for evaluation​​​‌ within a day-care hospital​ (DCH). This is possible​‌ thanks to the H2020​​ SPRING EU project, that​​​‌ assesses the interest of​ social robotics in the​‌ non-medical phases of a​​ regular day for elder​​​‌ patients in a DCH.​ We are evaluating the​‌ performance of our methods​​ for AV speaker tracking,​​​‌ AV speech enhancement, and​ AV sound source separation,​‌ for future technology transfer​​ to the robot manufacturer.​​​‌ This is the first​ step toward a robot​‌ that can be part​​ of the social environment​​​‌ of the DCH, helping​ to reduce patient and​‌ companion stress, at the​​ same time as being​​​‌ a useful tool for​ the medical personnel. We​‌ are confident that developing​​ robust AV perception and​​​‌ action capabilities for robots​ and autonomous systems, will​‌ make them more suitable​​ for environments populated with​​​‌ humans.

6.1 Final​‌ results of the H2020​​ SPRING project

As the​​​‌ H2020 SPRING project concludes,​ these joint results highlight​‌ the potential of socially​​ assistive robots (SARs) in​​​‌ geriatric care. This research​ evaluated the humanoid robot​‌ ARI in a Paris​​ day-care hospital, focusing on​​​‌ its ability to support​ older adults and caregivers​‌ through multi-modal conversational dialogue.​​ Across several experimental waves​​​‌ involving over 120 participants,​ the studies assessed system​‌ performance, user engagement, and​​ the impact of Large​​​‌ Language Model (LLM) integration.​ Results from the Acceptability​‌ E-Scale (AES) and System​​ Usability Scale (SUS) indicate​​​‌ that end-users are highly​ receptive to this technology.​‌ Key findings demonstrate that​​ while LLMs improve interaction​​​‌ fluency, overall success depends​ on the robot's ability​‌ to minimize errors in​​ cluttered, real-world environments. The​​​‌ study also identified that​ personal user characteristics and​‌ robot adaptability significantly influence​​ long-term adoption and emotional​​​‌ engagement. Ultimately, robust perception​ and flexible action skills​‌ proved essential for moving​​ beyond lab settings into​​​‌ dynamic clinical facilities. These​ contributions provide a vital​‌ framework for deploying AI-driven​​ robotics to alleviate healthcare​​​‌ workloads and reduce patient​ loneliness. By bridging the​‌ gap between technical development​​ and clinical reality, SPRING​​​‌ has paved the way​ for future geriatric assistive​‌ technologies 26, 27​​.

6.2 Onboarding of​​​‌ Stéphane Lathuilère

A significant​ milestone in the team's​‌ recent evolution was the​​ arrival of Stéphane Lathuilière,​​​‌ who joined as a​ permanent Research Scientist (ISFP)​‌ in January 2025. His​​ integration into RobotLearn—and subsequently​​​‌ ComLearn, see below—brings specialized​ expertise in deep generative​‌ models, image and video​​ generation, and multimodal learning.​​​‌ Having previously served as​ an Associate Professor at​‌ Télécom Paris and completed​​ his PhD within the​​​‌ predecessor Perception team at​ Inria, Stéphane provides a​‌ vital bridge between high-level​​ scene perception and the​​​‌ synthesis of realistic social​ signals. His research focus​‌ on generative AI and​​ "Human Behavior Understanding" directly​​​‌ supports the new team's​ mission to develop Multimodal​‌ Foundation Models (MFMs). By​​ strengthening the "generation" pillar​​ of the team, his​​​‌ presence accelerates the development‌ of artificial agents capable‌​‌ of more fluid, context-sensitive,​​ and human-centric interactions.

6.3​​​‌ The genesis of ComLearn‌

The creation of ComLearn‌​‌ marks a strategic merger​​ between the CRISSP (GIPSA-lab)​​​‌ and RobotLearn (Inria) teams,‌ unifying their world-class expertise‌​‌ in speech synthesis and​​ computer vision. By combining​​​‌ CRISSP’s mastery of multimodal‌ generation with RobotLearn's advanced‌​‌ audiovisual perception, ComLearn establishes​​ a powerhouse for next-generation​​​‌ social robotics. This synergy‌ aims to overcome the‌​‌ "last mile" of human-agent​​ interaction by developing Multimodal​​​‌ Foundation Models (MFMs) that‌ ground reasoning and generation‌​‌ in real-world communicative environments.​​ Leveraging a shared methodological​​​‌ foundation in Deep Generative‌ Models, the team will‌​‌ design artificial agents capable​​ of seamless, context-sensitive dialogue​​​‌ within multi-party groups. Beyond‌ technical innovation, the project‌​‌ serves as a bridge​​ between signal processing and​​​‌ cognitive science, providing tools‌ to simulate and better‌​‌ understand fundamental human communication​​ mechanisms. The merger provides​​​‌ the critical mass necessary‌ to lead international research‌​‌ in audio-visual scene analysis​​ and user-adaptive assistive technologies.​​​‌ Ultimately, ComLearn will empower‌ social robots to navigate‌​‌ complex, cluttered social spaces​​ with unprecedented fluency and​​​‌ interpretability.

6.4 Welcome Miroka!‌

The acquisition of a‌​‌ Miroka robotic platform represents​​ a transformative step for​​​‌ the RobotLearn/ComLearn team, providing‌ a state-of-the-art vehicle for‌​‌ testing Multimodal Foundation Models​​ (MFMs) in real-world settings.​​​‌ Unlike traditional platforms, Miroka's‌ unique globe-based locomotion and‌​‌ "character-driven" design allow it​​ to navigate crowded hospital​​​‌ environments with an agility‌ and social presence that‌​‌ mimics human movement. This​​ platform serves as the​​​‌ ideal physical anchor to‌ ground the team's research‌​‌ in audiovisual perception and​​ generative social signals, bridging​​​‌ the gap between theoretical‌ AI and embodied interaction.‌​‌ Its expressive animated interface​​ provides a high-fidelity canvas​​​‌ for our work in‌ generative behavior synthesis, enabling‌​‌ more nuanced and emotionally​​ resonant communication. Furthermore, Miroka's​​​‌ specialized social capabilities allow‌ the team to study‌​‌ complex multi-party interactions. This​​ investment ensures the team​​​‌ remains at the global‌ forefront of social robotics,‌​‌ moving beyond basic dialogue​​ to truly integrated, context-sensitive​​​‌ assistance. Ultimately, Miroka transforms‌ the lab's algorithmic breakthroughs‌​‌ into tangible, observable social​​ behaviors.

7.1 New platforms

Participants:‌​‌ Xavier Alameda-Pineda, Ahamed​​ Mohamed, Stéphane Lathuiliere​​​‌, Nicolas Turro,‌ Soraya Arias.

During‌​‌ 2025, the RobotLearn team​​ has acquired the Miroka​​​‌ platform, see Figure 3‌ (right). This platform is‌​‌ built by EnchantedTools (a​​ startup in Paris). It​​​‌ has some similarities with‌ our previous platform ARI‌​‌ (that we will keep),​​ namely: the soft appearance,​​​‌ the design intenteded for‌ social interaction, and multi-sensory‌​‌ capabilities. However, it has​​ some important differences. First,​​​‌ Miroka's face is projected,‌ and therefore much more‌​‌ expressive than the static​​ face of ARI. Second,​​​‌ Miroka comes with integrated‌ LIDAR, which would potentially‌​‌ and significally help its​​ navigation skills. Third, Miroka​​​‌ moves with a self-balancing‌ strategy over a sphere.‌​‌ While this is more​​ complex to handle, it​​​‌ means that Miroka is‌ a holonomic robot and‌​‌ can move in any​​​‌ direction. We hope it​ simplifies the issues related​‌ to “manouvering” in social​​ settings.

The acquisition of​​​‌ a Miroka robotic platform​ represents a transformative step​‌ for the RobotLearn/ComLearn team,​​ providing a state-of-the-art vehicle​​​‌ for testing Multimodal Foundation​ Models (MFMs) in real-world​‌ settings. Unlike traditional platforms,​​ Miroka's unique globe-based locomotion​​​‌ and "character-driven" design allow​ it to navigate crowded​‌ hospital environments with an​​ agility and social presence​​​‌ that mimics human movement.​ This platform serves as​‌ the ideal physical anchor​​ to ground the team's​​​‌ research in audiovisual perception​ and generative social signals,​‌ bridging the gap between​​ theoretical AI and embodied​​​‌ interaction. Its expressive animated​ interface provides a high-fidelity​‌ canvas for our work​​ in generative behavior synthesis,​​​‌ enabling more nuanced and​ emotionally resonant communication. Furthermore,​‌ Miroka's specialized social capabilities​​ allow the team to​​​‌ study complex multi-party interactions.​ This investment ensures the​‌ team remains at the​​ global forefront of social​​​‌ robotics, moving beyond basic​ dialogue to truly integrated,​‌ context-sensitive assistance. Ultimately, Miroka​​ transforms the lab's algorithmic​​​‌ breakthroughs into tangible, observable​ social behaviors.

8.1​ Deep Probabilistic Models

8.2​ Human Behavior Understanding

8.3 Learning and‌ Control for Social Robots‌​‌

8.4​ Integrating a Large Language​‌ Model Into a Socially​​ Assistive Robot in a​​​‌ Hospital Geriatric Unit: Two-Wave​ Comparative Study on Performance,​‌ Engagement, and User Perceptions​​

Participants: Xavier Alameda Pineda​​​‌.

Healthcare systems struggle​ to meet the complex​‌ needs of older adults​​ in resource-limited settings. Socially​​​‌ assistive robots (SARs) offer​ a potential solution by​‌ providing information and practical​​ support. This study evaluated​​​‌ the integration of Large​ Language Models (LLMs) into​‌ SARs to improve interaction​​ fluency. Researchers compared a​​​‌ basic dialogue system (Wave​ 1) to an LLM-based​‌ system (Wave 2). The​​ evaluation focused on system​​​‌ performance, interaction success, and​ multidimensional user engagement. Conducted​‌ over eight months in​​ a Paris geriatric hospital,​​​‌ the study involved 28​ participants aged 60+. Interactions​‌ were video-recorded to code​​ for technical errors and​​​‌ verbal, physical, and emotional​ engagement. Validated scales were​‌ used to measure the​​ robot's overall usability and​​​‌ user acceptability. Results analyzed​ how user characteristics influenced​‌ perceptions of the LLM-enhanced​​ technology. The findings aim​​​‌ to minimize conversational errors​ and optimize SAR adaptability​‌ for real-world use. This​​ research provides insights into​​​‌ successfully deploying AI-driven robotics​ in geriatric care.

8.5​‌ Acceptability and Usability of​​ a Socially Assistive Robot​​​‌ Integrated With a Large​ Language Model for Enhanced​‌ Human-Robot Interaction in a​​ Geriatric Care Institution: Mixed​​​‌ Methods Evaluation

Participants: Xavier​ Alameda Pineda.

Socially​‌ assistive robots (SARs) aim​​ to support older adults​​​‌ and clinicians by promoting​ well-being and managing routine​‌ tasks. However, ensuring high​​ levels of acceptability and​​​‌ usability remains a significant​ hurdle in dynamic care​‌ settings. This study evaluated​​ these factors by deploying​​​‌ the ARI humanoid robot​ in a geriatric day​‌ care hospital. Over one​​ year, 97 participants—comprising 65​​​‌ older patients and 32​ informal caregivers—engaged with the​‌ robot. The evaluation took​​ place in a waiting​​​‌ area in Paris, where​ ARI utilized voice-based dialogue​‌ for interaction. Researchers employed​​ a mixed-methods approach to​​​‌ capture a holistic view​ of the user experience.​‌ Quantitative data were gathered​​ through the Acceptability E-scale​​​‌ and the System Usability​ Scale. These assessments were​‌ administered orally to accommodate​​ the participants' accessibility needs.​​​‌ Qualitative feedback was also​ collected to identify subjective​‌ perceptions and specific contextual​​ barriers. The study sought​​​‌ to pinpoint key factors​ influencing the adoption of​‌ SARs by both patients​​ and caregivers. Ultimately, the​​ findings provide a framework​​​‌ for improving robot integration‌ into real-world geriatric environments.‌​‌

9.1 International initiatives

9.2 International​​​‌ research visitors

9.3​​​‌ European initiatives

10.1 Promoting scientific activities​​

10.2 Teaching‌​‌ - Supervision - Juries​​ - Educational and pedagogical​​​‌ outreach

11.1 Major​ publications

• 1 articleL.​‌Louis Airale, D.​​Dominique Vaufreydaz and X.​​​‌Xavier Alameda-Pineda. SocialInteractionGAN:​ Multi-person Interaction Sequence Generation​‌.IEEE Transactions on​​ Affective ComputingMay 2022​​​‌HALDOI
• 2 article​Y.Yutong Ban,​‌ X.Xavier Alameda-Pineda,​​ C.Christine Evers and​​​‌ R.Radu Horaud.​ Tracking Multiple Audio Sources​‌ with the Von Mises​​ Distribution and Variational EM​​​‌.IEEE Signal Processing​ Letters266June​‌ 2019, 798 -​​ 802HALDOI
• 3​​​‌ articleY.Yutong Ban​, X.Xavier Alameda-Pineda​‌, L.Laurent Girin​​ and R.Radu Horaud​​​‌. Variational Bayesian Inference​ for Audio-Visual Tracking of​‌ Multiple Speakers.IEEE​​ Transactions on Pattern Analysis​​​‌ and Machine Intelligence43​5May 2021,​‌ 1761-1776HALDOIback​​ to text
• 4 article​​​‌X.Xiaoyu Bie,​ S.Simon Leglaive,​‌ X.Xavier Alameda-Pineda and​​ L.Laurent Girin.​​​‌ Unsupervised Speech Enhancement using​ Dynamical Variational Autoencoders.​‌IEEE/ACM Transactions on Audio,​​ Speech and Language Processing​​​‌30September 2022,​ 2993 - 3007HAL​‌DOI
• 5 inproceedingsG.​​Guillaume Delorme, Y.​​​‌Yihong Xu, S.​Stéphane Lathuilière, R.​‌Radu Horaud and X.​​Xavier Alameda-Pineda. CANU-ReID:​​​‌ A Conditional Adversarial Network​ for Unsupervised person Re-IDentification​‌.ICPR 2020 -​​ 25th International Conference on​​​‌ Pattern RecognitionMilano, Italy​IEEE2021, 4428-4435​‌HALDOI
• 6 article​​G.Georgios Evangelidis and​​​‌ R.Radu Horaud.​ Joint Alignment of Multiple​‌ Point Sets with Batch​​ and Incremental Expectation-Maximization.​​​‌IEEE Transactions on Pattern​ Analysis and Machine Intelligence​‌406June 2018​​, 1397 - 1410​​​‌HALDOI
• 7 article​I.Israel Gebru,​‌ S.Sileye Ba,​​ X.Xiaofei Li and​​​‌ R.Radu Horaud.​ Audio-Visual Speaker Diarization Based​‌ on Spatiotemporal Bayesian Fusion​​.IEEE Transactions on​​​‌ Pattern Analysis and Machine​ Intelligence405July​‌ 2018, 1086 -​​ 1099HALDOI
• 8​​​‌ articleL.Laurent Girin​, S.Simon Leglaive​‌, X.Xiaoyu Bie​​, J.Julien Diard​​​‌, T.Thomas Hueber​ and X.Xavier Alameda-Pineda​‌. Dynamical Variational Autoencoders:​​ A Comprehensive Review.​​​‌Foundations and Trends in​ Machine Learning151-2​‌December 2021, 1-175​​HALDOI
• 9 article​​​‌Z.Zhiqi Kang,​ M.Mostafa Sadeghi,​‌ R.Radu Horaud and​​ X.Xavier Alameda-Pineda.​​​‌ Expression-preserving face frontalization improves​ visually assisted speech processing​‌.International Journal of​​ Computer VisionJanuary 2023​​​‌HALDOIback to​ text
• 10 articleS.​‌Stéphane Lathuilière, B.​​Benoît Massé, P.​​Pablo Mesejo and R.​​​‌Radu Horaud. Neural‌ Network Based Reinforcement Learning‌​‌ for Audio-Visual Gaze Control​​ in Human-Robot Interaction.​​​‌Pattern Recognition Letters118‌February 2019, 61-71‌​‌HALDOI
• 11 article​​S.Stéphane Lathuilière,​​​‌ P.Pablo Mesejo,‌ X.Xavier Alameda-Pineda and‌​‌ R.Radu Horaud.​​ A Comprehensive Analysis of​​​‌ Deep Regression.IEEE‌ Transactions on Pattern Analysis‌​‌ and Machine Intelligence42​​9September 2020,​​​‌ 2065-2081HALDOI
• 12‌ articleX.Xiaofei Li‌​‌, Y.Yutong Ban​​, L.Laurent Girin​​​‌, X.Xavier Alameda-Pineda‌ and R.Radu Horaud‌​‌. Online Localization and​​ Tracking of Multiple Moving​​​‌ Speakers in Reverberant Environments‌.IEEE Journal of‌​‌ Selected Topics in Signal​​ Processing131March​​​‌ 2019, 88-103HAL‌DOI
• 13 articleX.‌​‌Xiaofei Li, S.​​Sharon Gannot, L.​​​‌Laurent Girin and R.‌Radu Horaud. Multichannel‌​‌ Identification and Nonnegative Equalization​​ for Dereverberation and Noise​​​‌ Reduction based on Convolutive‌ Transfer Function.IEEE/ACM‌​‌ Transactions on Audio, Speech​​ and Language Processing26​​​‌10May 2018,‌ 1755-1768HALDOI
• 14‌​‌ articleX.Xiaofei Li​​, L.Laurent Girin​​​‌, S.Sharon Gannot‌ and R.Radu Horaud‌​‌. Multichannel Speech Separation​​ and Enhancement Using the​​​‌ Convolutive Transfer Function.‌IEEE/ACM Transactions on Audio,‌​‌ Speech and Language Processing​​273March 2019​​​‌, 645-659HALDOI‌
• 15 articleX.Xiaofei‌​‌ Li, S.Simon​​ Leglaive, L.Laurent​​​‌ Girin and R.Radu‌ Horaud. Audio-noise Power‌​‌ Spectral Density Estimation Using​​ Long Short-term Memory.​​​‌IEEE Signal Processing Letters‌266June 2019‌​‌, 918-922HALDOI​​
• 16 articleX.Xiaoyu​​​‌ Lin, L.Laurent‌ Girin and X.Xavier‌​‌ Alameda-Pineda. Mixture of​​ Dynamical Variational Autoencoders for​​​‌ Multi-Source Trajectory Modeling and‌ Separation.Transactions on‌​‌ Machine Learning Research Journal​​2024, 1-19HAL​​​‌
• 17 articleB.Benoît‌ Massé, S.Silèye‌​‌ Ba and R.Radu​​ Horaud. Tracking Gaze​​​‌ and Visual Focus of‌ Attention of People Involved‌​‌ in Social Interaction.​​IEEE Transactions on Pattern​​​‌ Analysis and Machine Intelligence‌4011November 2018‌​‌, 2711 - 2724​​HALDOI
• 18 article​​​‌M.Mostafa Sadeghi,‌ S.Simon Leglaive,‌​‌ X.Xavier Alameda-Pineda,​​ L.Laurent Girin and​​​‌ R.Radu Horaud.‌ Audio-Visual Speech Enhancement Using‌​‌ Conditional Variational Auto-Encoders.​​IEEE/ACM Transactions on Audio,​​​‌ Speech and Language Processing‌28May 2020,‌​‌ 1788-1800HALDOIback​​ to text
• 19 article​​​‌S.Samir Sadok,‌ S.Simon Leglaive,‌​‌ L.Laurent Girin,​​ X.Xavier Alameda-Pineda and​​​‌ R.Renaud Séguier.‌ A Multimodal Dynamical Variational‌​‌ Autoencoder for Audiovisual Speech​​ Representation Learning.Neural​​​‌ Networks172April 2024‌, 106120HALDOI‌​‌
• 20 articleA.Aliaksandr​​ Siarohin, G.Gloria​​​‌ Zen, C.Cveta‌ Majtanovic, X.Xavier‌​‌ Alameda-Pineda, E.Elisa​​ Ricci and N.Nicu​​​‌ Sebe. Increasing Image‌ Memorability with Neural Style‌​‌ Transfer.ACM Transactions​​ on Multimedia Computing, Communications​​​‌ and Applications152‌June 2019HALDOI‌​‌
• 21 inproceedingsL.Lorenzo​​​‌ Vaquero, Y.Yihong​ Xu, X.Xavier​‌ Alameda-Pineda, V. M.​​Victor M. Brea and​​​‌ M.Manuel Mucientes.​ Lost and Found: Overcoming​‌ Detector Failures in Online​​ Multi-Object Tracking.ECCV​​​‌ 24 - 18th European​ Conference on Computer Vision​‌Milan (Italie), ItalyJuly​​ 2024, 1-30HAL​​​‌
• 22 articleD.Dan​ Xu, X.Xavier​‌ Alameda-Pineda, W.Wanli​​ Ouyang, E.Elisa​​​‌ Ricci, X.Xiaogang​ Wang and N.Nicu​‌ Sebe. Probabilistic Graph​​ Attention Network with Conditional​​​‌ Kernels for Pixel-Wise Prediction​.IEEE Transactions on​‌ Pattern Analysis and Machine​​ Intelligence445May​​​‌ 2022, 2673-2688HAL​DOI
• 23 articleY.​‌Yihong Xu, Y.​​Yutong Ban, G.​​​‌Guillaume Delorme, C.​Chuang Gan, D.​‌Daniela Rus and X.​​Xavier Alameda-Pineda. TransCenter:​​​‌ Transformers With Dense Representations​ for Multiple-Object Tracking.​‌IEEE Transactions on Pattern​​ Analysis and Machine Intelligence​​​‌November 2022, 1-16​HALDOI
• 24 article​‌G.Guanglei Yang,​​ E.Enrico Fini,​​​‌ D.Dan Xu,​ P.Paolo Rota,​‌ M.Mingli Ding,​​ M.Moin Nabi,​​​‌ X.Xavier Alameda-Pineda and​ E.Elisa Ricci.​‌ Uncertainty-aware Contrastive Distillation for​​ Incremental Semantic Segmentation.​​​‌IEEE Transactions on Pattern​ Analysis and Machine Intelligence​‌March 2022, 1-14​​HALDOI
• 25 article​​​‌G.Guanglei Yang,​ E.Enrico Fini,​‌ D.Dan Xu,​​ P.Paolo Rota,​​​‌ M.Mingli Ding,​ H.Hao Tang,​‌ X.Xavier Alameda-Pineda and​​ E.Elisa Ricci.​​​‌ Continual Attentive Fusion for​ Incremental Learning in Semantic​‌ Segmentation.IEEE Transactions​​ on MultimediaApril 2022​​​‌HALDOI

11.2 Publications​ of the year

11.3 Cited publications​​

• 45 inproceedingsT.Triantafyllos​​​‌ Afouras, A.Andrew​ Owens, J. S.​‌Joon Son Chung and​​ A.Andrew Zisserman.​​​‌ Self-supervised learning of audio-visual​ objects from video.​‌Computer Vision--ECCV 2020: 16th​​ European Conference, Glasgow, UK,​​​‌ August 23--28, 2020, Proceedings,​ Part XVIII 16Springer​‌2020, 208--224back​​ to text
• 46 article​​​‌S.Sileye Ba,​ X.Xavier Alameda-Pineda,​‌ A.Alessio Xompero and​​ R.Radu Horaud.​​​‌ An On-line Variational Bayesian​ Model for Multi-Person Tracking​‌ from Cluttered Scenes.​​Computer Vision and Image​​​‌ Understanding153December 2016​, 64--76HALDOI​‌back to text
• 47​​ miscA.Anand Ballou​​, X.Xavier Alameda-Pineda​​​‌ and C.Chris Reinke‌. Variational Meta Reinforcement‌​‌ Learning for Social Robotics​​.December 2022HAL​​​‌back to text
• 48‌ inproceedingsY.Yutong Ban‌​‌, X.Xavier Alameda-Pineda​​, F.Fabien Badeig​​​‌, S.Sileye Ba‌ and R.Radu Horaud‌​‌. Tracking a Varying​​ Number of People with​​​‌ a Visually-Controlled Robotic Head‌.IEEE/RSJ International Conference‌​‌ on Intelligent Robots and​​ SystemsVancouver, CanadaIEEE​​​‌September 2017, 4144-4151‌HALDOIback to‌​‌ text
• 49 articleY.​​Yutong Ban, X.​​​‌Xavier Alameda-Pineda, C.‌Christine Evers and R.‌​‌Radu Horaud. Tracking​​ Multiple Audio Sources with​​​‌ the Von Mises Distribution‌ and Variational EM.‌​‌IEEE Signal Processing Letters​​266June 2019​​​‌, 798 - 802‌HALDOIback to‌​‌ text
• 50 inproceedingsD.​​Drażen Bršċić, H.​​​‌Hiroyuki Kidokoro, Y.‌Yoshitaka Suehiro and T.‌​‌Takayuki Kanda. Escaping​​ from children's abuse of​​​‌ social robots.Proceedings‌ of the tenth annual‌​‌ acm/ieee international conference on​​ human-robot interaction2015,​​​‌ 59--66back to text‌
• 51 inproceedingsW.-L.Wan-Ling‌​‌ Chang, J. P.​​Jeremy P White,​​​‌ J.Joohyun Park,‌ A.Anna Holm and‌​‌ S.Selma Šabanović.​​ The effect of group​​​‌ size on people's attitudes‌ and cooperative behaviors toward‌​‌ robots in interactive gameplay​​.RO-MAN International Symposium​​​‌ on Robot and Human‌ Interactive CommunicationIEEE2012‌​‌, 845--850back to​​ text
• 52 inproceedingsC.​​​‌Changan Chen, U.‌Unnat Jain, C.‌​‌Carl Schissler, S.​​ V.Sebastia Vicenc Amengual​​​‌ Gari, Z.Ziad‌ Al-Halah, V. K.‌​‌Vamsi Krishna Ithapu,​​ P.Philip Robinson and​​​‌ K.Kristen Grauman.‌ Soundspaces: Audio-visual navigation in‌​‌ 3d environments.Computer​​ Vision--ECCV 2020: 16th European​​​‌ Conference, Glasgow, UK, August‌ 23--28, 2020, Proceedings, Part‌​‌ VI 16Springer2020​​, 17--36back to​​​‌ text
• 53 articleM.‌ E.Mary Ellen Foster‌​‌, A.Andre Gaschler​​ and M.Manuel Giuliani​​​‌. Automatically classifying user‌ engagement for dynamic multi-party‌​‌ human--robot interaction.International​​ Journal of Social Robotics​​​‌952017,‌ 659--674back to text‌​‌
• 54 inproceedingsR.Ruohan​​ Gao and K.Kristen​​​‌ Grauman. Visualvoice: Audio-visual‌ speech separation with cross-modal‌​‌ consistency.IEEE/CVF CVPR​​2021back to text​​​‌
• 55 articleL.Laurent‌ Girin, S.Simon‌​‌ Leglaive, X.Xiaoyu​​ Bie, J.Julien​​​‌ Diard, T.Thomas‌ Hueber and X.Xavier‌​‌ Alameda-Pineda. Dynamical Variational​​ Autoencoders: A Comprehensive Review​​​‌.Foundations and Trends‌ in Machine Learning15‌​‌1-2December 2021,​​ 1-175HALDOIback​​​‌ to textback to‌ text
• 56 articleX.‌​‌Xiaofei Li, Y.​​Yutong Ban, L.​​​‌Laurent Girin, X.‌Xavier Alameda-Pineda and R.‌​‌Radu Horaud. Online​​ Localization and Tracking of​​​‌ Multiple Moving Speakers in‌ Reverberant Environments.IEEE‌​‌ Journal of Selected Topics​​ in Signal Processing13​​​‌1March 2019,‌ 88-103HALDOIback‌​‌ to text
• 57 misc​​X.Xiaoyu Lin,​​​‌ L.Laurent Girin and‌ X.Xavier Alameda-Pineda.‌​‌ Unsupervised Multiple-Object Tracking with​​​‌ a Dynamical Variational Autoencoder​.February 2022HAL​‌back to text
• 58​​ miscC.Chris Reinke​​​‌ and X.Xavier Alameda-Pineda​. Successor Feature Representations​‌.May 2022HAL​​back to text
• 59​​​‌ articleS.Sarah Sebo​, B.Brett Stoll​‌, B.Brian Scassellati​​ and M. F.Malte​​​‌ F Jung. Robots​ in groups and teams:​‌ a literature review.​​Proceedings of the ACM​​​‌ on Human-Computer Interaction4​CSCW22020, 1--36​‌back to text
• 60​​ bookR. S.Richard​​​‌ S Sutton and A.​ G.Andrew G Barto​‌. Reinforcement learning: An​​ introduction.MIT press​​​‌2018back to text​
• 61 articleM.Mateusz​‌ Żarkowski. Multi-party turn-taking​​ in repeated human--robot interactions:​​​‌ an interdisciplinary evaluation.​International Journal of Social​‌ Robotics1152019​​, 693--707back to​​​‌ text
• 62 articleJ.​Jingwei Zhang, L.​‌Lei Tai, P.​​Peng Yun, Y.​​​‌Yufeng Xiong, M.​Ming Liu, J.​‌Joschka Boedecker and W.​​Wolfram Burgard. Vr-goggles​​​‌ for robots: Real-to-sim domain​ adaptation for visual control​‌.IEEE Robotics and​​ Automation Letters42​​​‌2019, 1148--1155back​ to text