4.1 Education

Education is at the core of the motivations of the Potioc group. Indeed, we are convinced that the approaches we investigate—which target motivation, curiosity, pleasure of use and high level of interactivity—may serve education purposes. To this end, we collaborate with experts in Educational Sciences and teachers for exploring new interactive systems that enhance learning processes. We are currently investigating the fields of astronomy, optics, and neurosciences. We have also worked with special education centres for the blind on accessible augmented reality prototypes. Currently, we collaborate with teachers to enhance collaborative work for K-12 pupils. In the future, we will continue exploring new interactive approaches dedicated to education, in various fields. Popularization of Science is also a key domain for Potioc. Focusing on this subject allows us to get inspiration for the development of new interactive approaches.

4.2 Art

Art, which is strongly linked with emotions and user experiences, is also a target area for Potioc. We believe that the work conducted in Potioc may be beneficial for creation from the artist point of view, and it may open new interactive experiences from the audience point of view. As an example, we have worked with colleagues who are specialists in digital music, and with musicians. We have also worked with jugglers and we are currently working with a scenographer with the goal of enhancing interactivity of physical mockups and improve user experience.

4.3 Entertainment

Similarly, entertainment is a domain where our work may have an impact. We notably explored BCI-based gaming and non-medical applications of BCI, as well as mobile Augmented Reality games. Once again, we believe that our approaches that merge the physical and the virtual world may enhance the user experience. Exploring such a domain will raise numerous scientific and technological questions.

4.4 Well-being

Finally, well-being is a domain where the work of Potioc can have an impact. We have notably shown that spatial augmented reality and tangible interaction may favor mindfulness activities, which have been shown to be beneficial for well-being. More generally, we explore introspectibles objects, which are tangible and augmented objects that are connected to physiological signals and that foster introspection. We explore these directions for general public, including people with special needs.

5.1 Accessible e-learning tools and accessible board games

Among our explorations in Human-Computer Interaction, we have been interested in designing and developing tools that can be used by people suffering from physiological or cognitive disorders.

In particular, we used the PapARt tool developed in the team to contribute making board games accessible for people suffering from visual impairment 16.

We also designed a MOOC player, called AIANA, in order to ease the access of people with cognitive disorders (attention, memory).

5.2 Augmented reality for environmental challenges

Face to the big challenge of climate change, we are currently orienting our research towards approaches that may contribute to pro-environmental behaviors. To do so, we are framing research directions and building projects with the objective of putting our expertise in HCI, visualization, and mixed-reality to work for the reduction of human impact on the planet.

5.3 Humanitarian information visualization

Members of the team have been involved in research promoting humanitarian goals, such as studying how to design visualizations of humanitarian data in a way that promotes charitable donations to traveling migrants and other populations in need. See sec:anthropographics.

6.1 Awards

Best paper award for :

• IEEE VIS 2021: Wesley Willett, Bon Adriel Aseniero, Sheelagh Carpendale, Pierre Dragicevic, Yvonne Jansen, Lora Oehlberg, and Petra Isenberg. "Perception! Immersion! Empowerment!: Superpowers as Inspiration for Visualization." IEEE Transactions on Visualization and Computer Graphics (2021) 21.
• ISS 2021: Lauren Thevin, Nicolas Rodier, Bernard Oriola, Martin Hachet, Christophe Jouffrais, Anke M. Brock. Inclusive Adaptation of Existing Board Games for Gamers with and without Visual Impairments using a Spatial Augmented Reality Framework for Touch Detection and Audio Feedback 16.

Young investigator award for :

• A. Appriou, F. Lotte, “Tools for affective, cognitive and conative states estimation from both EEG and physiological signals”, Proceedings of the Third International Neuroergonomics Conference, 2021.

Best student talk for :

• J. Mladenovic, J. Frey, J. Mattout, F. Lotte, "Biased feedback influences learning of Motor Imagery BCI", 8th International BCI Meeting, 2021.

Best student poster for :

• M.S. Yamamoto, K. Sadatnejad, R. M. Islam, F. Lotte, T. Tanaka. "Reliable outlier detection by spectral clustering on Riemannian manifold of EEG covariance matrix", 8th International BCI meeting, 2021.

7.1 New software

7.2 New platforms

7.2.4 Brain Hero

Participants: Thibaut Monseigne, Aline Roc, Fabien Lotte.

Brain Hero is a gamification of the traditional feedback of mental imagery BCI protocols. The principle is inspired from rhythmic games such as Guitar Hero in order to make the BCI sessions more interesting for the subjects. The game was implemented under Unity and communicates with the OpenViBE software (used in the team for BCI protocols) using the LSL (Lab Straming Layer) protocol.

This picture represents the visual interface of the Brain Hero visual BCI training environment. It looks like notes on a guitar FretBoard, each note representing a specific mental task for the BCI.

8.1 Avatars in Augmented Reality

Participants: Adélaïde Genay, Martin Hachet.

External collaborators: Anatole Lécuyer [Inria Rennes - Hybrid].

In 2021, we have continued our work on avatarization in augmented reality. The study described in our 2020 activity report about how surrounding objects influence the sense of embodiment in optical see-through experiences has been published in ”Frontiers in Virtual Reality” 15.

In addition, we have proposed a literature review published in ”TVCG” 14 covering the embodiment of virtual self-avatars in AR. Our goal was (i) to guide readers through the different options and challenges linked to the implementation of AR embodiment systems, (ii) to provide a better understanding of AR embodiment perception by classifying the existing knowledge, and (iii) to offer insight on future research topics and trends for AR and avatar research. To do so, we introduce a taxonomy of virtual embodiment experiences by defining a “body avatarization” continuum (see Figure 4). The presented knowledge suggests that the sense of embodiment evolves in the same way in AR as in other settings, but this possibility has yet to be fully investigated. We suggest that, whilst it is yet to be well understood, the embodiment of avatars has a promising future in AR and conclude by discussing possible directions for research.

We are now starting a collaboration with NARA Institute of technology - CARE lab (Japan), to push these explorations further.

A scheme shows the continuum.

8.2 Control of Facial Expressions in VR

Participants: Martin Hachet.

External collaborators: Marc Baloup [Inria Lille - Loki], Gery Casiez [Univ Lille - Loki], Thomas Pietrzak [Univ Lille - Loki].

This work, published in VRST 2021 23, is at the heart of the PhD project from Marc Baloup, Inria Lille - Loki Team, co-advised by Martin Hachet.

The control of an avatar's facial expressions in virtual reality is mainly based on the automated recognition and transposition of the user's facial expressions. These isomorphic techniques are limited to what users can convey with their own face and have recognition issues. To overcome these limitations, non-isomorphic techniques rely on interaction techniques using input devices to control the avatar's facial expressions. Such techniques need to be designed to quickly and easily select and control an expression, and not disrupt a main task such as talking. We present the design of a set of new non-isomorphic interaction techniques for controlling an avatar facial expression in VR using a standard VR controller. These techniques, as the one illustrated in Figure 5 have been evaluated through two controlled experiments to help designing an interaction technique combining the strengths of each approach. This technique was evaluated in a final ecological study showing it can be used in contexts such as social applications.

This image shows a VR controller with a ray pointing to a disc where emojis are displayed, and an avatar's head with the corresponding expression.

8.3 HOBIT+

Participants: Vincent Casamayou, Martin Hachet.

External collaborators: Bruno Bousquet [Univ. Bordeaux], Lionel Canioni [Univ. Bordeaux], Jean-Paul Guillet [Univ. Bordeaux].

We have started extending the concept of HOBIT, a hybrid (physical/virtual) platform dedicated to the learning of wave optics 3736, to a purely virtual version of the system (see Figure 6). The main objective is for remote students to continue doing practical work, as if they were manipulating the physical table. At the end, we envision to connect both the physical table and its virtual counterparts for exploring hybrid (insitu/remote) collaborative training sessions.

A view of the desktop version of the HOBIT software

8.4 Interaction in Mixed Reality and Accessible Board Games

Participants: Rajkumar Darbar, Arnaud Prouzeau, Martin Hachet.

External collaborators: Lauren Thevin [UCO Bretagne], Anke Brock [ENAC], Christophe Jouffrais [CNRS], Bernard Oriola [CNRS].

Two of our projects started in 2020 were published in 2021.

The first one, led by Rajkumar Darbar focuses on text selection with AR Head-mounted displays 25. The main idea is to use a mobile device for interacting with displayed text. Four techniques were proposed and evaluated. See the dedicated section of our 2020 report.

The second one is the result of a collaboration with Lauren Thevin and Anke Brock, former members of Potioc, as well as other collaborators. We have explored how PapARt, a interactive projection technology developed in the team, could make board games accessible to players with visual impairments (See 2020 description). This work was published at ISS 2021 and obtained a best paper award 16.

8.5 Investigating Around Field of View Glancable Command Selection in AR-HMDs

Participants: Rajkumar Darbar, Arnaud Prouzeau, Martin Hachet.

Designing menus for augmented reality head-mounted displays (AR-HMDs) is challenging because of the limited display field of view (FOV) of the current devices. In this work, we are exploring an approach where users can quickly access contextual commands in AR-HMDs by interacting below the display area. Compared to previous work, our technique provides the following advantages: commands are always available in the user’s FOV (i.e., head referenced) with a very minimum occlusion and can be issued anytime with a single tap; there is no need to switch attention explicitly from the ongoing task; it does not require any external device except the AR-HMD having hand tracking available.

A user wears a Hololens. She is selecting a button.

8.6 Hidden Objects in Situated Visualization

Participants: Ambre Assor, Arnaud Prouzeau, Pierre Dragicevic, Martin Hachet.

As part of the PhD thesis of Ambre Assor, we are defining a design space of techniques for handling non-visible physical referents in situated visualizations. Although visual occlusion has been rarely discussed in the literature on situated visualizations, it is a known problem in the areas of 3D user interfaces and 3D visualization, as well as in AR and virtual reality (VR). To construct our design space, we take inspiration from the techniques proposed in these research areas. These techniques served to identify the main dimensions, to which we add specific ones dedicated to situated visualizations. The final design space includes dimensions related to: (1) the visual representation of the physical representation of the physical referent; (2) the interactivity and degree of embodiment of the visual representation of the physical referent; (3) the view paradigm of the technique; (4) the data visualisation associated with the physical referent. With this design space, we define families of potential techniques, and we explore the challenges and opportunities associated with the design of techniques for situated visualizations with non-visible objects.

8.7 Tangible interfaces for Railroad Traffic Monitoring

Participants: Arnaud Prouzeau, Martin Hachet.

External collaborators: Maudeline Marlier [SNCF].

Maudeline Marlier did her end-of-study internship at Potioc and at SNCF working on the use of tangible interaction in railroad traffic monitoring control rooms. More specifically, the aim was to work on the concept of a tangible interface and to develop a proof of concept to propose an idea to operators before testing it with them. It was, first, important to understand the context and activities performed in such control rooms. To this purpose, several observation sessions and interviews were organised in control rooms around Paris. Results from these sessions suggest that communication and information sharing are essential in these rooms, especially during a crisis, in which operators stand up, discuss and watch information on their colleague's screens. In this way, they can analyse what others are doing thanks to non-verbal communication to anticipate their needs: this represents a typical use case for tangible interfaces. Maudeline then developed a first prototype of a tangible interface working on a tabletop (see Figure 8). This work will be continued in a CIFRE PhD. with Maudeline and SNCF.

A user is moving a token on a multi-touch tabletop.

8.8 Superpower-Inspired Framework for Data Visualization

Participants: Pierre Dragicevic, Yvonne Jansen.

External collaborators: Wesley Willett [Univ. Calgary], Bon Adriel Aseniero [Autodesk], Sheelagh Carpendale [Simon Fraser Univ.], Lora Oehlberg [Univ. Calgary], Petra Isenberg [Inria Paris-Saclay, correspondent].

We explored how the lens of fictional superpowers can help characterize how visualizations empower people and provide inspiration for new visualization systems. Researchers and practitioners often tout visualizations' ability to “make the invisible visible” and to “enhance cognitive abilities.” Meanwhile superhero comics and other modern fiction often depict characters with similarly fantastic abilities that allow them to see and interpret the world in ways that transcend traditional human perception. We investigated the intersection of these domains, and showed how the language of superpowers can be used to characterize existing visualization systems and suggest opportunities for new and empowering ones. We introduced two frameworks: The first characterizes seven underlying mechanisms that form the basis for a variety of visual superpowers portrayed in fiction. The second identifies seven ways in which visualization tools and interfaces can instill a sense of empowerment in the people who use them. Building on these observations, we illustrated a diverse set of “visualization superpowers” and highlighted opportunities for the visualization community to create new systems and interactions that empower new experiences with data. Material and illustrations are available under CC-BY 4.0 at osf.io/8yhfz.

8.9 Visualizations for Promoting Empathy and Compassion

Participants: Luiz Morais [Universidade Federal de Campina Grande], Yvonne Jansen [Universidade Federal de Campina Grande], Nazareno Andrade [Universidade Federal de Campina Grande], Pierre Dragicevic [correspondant].

When showing data about people, visualization designers and data journalists often use design strategies that presumably help the audience relate to those people. The term “anthropographics” has been recently coined to refer to this practice and the resulting visualizations. Empirical studies have recently examined whether anthropographics promote empathy, compassion, or the likelihood of prosocial behavior, but findings have been inconclusive so far. In an article published at ACM CHI  47, we contributed a detailed overview of past experiments, and two new experiments that use large samples and a combination of design strategies to maximize the possibility of finding an effect. We tested an information-rich anthropographic against a simple bar chart (Figure 10), asking participants to allocate hypothetical money in a crowdsourcing study. We found that the anthropographic had, at best, a small effect on money allocation. Such a small effect may be relevant for large-scale donation campaigns, but the large sample sizes required to observe an effect and the noise involved in measuring it make it very difficult to study in more depth. We discussed implications for our findings, such as the importance of testing alternative anthropographic design strategies as well as using alternative methodologies, such as real money donation tasks or observations in the field, for example, by testing alternative designs on the website of a real charity. Data and code are available at osf.io/xqae2.

8.10 Review of Data Physicalization

Participants: Pierre Dragicevic [KU Leuven], Yvonne Jansen [KU Leuven], Andrew Vande Moere [KU Leuven].

Data physicalization is a rich and vast research area that studies the use of physical artifacts to convey data. It overlaps with a number of research areas including information visualization, scientific visualization, visual analytics, tangible user interfaces, shape-changing interfaces, personal fabrication interfaces, as well as graphic design, architecture, and art. We published a book chapter 46 in an HCI handbook that surveys academic work on data physicalization and also provides a broad overview of nonacademic work. It discusses how data physicalization has been used for analytical purposes, communication and education, accessibility, self-reflection and self-expression, and finally for enjoyment and meaning. It also discusses enabling technologies, reviews empirical studies, and surveys models and theories of data physicalization.

8.11 Use of Space in Immersive Analytics

Participants: Arnaud Prouzeau.

External collaborators: Maxime Cordeil [University of Queensland], Thibault Friedrich [Ecole de Technologie Superieure in Montreal], Michael McGuffin [Ecole de Technologie Superieure in Montreal], Jiazhou Liu [Monash University], Barrett Ens [Monash University], Tim Dwyer [Monash University], Benjamin Lee [Monash University], Yidan Zhang [Monash University], Sarah Goodwin [Monash University].

An immersive system is one whose technology allows us to “step through the glass” of a computer display to engage in a visceral experience of interaction with digitally-created elements. As immersive technologies have rapidly matured to bring commercially successful virtual and augmented reality (VR/AR) devices and mass market applications, researchers have sought to leverage its benefits, such as enhanced sensory perception and embodied interaction, to aid human data understanding and sensemaking 26, 18. In 2021, we have mainly explored the use of the 3D workspace in immersive analytics systems.

In immersive environments, positioning data around the user to utilise the abundant display space has been advocated as an advantage over traditional flat screens. However, other than limiting the distance users must walk, there is no clear design rationale behind this common practice, and little research on the impact of wraparound layouts on visualisation tasks. One aspect of user performance in such tasks is the ability to remember the spatial locations of visualisations as users shift their focus between them. Motivated by these use-cases, we study the effects of flat-wall layout versus circular-wraparound layout on a visuo-spatial memory task in a virtual environment. The results demonstrate that participants are able to recall spatial patterns with greater accuracy using flat layouts than with wraparound layouts. Our study participants also preferred the flat over the wraparound layout, and subjectively reported less mental and physical effort with a flat layout. A second study highlighted the fact that this better performance was not counterbalanced with the use of landmarks, or with when we remove the possibility for the participants to get an overview of the workspace. This work have been performed in collaboration with Monash University and will be submitted at the conference ACM ISS 2022.

Another method to improve spatial memory is to increase the size of the workspace to encourage users to physically move inside. In collaboration with ETS Montreal, we performed a study testing this hypothesis and showed only a very small benefits of a large workspace over a small one. This has been presented at GI 2021 27.

There are six small multiples demonstrating the TimeTables prototype. In Virtual Reality, we can see a table with a map of a campus. 3D representations of the buildings are placing on top of each other to represent different moment in time and their color encode the energy consumption.

In collaboration with Monash University, we explore the potential of using VR to explore spatio-temporal dataset. We developed a prototype, called TimeTables, that focuses on the visualisation and manipulation of space-time cubes. TimeTables uses multiple space-time cubes on virtual tabletops, which users can manipulate by extracting time layers or individual buildings to create new tabletop views (see Figure 11). The surrounding environment includes a large space for multiple linked tabletops and a storage wall. TimeTables presents information at different time scales by stretching layers to drill down in time. Users can also jump into tabletops to inspect data from an egocentric perspective. We present a use case scenario of energy consumption displayed on a university campus to demonstrate how our system could support data exploration and analysis over space and time. This work has been accepted at IEEE VR 2022.

Finally, working in a 3D space does not mean you should only use 3D visualisations. In collaboration with Monash University, we explored the design space of transformation between 2D and 3D visualisations. To explore this, we first establish an overview of the different states that a data visualisation can take in AR/VR, followed by how transformations between these states can facilitate common visualisation tasks. We then describe a design space of how these transformations function, in terms of the different stages throughout the transformation, and the user interactions and input parameters that affect it. This design space is then demonstrated with multiple exemplary techniques based in AR/VR. This work has been accepted at ACM CHI 2022.

A workshop on spatial sensemaking in immersive analytics has been accepted at ACM CHI 2022 and Arnaud is part of the organising committee.

8.12 Virtual Reality to Amplify Episodic Future Thinking

Participants: Adelaide Genay, Arnaud Prouzeau.

External collaborators: Verity Petratos [Monash University], Simon Van Baal [Monash University], Alex Robinson [Monash University], Antonio Verdejo-Garcia [Monash University].

Neuroscience research discovered that when we imagine the future with the same level of detail and affective quality as when we remember the past, we tend to make more future-oriented decisions (e.g. staying home to protect others). Episodic future thinking (EFT) is a prospection-based training that stimulates this ability and has shown to improve future-oriented choices while simultaneously reducing anxiety about the future. Our team is currently working on two ways to improve the effectiveness and scalability of EFT: (1) building an immersive virtual reality (VR) scenario to facilitate imagining the future with sufficient detail and infusing it with positive emotions; (2) exploring the potential of collective EFT (imagining the future together) via VR communities. In 2021, a first study was performed at Monash University to assess the impact of using a VR training (see Figure 12) during an EFT training and showed encouraging regarding delay discounting, an important mechanism of impulsivity. A second wider study focused on a clinical population (food addiction) is planned in 2022.

View of a 3D virtual scene of a forest track with picture panels.

8.13 VR Authoring Systems

Participants: Edwige Gros, Arnaud Prouzeau, Martin Hachet.

Edwige Gros performed a 6-months internship at Potioc during which she explored the design of an immersive tool to author interactive experience in virtual reality using demonstration and direct manipulation. Authoring by demonstration and direct manipulation is an intuitive way of creation as users directly visualize the outcome of their actions and create the interaction with a first-person point of view inside the context. Yet authoring by demonstration raises several challenges. In the first place, as a user demonstrates the interaction, the system has several ambiguities to solve. It needs to identify the expected action, the referent to the action as well as its spatial and temporal context and properties. Secondly, some concepts required for interaction authoring are either intangible, abstract or both.

We proposed a framework to define an interaction thanks to demonstration (see Figure 13) and developed an immersive authoring prototype in VR. Users can record their actions directly or manipulate a puppet in order to specify the final user’s action. Users can also record objects modification, interaction and movements. Testing is also enabled during the creation process to allow an iterative authoring method and facilitate error processing. Edwige is now following-up on this topic in a PhD and now explore the collaborative aspect of this design experience.

Table representing the framework for authoring interaction by demonstration.

8.14 Accessibility of E-Learning Systems

Participants: Pascal Guitton.

External collaborators: Hélène Sauzéon [EP Flowers].

In 2021, we concluded our work on new digital teaching systems such as MOOCs. Unfortunately, accessibility for people with disabilities is often forgotten, which excludes them, particularly those with cognitive impairments for whom accessibility standards are far from being established. We have shown in 44 that very few research activities deal with this issue.

In past years, we have proposed new design principles based on knowledge in the areas of accessibility (Ability-based Design and Universal Design), digital pedagogy (Instruction Design with functionalities that reduce the cognitive load : navigation by concept, slowing of the flow…), specialized pedagogy (Universal Design for Learning, e.g., automatic note-taking, and Self Determination Theory, e.g., configuration of the interface according to users needs and preferences) and psychoeducational interventions (e.g., support the joint teacher-learner attention), but also through a participatory design approach involving students with disabilities and experts in the field of disability (Figure 14). From these frameworks, we have designed interaction features which have been implemented in a specific MOOC player called Aïana. Moreover, we have produced a MOOC on digital accessibility which is published on the national MOOC platform (FUN) using Aïana (4 sessions since 2016 with more than 13 000 registered participants). Our first field studies demonstrated the benefits of using Aïana for disabled participants 45. We published one of the first systematic review on accessibility of e-learning system 43. Finally in 2021, early interaction (with accessibility features), participation, learning and learner experience were assessed thanks to a field study (N=546) without equivalent in the literature, including learners with and without disabilities 13.

Image showing an interface design session with a real user (left side) and a slide displayed using Aïana (right side)

8.15 Tools for affective, cognitive and conative states estimation from both EEG and physiological signals

Participants: Aurélien Appriou, David Trocellier, Fabien Lotte.

External collaborators: Pierre-Yves Oudeyer [Inria Bordeaux Sud-Ouest, Flowers], Edith Law [Univ. Waterloo], Jessie Ceha [Univ. Waterloo], Dan Dutartre [Inria Bordeaux Sud-Ouest, SED], Andrzej Cichocki [SKOLTECH].

Passive BCIs could estimate cognitive (process of coming to know and understand), conative (personal, intentional and motivational drives to process the information) and affective states (emotional interpretation of perceptions, information, or knowledge). Being able to estimate such mental states from EEG and/or physiological signals, using passive BCIs, could improve, for instance the understanding of individual users’ capabilities and motivations to learn. However, being able to do so is no easy task. Indeed, this requires psychological models and protocols to induce and manipulate such mental states, as well as machine learning algorithms and software tools to robustly estimate them from EEG signals. Our recent research works with passive BCIs addressed these issues. In particular we proposed 1) machine learning algorithms we designed and/or studied to classify cognitive and affective states from EEG; 2) BioPyC, an open-source python toolbox for easy classification of EEG and physiological signals 11; and 3) a protocol and study demonstrating that a conative state can be newly estimated from EEG and physiological signals: curiosity. This work was presented at the third Neuroergonomics conference and received the "Young Investigator Award". 

8.16 Riemannian channel selection for BCI with between-session non-stationarity reduction capabilities

Participants: Khadijeh Sadatnejad, Fabien Lotte.

Between-session non-stationarity is a major challenge of current BCIs that affects system performance. We investigated the use of channel selection for reducing between-session non-stationarity with Riemannian BCI classifiers. We use the Riemannian geometry framework of covariance matrices due to its robustness and promising performances. Current Riemannian channel selection methods do not consider between-session non-stationarity and are usually tested on a single session. We proposed a new channel selection approach that specifically considers non-stationarity effects and is assessed on multi-session BCI data sets. To do so, we removed the least significant channels using a sequential floating backward selection search strategy. Our contributions include: 1) quantifying the non-stationarity effects on brain activity in multi-class problems by different criteria in a Riemannian framework and 2) a method to predict whether BCI performance can improve using channel selection. We evaluated the proposed approaches on three multisession and multi-class mental tasks (MT)-based BCI datasets. They could lead to significant improvements in performance as compared to using all channels for datasets affected by between-session non-stationarity and to significant superiority to the state-of-the-art Riemannian channel selection methods over all datasets, notably when selecting small channel set sizes. This work was very recently accepted in the journal IEEE transactions on Neural System and Rehabilitation Engineering.

8.17 Towards identifying optimal biased feedback for various user states and traits in motor imagery BCI

Participants: Smeety Pramij, Fabien Lotte.

External collaborators: Jelena Mladenovic [Univ. Belgrade], Jérémy Frey [Ullo], Jérémie Mattout [Inserm Lyon].

Neural self-regulation is necessary for achieving control over BCIs. This can be an arduous learning process especially for motor imagery BCI. Various training methods were proposed to assist users in accomplishing BCI control and increase performance. Notably the use of biased feedback, i.e. non-realistic representation of performance. Benefits of biased feedback on performance and learning vary between users (e.g. depending on their initial level of BCI control) and remain speculative. To disentangle the speculations, we investigated what personality type, initial state and calibration performance (CP) could benefit from a biased feedback. We conducted an experiment (n=30 for 2 sessions) in which the feedback provided to each group (n=10) was either positively, negatively or not biased. Statistical analyses suggested that interactions between bias and: 1) workload, 2) anxiety, and 3) self-control significantly affected online performance. For instance, low initial workload paired with negative bias was associated to higher peak performances (86%) than without any bias (69%). High anxiety related negatively to performance no matter the bias (60%), while low anxiety matched best with negative bias (76%). For low CP, learning rate (LR) increased with negative bias only short term (LR=2%) as during the second session it severely dropped (LR=-1%). Overall, we unveiled many interactions between said human factors and bias. Additionally, we used prediction models to confirm and reveal even more interactions. This work was published in the journal IEEE Transactions on Biomedical Engineering 17.

8.18 Multi-Session Influence of Two Modalities of feedback and Their Order of Presentation on MI-BCI User Training

Participants: Fabien Lotte.

External collaborators: Léa Pillette [INCIA], Bernard N'Kaoua [Univ. Bordeaux], Romain Sabau [Univ. Bordeaux].

By performing motor-imagery tasks, for example, imagining hand movements, Motor-Imagery based BCIs (MI-BCIs) users can control digital technologies, for example, neuroprosthesis, using their brain activity only. MI-BCI users need to train, usually using a unimodal visual feedback, to produce brain activity patterns that are recognizable by the system. The literature indicates that multimodal vibrotactile and visual feedback is more effective than unimodal visual feedback, at least for short term training. However, the multi-session influence of such multimodal feedback on MI-BCI user training remained unknown, so did the influence of the order of presentation of the feedback modalities. In our experiment, 16 participants trained to control a MI-BCI during five sessions with a realistic visual feedback and five others with both a realistic visual feedback and a vibrotactile one. training benefited from a multimodal feedback, in terms of performances and self-reported mindfulness. There was also a significant influence of the order presentation of the modality. Participants who started training with a visual feedback had higher performances than those who started training with a multimodal feedback. We recommend taking into account the order of presentation for future experiments assessing the influence of several modalities of feedback. This work was published in the journal MDPI Multimodal Technologies and Interaction 19.

A picture showing our vibrotactile gloves worn by a BCI participant who can also see 3D hands a screen on top of her real hands.

8.19 Modeling Mental Imagery-based BCI user performances

Participants: Camille Benaroch, Aline Roc, Fabien Lotte.

External collaborators: Camille Jeunet [INCIA/CNRS/Univ. Bordeaux], Maria Sayu Yamamoto [Univ. Paris-Saclay].

Models based on Neurophysiological Factors: 

In that work, we tried to understand the mechanisms underlying MT-BCI user training, notably by identifying neurophysiological factors related/associated to it. Such factors of interest should generalize across datasets. However, so far, most neurophysiological factors that have been identified, individually, as factors related to BCI performance but not as predictors of performance of unseen users. Furthermore, they were studied in single experiments and/or data sets. Therefore, in our work:

• We computed and compared neurophysiological predictors of MI-BCI performance found in the literature using two datasets.
• We proposed new neurophysiological predictors of MI-BCI performance.
• We studied whether statistical models can predict and explain MT-BCI user performance across experiments, based on users' neurophysiological characteristics.

We used data from 85 subjects, for a total of 340 online MI-BCI runs collected from two different studies based on the same MI-BCI paradigm. We used machine learning regressions with a leave-one-subject-out cross validation on one dataset ( to build our predictive model of performance and, to study the reliability of the model, we used it to predict the performance of an other dataset. Our results suggested that it was possible to predict MI-BCI performances of unseen users using neurophysiological characteristics of a user with an error of 9.3% (p = 0.005). In addition, the proposed new predictors were found to be relevant predictors of MI-BCI performances in our analyses. 

Models based on characteristics of the machine learning algorithms used: 

In this work, we proposed to study and extract predictors from two frequently used data-driven algorithms during the MI-BCI system calibration (i.e., the common spatial pattern (CSP) algorithm and the Linear Discriminant analyses (LDA)). In particular:

• We extracted characteristics from the trained CSP and LDA weights.
• We studied the relationships between MI-BCI performances and subject-specific characteristics extracted from the CSP algorithm and LDA.
• We studied whether statistical models can predict and explain MI-BCI user performance across experiments, based on subject-specific characteristics extracted from the machine (i.e., from the results of the calibration algorithms).

Our results suggested that a large majority of our proposed characteristics were strongly correlated to MI-BCI performances. 

When should MI-BCI feature optimization include prior knowledge, and which one?: 

MI-BCIs rely on interactions between humans and machines. Therefore, the (learning) characteristics of both components are key to understand and improve performances. Data-driven methods are often used to select/extract features with very little neurophysiological prior. Should such approach include prior knowledge and, if so, which one? Can such approach reach optimal performances? Our work studied the relationship between BCI performances and characteristics of the subject-specific Most Discriminant Frequency Band (MDFB) selected by a popular heuristic algorithm. First, our results showed a correlation between the selected MDFB characteristics (mean and width) and performances 35. Then, to investigate a possible causality link, we compared, online, performances obtained with a constrained (enforcing characteristics associated to high performances) and an unconstrained algorithm. Although we could not conclude on causality, average performances using the constrained algorithm were the highest. Finally, to understand the relationship between MDFB characteristics and performances better, we used machine learning to 1) predict MI-BCI performances using MDFB characteristics and 2) select automatically the optimal algorithm (constrained or unconstrained) for each subject. Our results revealed that the constrained algorithm could improve performances for subjects with either clearly distinct or no distinct EEG patterns. This work was submitted to the Brain-Computer Interfaces journal, and was accepted conditionally with minor revisions. 

8.20 Long-term BCI training of a Tetraplegic User: Adaptive Riemannian Classifiers and User Training

Participants: Camille Benaroch, Aline Roc, Smeety Pramij, Khadijeh Sadatnejad, Thibaut Monseigne, Fabien Lotte.

External collaborators: Aurélien Appriou [Flitsport], Camille Jeunet [INCIA/CNRS/Univ. Bordeaux], Léa Pillette [Centrale Nantes], Jelena Mladenovic [Univ. Belgrade].

While often presented as promising assistive technologies for motor-impaired users, EEG-based BCIs remain barely used outside laboratories due to low reliability in real-life conditions. There is thus a need to design long-term reliable BCIs that can be used outside-of-the-lab by end-users, e.g., severely motor-impaired ones. Therefore, we proposed and evaluated the design of a multi-class Mental Task (MT)-based BCI for longitudinal training (20 sessions over 3 months) of a tetraplegic user for the CYBATHLON BCI series 2019. In this BCI championship, tetraplegic pilots are mentally driving a virtual car in a racing video game. We aimed at combining a progressive user MT-BCI training with a newly designed machine learning pipeline based on adaptive Riemannian classifiers shown to be promising for real-life applications. We followed a two step training process: the first 11 sessions served to train the user to control a 2-class MT-BCI by performing either two cognitive tasks (REST and MENTAL SUBTRACTION) or two motor-imagery tasks (LEFT-HAND and RIGHT-HAND). The second raining step (9 remaining sessions) applied an adaptive, session-independent Riemannian classifier that combined all 4 MT classes used before. Moreover, as our Riemannian classifier was incrementally updated in an unsupervised way it would capture both within and between-session non-stationarity. Experimental evidences confirmed the effectiveness of this approach. Namely, the classification accuracy improved by about 30% at the end of the training compared to initial sessions. We also studied the neural correlates of this performance improvement. Using a newly proposed BCI user learning metric, we could show our user learned to improve his BCI control by producing EEG signals matching increasingly more the BCI classifier training data distribution, rather than by improving his EEG class discrimination. Note that, when re-analyzing past data sets, we could also observe this new type of learning in some subjects 40. However, the resulting improvement was effective only on synchronous (cue-based) BCI and it did not translate into improved CYBATHLON BCI game performances. This work was published in the journal Frontiers in Human Neuroscience: Brain-Computer Interfaces 12.

8.21 Training programs for for Mental-Task based BCI user training:

Participants: Aline Roc, Smeety Pramij, Thibaut Monseigne, Pauline Dreyer, Fabien Lotte.

A Framework for Mental-Task based BCI user training: 

Based on the review and study of related works, we are working towards identifying the relevant parameters allowing for a more comprehensive description of user training, such as interaction properties (what to perform, when, how, targeted skill, etc.), environment (social and physical), information communicated through instructions or through feedback/interface design (modality/dimensionality, etc.). Such framework can help studying and describing training in a more structured way, perhaps providing a template to support a better design, evaluation, comparison or reporting of experimental research. This framework was presented at the (virtual) international BCI meeting 39. 

Studying User states during MT-BCI training 

We are investigating user states through subjective feedback from the user and passive BCI classification. We recently assembled a dataset with 24 participants training over three Mental Tasks (MT)-BCI sessions (with CSP+LDA classification) to control a standard MT-BCI (left vs. right-hand imagery) during 12 short runs (16 trials/run). Before each session, they all completed two different cognitive tasks (N-back & Rspan) both involving a high and low cognitive load while recording EEG activity. Participants also provided subjective measures of their cognitive load during BCI interaction after each 16-trials run (with 10-points Likert scales over 6 items).

In ongoing work, we attempt to build a classifier based on these Working-Memory Tasks and use it to assess and study users' cognitive load level during MI-BCI training.

We also showed different trends in users' self-assessments, especially higher reported load in the first session compared to 2nd and next sessions. It also seem that users tend to gradually improve their capability to estimate their own performance based on the feedback they receive.

We also conducted semi-structured interviews intended for the purpose of better documenting (1) what people choose to imagine and (2) how they feel about the execution of these tasks. 

Involving the users in their training 

In current BCI user training paradigms, the interaction is generally both synchronous and guided, which means that the user is provided with both temporal cues and content cues. For example, in the Graz paradigm, users are provided with a specific time period of a few seconds during which they are instructed to perform a specific task, then another time period in which they have to perform another task, and so on. However, synchronous and guided training is repetitive and ultimately boring for the user. Additionally, the literature in human learning suggests that learners should train with individualized learning objectives and proceed at their own pace 38. Thus, we designed an experimental protocol with 2 experimental groups who will train with feedback to control a MT-BCI. The control group will undergo a standard closed training, with fully synchronous and guided copy tasks, while the other group will undergo a modified version of the protocol where a percentage of the training will be user-initiated instead (with no cues indicating which task to perform or when). Such runs will allow for internally-paced and internally-guided decisions during user-device interactions. For both groups and for future experiments, an experimental platform for user training was designed and implemented (it is named Brain Hero) to allow for easy modification of many trial parameters (e.g. presence, speed or aspect of the cues) and to improve the collection of online metrics. Additionally, as compared to the standard one, the user interface is inspired by rhythm games and is expected to provide improved user experience through gamification.

8.22 Public BCI data sharing:

Participants: Pauline Dreyer, Aline Roc, Camille Benaroch, Thibaut Monseigne, Sébastien Rimbert, Fabien Lotte.

The numerous BCI experiments carried out within the BrainConquest project allow the creation of a large database, that has been analyzed and verified in order to be shared permanently and free of charge with the scientific community. In particular, we will share: the protocols used for BCI experiments, the EEG and physiological signals of 87 BCI participants; information about their profile: demographics, current state (e.g., their state of fatigue or motivation, measured through questionnaires), their personality traits and their BCI performance; the OpenViBE codes and data (e.g., the parameters of the machine learning algorithms) generated during each experiment. This will allow the community to extend this research, for example by using our extensive EEG data to design new machine learning algorithms or to identify new factors influencing BCI learning.

8.23 Understanding the fundamental interrelation between ERD/ERS and MI-BCI use

Participants: Sébastien Rimbert, Fabien Lotte.

One of the most prominent BCI types of interaction is Motor Imagery (MI)-based BCI. Users control a system by performing MI tasks, e.g., imagining hand or foot movements detected from EEG signals. Indeed, movement and imagination of movement activate similar neural networks, enabling the MI-based BCI to exploit the modulation of sensorimotor rhythms (SMR) over the motor cortex, respectively known as Event-Related Desynchronization (ERD) and Event-Related Synchronization (ERS), coming from the mu (7-13 Hz) and beta (15-30 Hz) frequency bands. Despite current research, the relationship between ERD/ERS generated during the MI task is still not well understood. Indeed, numerous studies have previously shown that there is a lot of inter-subject and intra-subject variability concerning the ERDs and ERSs patterns, but difficulties remain to understand the origin of such phenomenon. This lack of knowledge about temporal variability of cerebral motor patterns limits the possibilities of improving the performance of BCI interfaces, which remain quite poor on average. All these unknowns would be problematic for such targeted applications. A variation in the nature or quality of the brain inputs can affect the overall quality of the BCI, and therefore the corresponding user experience, and vice versa, creating a potential mutual degradation of the BCI-loop.

We believe that a better understanding of the intra-individual variability of ERD/ERS during BCI use is crucial for developing effective interfaces. Indeed, individual activation patterns during MI are largely neglected in most studies, which have mainly focused on identifying patterns common to all participants. In particular, how the cerebral motor patterns underlying the MI task (i.e., ERDs and ERSs) are modulated depending (i) on the context of the BCI experiment, (ii) on the characteristics of the user (e.g., right-handed/left-handed, age, gender, athlete etc.) or on the user's experience with the BCI overtime needs to be better understood. Particularly, how the user is changing, in terms of mastery of the task, acceptability of the interface, subjective perception of motivation, with continued use of the interface has not been sufficiently investigated. Thus, we propose to analyze large MI-BCI databases (> 150 subjects) to show how ERD/ERS are modulated according to the experimental context, the characteristics of the user or their specific experience with the BCI. We will see that this enhanced knowledge will enable the design of more efficient MI-BCIs adapted to each user.

8.24 Guidelines to use Transfer Learning for Motor Imagery Detection: an experimental study

Participants: Sébastien Rimbert, Fabien Lotte.

External collaborators: Laurent Bougrain [Univ. Lorraine], Pedro Luiz Coelho Rodrigues [Inria Paris-Saclay, Parietal], Geoffrey Canron [Univ. Lorraine].

MI-BCI have shown promising results for motor recovery, intraoperative awareness detection or assistive technology control. However, they suffer from several limitations due to the high variability of EEG signals, mainly lengthy and tedious calibration times usually required for each new day of use, and a lack of reliability for all users. Such problems can be addressed, to some extent, using transfer learning algorithms. However, the performance of such algorithms has been very variable so far, and when they can be safely used is still unclear. Therefore, in this work, we studied the performance of various state-of-the-art Riemannian transfer learning algorithms on a MI-BCI database (30 users), for various conditions: 1) supervised and unsupervised transfer learning; 2) for various amount of available training EEG data for the target domain; 3) intra-session or inter-session transfer; 4) for both users with good and less good MI-BCI performances. From such experiments, we derived guidelines about when to use which algorithm. Re-centering the target data is effective as soon as a few samples of this target set are taken into account. This is true even for an intra-session transfer learning. Likewise, re-centering is particularly useful for subjects who have difficulty producing stable motor imagery from session to session. This work was published in the IEEE NER'21 conference 24.

9.1 Bilateral contracts with industry

Wisear:

• Duration: 2020-2021
• Local coordinator: Fabien Lotte
• Consulting project on in-ear EEG-based BCI

10.1 International initiatives

10.2 European initiatives

10.3 National initiatives

eTAC: Tangible and Augmented Interfaces for Collaborative Learning:

• Funding: EFRAN
• Duration: 2017-2021
• Coordinator: Université de Lorraine
• Local coordinator: Martin Hachet
• Partners: Université de Lorraine, Inria, ESPE, Canopé, OpenEdge,
• the e-TAC project proposes to investigate the potential of technologies ”beyond the mouse” in order to promote collaborative learning in a school context. In particular, we will explore augmented reality and tangible interfaces, which supports active learning and favors social interaction.
• website: eTac

ANR Project EMBER:

• Duration: 2020-2023
• Partners: Inria/AVIZ, Sorbonne Université
• Coordinator: Pierre Dragicevic
• Local coordinator: Martin Hachet
• The goal of the project will be to study how embedding data into the physical world can help people to get insights into their own data. While the vast majority of data analysis and visualization takes place on desktop computers located far from the objects or locations the data refers to, in situated and embedded data visualizations, the data is directly visualized near the physical space, object, or person it refers to.
• website: Ember

Inria Project Lab AVATAR:

• Duration: 2018-2022
• Partners: Inria project-teams: GraphDeco, Hybrid, Loki, MimeTIC, Morpheo
• Coordinator: Ludovic Hoyet (Inria Rennes)
• Local coordinator: Martin Hachet
• This project aims at designing avatars (i.e., the user’s representation in virtual environments) that are better embodied, more interactive and more social, through improving all the pipeline related to avatars, from acquisition and simulation, to designing novel interaction paradigms and multi-sensory feedback.
• website: Avatar

11.1 Promoting scientific activities

11.2 Teaching - Supervision - Juries

11.3 Popularization

12.1 Major publications

• 1 articleC.Camille Benaroch, K.Khadijeh Sadatnejad, A.Aline Roc, A.Aurélien Appriou, T.Thibaut Monseigne, S.Smeety Pramij, J.Jelena Mladenović, L.Léa Pillette, C.Camille Jeunet and F.Fabien Lotte. Long-Term BCI Training of a Tetraplegic User: Adaptive Riemannian Classifiers and User Training.Frontiers in Human Neuroscience15March 2021
  HALDOI
• 2 inproceedingsJ.Jérémy Frey, M.Maxime Daniel, J.Julien Castet, M.Martin Hachet and F.Fabien Lotte. Framework for Electroencephalography-based Evaluation of User Experience.CHI '16 - SIGCHI Conference on Human Factors in Computing SystemSan Jose, United StatesMay 2016
  HALDOI
• 3 inproceedingsJ.Jérémy Frey, R.Renaud Gervais, S.Stéphanie Fleck, F.Fabien Lotte and M.Martin Hachet. Teegi: Tangible EEG Interface.UIST-ACM User Interface Software and Technology SymposiumHonolulu, United StatesACMOctober 2014
  HAL
• 4 inproceedingsD.David Furio, S.Stéphanie Fleck, B.Bruno Bousquet, J.-P.Jean-Paul Guillet, L.Lionel Canioni and M.Martin Hachet. HOBIT: Hybrid Optical Bench for Innovative Teaching.CHI'17 - Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsDenver, United StatesMay 2017
  HALDOI
• 5 articleA.Adélaïde Genay, A.Anatole Lécuyer and M.Martin Hachet. Being an Avatar "for Real": a Survey on Virtual Embodiment in Augmented Reality.IEEE Transactions on Visualization and Computer Graphics2021, 1 - 20
  HALDOI
• 6 inproceedingsJ.Jacek Jankowski and M.Martin Hachet. A Survey of Interaction Techniques for Interactive 3D Environments.Eurographics 2013 - STARGirona, SpainMay 2013
  HAL
• 7 articleL.Léa Pillette, C.Camille Jeunet, B.Boris Mansencal, R.R N'Kambou, B.Bernard N'Kaoua and F.Fabien Lotte. A physical learning companion for Mental-Imagery BCI User Training.International Journal of Human-Computer Studies136102380April 2020
  HALDOI
• 8 articleA.Aline Roc, L.Léa Pillette, J.Jelena Mladenović, C.Camille Benaroch, B.Bernard N'Kaoua, C.Camille Jeunet and F.Fabien Lotte. A review of user training methods in brain computer interfaces based on mental tasks.Journal of Neural Engineering2020
  HALDOI
• 9 inproceedingsJ.Joan Sol Roo, R.Renaud Gervais, J.Jérémy Frey and M.Martin Hachet. Inner Garden: Connecting Inner States to a Mixed Reality Sandbox for Mindfulness.CHI '17 - International Conference of Human-Computer InteractionCHI '17ACMDenver, United StatesACMMay 2017
  HALDOI
• 10 inproceedingsJ.Joan Sol Roo and M.Martin Hachet. One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities.UIST 2017 - 30th ACM Symposium on User Interface Software and TechnologyACMQuebec City, CanadaOctober 2017
  HALDOI

12.2 Publications of the year

12.3 Other

12.4 Cited publications

• 43 articleP.-A.Pierre-Antoine Cinquin, P.Pascal Guitton and H.Hélène Sauzéon. Designing accessible MOOCs to expand educational opportunities for persons with cognitive impairments.Behaviour and Information Technologyhttps://www.tandfonline.com/doi/full/10.1080/0144929X.2020.1742381March 2020
  HALDOIback to text
• 44 articleP.-A.Pierre-Antoine Cinquin, P.Pascal Guitton and H.Hélène Sauzéon. Online e-learning and cognitive disabilities: A systematic review.Computers and Education130March 2019, 152-167
  HALDOIback to text
• 45 inproceedingsP.-A.Pierre-Antoine Cinquin, P.Pascal Guitton and H.Hélène Sauzéon. Towards Truly Accessible MOOCs for Persons with Cognitive Disabilities: Design and Field Assessment. ICCHP 2018 - 16th International Conference on Computers Helping People with Special NeedsLinz, AustriaJuly 2018
  HALback to text
• 46 incollectionP.Pierre Dragicevic, Y.Yvonne Jansen and A.Andrew Vande Moere. Data Physicalization.Springer Handbook of Human Computer InteractionSpringer ReferenceSpringer, Cham2021
  HALDOIback to text
• 47 inproceedingsL.Luiz Morais, Y.Yvonne Jansen, N.Nazareno Andrade and P.Pierre Dragicevic. Can Anthropographics Promote Prosociality? A Review and Large-Sample Study.CHI 2021 - Conference on Human Factors in Computing SystemsYokohama / Virtual, JapanMay 2021
  HALDOIback to text