7.1.1 grimgep

• Name:
  GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Keywords:
  Machine learning, Reinforcement learning, Artificial intelligence, Exploration, Intrinsic motivations, Git, Deep learning
• Functional Description:
  Source code for the GRIMGEP paper (https://arxiv.org/abs/2008.04388) Contains: - Implementation of the GRIMGEP framework on top of three different underlying imgeps (Skew-fit, CountBased, OnlineRIG). - image-based 2D environment (PlaygroundRGB)
• URL:
  https://­gitlab.­com/­Grg/­grimgep
• Contact:
  Grgur Kovac

7.1.2 SocialAI

• Name:
  SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Keywords:
  Artificial intelligence, Deep learning, Reinforcement learning
• Functional Description:

  Source code for the paper https://arxiv.org/abs/2107.00956.

  A suite of environments for testing socio-cognitive abilities of RL agents. Simple RL baselines.

• URL:
  https://­gitlab.­inria.­fr/­gkovac/­act-and-speak
• Contact:
  Grgur Kovac

7.1.3 AutoDisc

• Keyword:
  Complex Systems
• Functional Description:
  AutoDisc is a software built for automated scientific discoveries in complex systems (e.g. self-organizing systems). It can be used as a tool to experiment automated discovery of various systems using exploration algorithms (e.g. curiosity-driven). Our software is fully Open Source and allows user to add their own systems, exploration algorithms or visualization methods.
• URL:
  https://­gitlab.­inria.­fr/­cromac/­AutomatedDiscoveryTool
• Contact:
  Clément Romac

7.1.4 Kids Ask

• Keywords:
  Human Computer Interaction, Cognitive sciences
• Functional Description:
  Kids Ask is a web-based educational platform that involves an interaction between a child and a conversational agent. The platform is designed to teach children how to generate curiosity-based questions and use them in their learning in order to gain new knowledge in an autonomous way.
• News of the Year:
  The kids Ask platform was used during two experiments with two different French primary schools, with a total of 53 participants that used the different functions of it.
• URL:
  https://­github.­com/­RaniaAbdelghani/­KidAsk
• Contact:
  Rania Abdelghani

7.1.5 ToGather

• Keywords:
  Education, Handicap, Environment perception
• Scientific Description:
  With participatory design methods, we have designed an interactive website application for educational purposes. This application aims to provide interactive services with continuously updated content for the stakeholders of school inclusion of children with specific educational needs.
• Functional Description:
  Website gathering information on middle school students with neurodevelopmental disorders. Authentication is required to access the site's content. Each user can only access the student file(s) of the young person(s) they are accompanying. A student file contains 6 tabs, in which each type of user can add, edit or delete information: 1. Profile: to quickly get to know the student 2. Skills: evaluation at a given moment and evolution over time 3. Compendium of tips: includes psycho-educational tips 4. Meetings: manager and reports 5. News: share information over time 6. Contacts: contact information for stakeholders The student only has the right to view information about him/her.
• Publication:
  https://hal.inria.fr/hal-03436355/
• Contact:
  Cécile Mazon
• Participants:
  Isabeau Saint-Supery, Cécile Mazon, Eric Meyer, Hélène Sauzéon

7.1.6 mc_training

• Name:
  Platform for metacognitive training
• Keywords:
  Human Computer Interaction, Education
• Functional Description:

  This is a web platform for children between 9 and 11 years old, designed to help children practice 4 metacognitive skills that are thought to be involved in curiosity-driven learning: - the ability to identify uncertainties - the ability to generate informed hypotheses - the ability to ask questions - the ability to evaluate the value of a preconceived inference.

  Children work on a reading-comprehension tasks and, for each of these skills, the platform offers help through a "conversation" with conversational agents that give instructions to perform the task, with respect to every skill, and can give suggestions if the child asks for it.

• Contact:
  Rania Abdelghani

7.1.7 Evolution of adaptation mechanisms in complex environments

• Name:
  Plasticity and evolvability under environmental variability: the joint role of fitness-based selection and niche-limited competition
• Keywords:
  Evolution, Ecology, Dynamic adaptation
• Functional Description:

  This is the code accompannying our paper Plasticity and evolvability under environmental variability: the joint role of fitness-based selection and niche-limited competition" which is to be presented at the Gecco 2022 conference.

  In this work we have studied the evolution of a population of agents in a world where the fitness landscape changes with generations based on climate function and a latitudinal model that divides the world in different niches. We have implemented different selection mechanisms (fitness-based selection and niche-limited competition).

  The world is divided into niches that correspond to different latitudes and whose state evolves based on a common climate function.

  We model the plasticity of an individual using tolerance curves originally developed in ecology. Plasticity curves have the form of a Gaussian the capture the benefits and costs of plasticity when comparing a specialist (left) with a generalist (right) agent.

  The repo contains the following main elements :

  folder source contains the main functionality for running a simulation scripts/run/reproduce_gecco.py can be used to rerun all simulations in the paper scripts/evaluate contains scripts for reproducing figures. reproduce_figures.py will produce all figures (provided you have already run scripts/run/reproduce_gecco.py to generate the data) folder projects contains data generated from running a simulation How to run To install all package dependencies you can create a conda environment as:

  conda env create -f environment.yml

  All script executions need to be run from folder source. Once there, you can use simulate.py, the main interface of the codebase to run a simulation, For example:

  python simulate.py –project test_stable –env_type stable –num_gens 300 –capacity 1000 –num_niches 10 –trials 10 –selection_type NF –climate_mean_init 2

  will run a simulation with an environment with a climate function whose state is constantly 2 consisting of 100 niches for 300 generations and 10 independent trials. The maximum population size will be 1000*2 and selection will be fitness-based (higher fitness means higher chances of reproduction) and niche limited (individuals reproduce independently in each niche and compete only within a niche),

  You can also take a look at scripts/run/reproduce_gecco.py to see which flags were used for the simulations presented in the paper.

  Running all simulations requires some days. You can instead download the data produced by running scripts/run/reproduce_gecco.py from this google folder and unzip them under the projects directory.

• URL:
  https://­github.­com/­eleninisioti/­ClimateAndLearning
• Contact:
  Eleni Nisioti

7.1.8 SAPIENS

• Name:
  SAPIENS: Structuring multi-Agent toPology for Innovation through ExperieNce Sharing
• Keywords:
  Reinforcement learning, Multi-agent
• Functional Description:

  SAPIENS is a reinforcement learning algorithm where multiple off-policy agents solve the same task in parallel and exchange experiences on the go. The group is characterized by its topology, a graph that determines who communicates with whom.

  All agents are DQNs and exchange experiences have the form of transitions from their replay buffers.

  Using SAPIENS we can define groups of agents that are connected with others based on a a) fully-connected topology b) small-world topology c) ring topology or d) dynamic topology.

  Install required packages You can install all required python packages by creating a new conda environment containing the packages in environment.yml:

  conda env create -f environment.yml

  And then activating the environment:

  conda activate sapiens

  Example usages Under notebooks there is a Jupyter notebook that will guide you through setting up simulations with a fully-connected and a dynamic social network structure for solving Wordcraft tasks. It also explains how you can access visualizations of the metrics produced during th$

  Reproducing the paper results Scripts under the scripts directory are useful for reproducing results and figures appearing in the paper.

  With scripts/reproduce_runs.py you can run all simulations presented in the paper from scratch.

  This file is useful for looking at how the experiments were configured but better avoid running it: simulations will run locally and sequentially and will take months to complete.

  Instead, you can access the data files output by simulations on this online repo.

  Download this zip file and uncompress it under the projects directory. This should create a projects/paper_done sub-directory.

  You can now reproduce all visualization presented in the paper. Run:

  python scripts/reproduce_visuals.py

  This will save some general plots under visuals, while project-specific plots are saved under the corresponding project in projects/paper_done

• URL:
  https://­github.­com/­eleninisioti/­SAPIENS
• Contact:
  Eleni Nisioti

7.1.9 architect-builder-abig

• Name:
  Architect-Builder Iterated Guiding
• Keyword:
  Artificial intelligence
• Functional Description:

  Codebase for the paper Learning to guide and to be guided in the Architect-Builder Problem

  ABIG stands for Architect-Builder Iterated Guiding and is an algorithmic solution to the Architect-Builder Problem. The algorithm leverages a learned model of the builder to guide it while the builder uses self-imitation learning to reinforce its guided behavior.

• URL:
  https://­github.­com/­flowersteam/­architect-builder-abig
• Contact:
  Tristan Karch

7.1.10 EAGER

• Name:
  Exploit question-Answering Grounding for effective Exploration in language-conditioned Reinforcement learning
• Keywords:
  Reinforcement learning, Language, Question Generation Question Answering, Reward shaping
• Functional Description:
  A novel QG/QA framework for RL called EAGER In EAGER, an agent reuses the initial language goal sentence to generate a set of questions (QG): each of these self-generated questions defines an auxiliary objective. Here, generating a question consists in masking a word of the initial language goal. Then the agent tries to answer these questions (guess the missing word) only by observing its trajectory so far. When it manages to answer a question correctly (QA) it obtains an intrinsic reward proportional to its confidence in the answer. The QA module is trained using a set of successful example trajectories. If the agent follows a path too different from correct ones at some point in its trajectory, the QA module will not answer the question correctly, resulting in zero intrinsic reward. The sum of all the intrinsic rewards measures the quality of a trajectory in relation to the given goal. In other words, maximizing this intrinsic reward incentivizes the agent to produce behaviour that unambiguously explains various aspects of the given goal.
• URL:
  https://­github.­com/­flowersteam/­EAGER
• Contact:
  Thomas Carta

7.1.11 IMGC-MARL

• Name:
  Intrinsically Motivated Goal-Conditioned Reinforcement Learning in Multi-Agent Environments
• Keywords:
  Reinforcement learning, Multi-agent, Curiosity
• Functional Description:

  This repo contains the code base of the paper Intrinsically Motivated Goal-Conditioned Reinforcement Learning in Multi-Agent Environments.

  In this work, we have studied the importance of the alignment of goals in the training of instrinsically motivated agents in the multi agent goal conditioned RL case. We also proposed a simple algorithm called goal coordination game which allows such agent to learn, in a completely decentralized/selfish way, to communicate in order to align their goal.

  The repository contains the code to reproduce the results of the paper. Which includes a custom RL environment ( using SimplePlayground "game engine"), model used (architecture + hyperparameters) and custom training (mostly based on RLlib ) to train both the model and the communication. We also provide the scripts for the training of every condition we test and notebook to study the results.

• URL:
  https://­github.­com/­Reytuag/­imgc-marl
• Contact:
  Gautier Hamon

7.1.12 Flow-Lenia

• Name:
  Flow Lenia: Mass conservation for the study of virtual creatures in continuous cellular automata
• Keywords:
  Cellular automaton, Self-organization
• Functional Description:

  This repo contains the code to run the Flow Lenia system which is a continuous parametrized cellular automaton with mass conservation. This work extends the classic Lenia system with mass conservation and allows to implement new feature like local parameter, environment components etc

  Several declination of the system (1 or several channels etc ) are available

  Please refer to the associated paper for the details of the system

  Implemented in JAX

• URL:
  https://­github.­com/­erwanplantec/­FlowLenia
• Contact:
  Gautier Hamon

7.1.13 Kidlearn: money game application

• Functional Description:
  The games is instantiated in a browser environment where students are proposed exercises in the form of money/token games (see Figure 1). For an exercise type, one object is presented with a given tagged price and the learner has to choose which combination of bank notes, coins or abstract tokens need to be taken from the wallet to buy the object, with various constraints depending on exercises parameters. The games have been developed using web technologies, HTML5, javascript and Django.

  

  

  

  

• URL:
  https://­flowers.­inria.­fr/­research/­kidlearn/
• Contact:
  Benjamin Clement

7.1.14 cognitive-testbattery

• Name:
  Cognitive test battery of human attention and memory
• Keywords:
  Open Access, Cognitive sciences
• Scientific Description:
  Cognitive test batteries are widely used in diverse research fields, such as cognitive training, cognitive disorder assessment, or brain mechanism understanding. Although they need flexibility according to the objectives of their usage, most of the test batteries are not be available as open-source software and not be tuned by researchers in detail. The present study introduces an open-source cognitive test battery to assess attention and memory, using a javascript library, p5.js. Because of the ubiquitous nature of dynamic attention in our daily lives, it is crucial to have tools for its assessment or training. For that purpose, our test battery includes seven cognitive tasks (multiple-objects tracking, enumeration, go/no-go, load-induced blindness, task-switching, working memory, and memorability), common in cognitive science literature. By using the test battery, we conducted an online experiment to collect the benchmark data. Results conducted on two separate days showed the high cross-day reliability. Specifically, the task performance did not largely change with the different days. Besides, our test battery captures diverse individual differences and can evaluate them based on the cognitive factors extracted from latent factor analysis. Since we share our source code as open-source software, users can expand and manipulate experimental conditions flexibly. Our test battery is also flexible in terms of the experimental environment, i.e., it is possible to experiment either online or in a laboratory environment.
• Functional Description:
  The evaluation battery consists of 6 cognitive activities (serious games: multi-object tracking, enumeration, go/no-go, Corsi, load-induced blindness, taskswitching, memorability). Easily deployable as a web application, it can be re-used and modified for new experiments. The tool is documented in order to facilitate the deployment and the analysis of results.
• URL:
  https://­github.­com/­flowersteam/­cognitive-testbattery
• Publication:
  hal-03723887
• Contact:
  Maxime Adolphe
• Participants:
  Pierre-yves Oudeyer, Hélène Sauzéon, Masataka Sawayama, Maxime Adolphe

7.2.1 ToGather application

Participants: Cécile Mazon, Hélène Sauzéon, Eric Meyer, Isabeau Saint-Supery.

• Name:
  Application for Specialized education
• Keywords:
  Parent-professional relationships; user-centered design; school inclusion; autism spectrum disorder; ecosystemic approach
• Participants:
  Isabeau Saint-supery, Cécile Mazon, Hélène Sauzéon, Agilonaute
• Scientific Description:
  With participatory design methods, we have designed an interactive website application for educational purposes. This application aims to provide interactive services with continuously updated content for the stakeholders of school inclusion of children with specific educational needs. Especially, the services provide: 1) the student's profile with strengths and weaknesses; 2) an evaluation and monitoring over time of the student's repertoire of acquired, emerging or targeted skills; 3) a shared notebook of effective psycho-educational solutions for the student ; 4) a shared messaging system for exchanging "news" about the student and his/her family and, 5) a meeting manager allowing updates of evaluations (student progress). This application is currently assessed with a field study. Then, it will be transferred to the Academy of Nouvelle-Aquitaine-Bordeaux of the National Education Ministery.
• URL:
  The website is not online yet.
• Publication:
  hal-03436355

8.1.1 Testing the Learning Progress Hypothesis in Curiosity-Driven exploration in Human Adults

Participants: Pierre-Yves Oudeyer [correspondant], Alexandr Ten.

Alexandr Ten defended his PhD thesis 62, on this topic, which main results were published in 32.

8.2.1 A new research perspective: Autotelic agents with language and culture internalization

Participants: Cédric Colas [correspondant], Tristan Karch, Clément Moulin-frier, Pierre-Yves Oudeyer.

One of the fundamental goals of our team is to build autonomous agents able to grow open-ended repertoires of skills across their lives. As mentioned in previous section, a promising developmental approach recommends the design of intrinsically motivated agents that learn new skills by generating and pursuing their own goals—autotelic agents. But despite recent progress, existing algorithms still show serious limitations in terms of goal diversity, exploration, generalization, or skill composition. In a recent perspective paper  102 published in Nature Machine Intelligence, we call for the immersion of autotelic agents into rich socio-cultural worlds, an immensely important attribute of our environment that shapes human cognition but is mostly omitted in modern AI. Inspired by the seminal work of Vygotsky, we propose Vygotskian autotelic agents—agents able to internalize their interactions with others and turn them into cognitive tools (as illustrated in figure 2. We focus on language and show how its structure and informational content may support the development of new cognitive functions in artificial agents as it does in humans. We justify the approach by uncovering several examples of new artificial cognitive functions emerging from interactions between language and embodiment in recent works at the intersection of deep reinforcement learning and natural language processing. Looking forward, we highlight future opportunities and challenges for Vygotskian autotelic AI research, including the use of language models as cultural models supporting artificial cognitive development.

8.2.2 Autotelic agents in complex visuo-linguistic environments (ALFRED)

Participants: Laetitia Teodorescu [correspondant], Eric Yuan, Marc-Alexandre Côté, Pierre-Yves Oudeyer.

In the following subsections we describe projects and new results on language-augmented autotelic agents. Following long-standing work in the team on the concept of Intrinsically Motivated Goal Exploration Processes 117, the concept of the autotelic agent has been developed. Much like a child imagines play scenarios and creative goals to achieve, for the simple pleasure of doing novel and interesting things (build a castle, climb a tree), an autotelic agent is a goal-conditioned agent that imagines its own goals and tires to achieve them. By doing so it collects experience that allows it to master an open-ended repertoire of skills. This section describes work on language-conditioned autotelic agents (see 101 and also Section 8.2.1 for a definition of Vygotskian autotelic agents).

The aim of this project was to scale the autotelic agent framework to a richer setting consisting of visual observations of a simulated rich indoor environment, to showcase the abilities of an agent to learn to explore while interacting with a social partner in natural language. For this purpose we placed ourselves inside the ALFRED 168 indoor simulator. ALFRED is a framework for instruction-following agents (closely tied to research in robotics). It consists in a virtual environment containing several rooms with medium visual complexity (many objects, randomized textures and lighting) containing different interactable objects (ingredients, furniture, appliances, etc). The environment also features a simplified action space (discrete movement and symbolic interactions such as open fridge, turn on oven, etc). ALFRED comes with a set of predefined tasks that form the associated benchmark, as well as with a dataset of planner-generated trajectories for each of the predefined tasks and annotations in a symbolic as well as natural language (the latter being generated by a set of Amazon Mechanical Turk workers).

One of the difficulties in training language-based autotelic agents is language grounding 131; the ability of the agent to relate its linguistic instruction to its current observation. Such language grounding is actually needed in 2 modules: first to train a reward function that is able to take as input one or several visual observations as well as the language goal and predict a scalar reward quantifying how much the observations fit the goal (a multimodal similarity function reminiscent of CLIP 161). This (instruction-conditioned) reward function is then used to train the agent within the goal-conditioned reinforcement learning (RL) paradigm. The second difficulty concerns the multitask nature of autotelic agents. The most efficient way to train goal-conditioned agents is to use hindsight experience replay (HER) 81, allowing to leverage previous experience of the agent setting itself a goal that it did not manage to achieve, re-labelling it with the achieved goal instead. For HER to work, we need a so-called relabeller, a function that takes the trajectory as input and outputs one or several sentences describing the goals the agent actually managed to achieve.

In this project, the aim was to sidestep the reward function grounding by leveraging the dataset provided by the ALFRED framework to pre-train the agent with behavioral cloning (BC), thus sidestepping the problem of learning a reward function. We also leverage the dataset to pre-train a video-based captioner that we can use as our relabeler function for hindsight relabelling. We use as a base the code of the Episodic Transformer 159 paper, using a Transformer-based BC agent trained on the ALFRED dataset. For the captioner we used a transformer-based seq2seq model similar to the ones used in Natural Language Translation, taking as input the sequence of frames in an episode and outputting the corresponding language that describes what happened in the episode. The natural-language (NL) based captioner, trained on a total of 8k trajectories with an average of 3 NL annotations, was prone to hallucinations and repetitions (a common pitfall for language models trained on a small amount of data). The most common failure case was the captioner inventing navigation descriptions that did not exist; the captioner based on symbolic language was better behaved. However, these captioners only ever described goals that were part of the ALFRED training data. The Episodic Transformer agent was similarly trained on the same data, and thus was never able to use direct interaction with the environment to discover novel skills, thus getting stuck at the same success rate of about 30 percent on the benchmark tasks. Furthermore, we identified that navigation was the most important bottleneck in this environment, with a total success rate of about 60%; and since navigation is necessary to get to any object that the agent needs to navigate to this greatly impacts the ability of the agent to complete especially long tasks.

From these experiments we concluded that for language-based autotelic agents to learn more skills than they currently master, they need to leverage a source of linguistic knowledge that is broader than their prior knowledge, allowing them to correctly identify, label and integrate novel things they observe in their environment (see section 8.2.4 for further developments on leveraging external sources of linguistic knowledge).

8.2.3 Autotelic agents in complex textual environments (ScienceWorld)

Participants: Laetitia Teodorescu [correspondant], Eric Yuan, Marc-Alexandre Côté, Pierre-Yves Oudeyer.

To sidestep the harder part of language grounding and focus more specifically on exploration dynamics, we have performed series of experiments in text environments for language-based autotelic agents. Text worlds 105 are RL environments that feature textual observation and action spaces, and have first been proposed as a challenging framework for RL agents because of the large, combinatorial action space and necessity for language understanding. Additionally, they are an excellent framework to study language-instructed agents because of the ease of defining a reward function and relabelling function, thus sidestepping the problem of learning a multimodal reward function or a captioner.

In this project we are interested in studying different exploration drivers in the context of an autotelic agent's learning algorithm. We define 3 ways in which an autotelic agent can discover novel environment interactions it can use as goals:

• Failed goal achievement leading to relabelling of novel goals;
• Further random exploration after a goal has been achieved;
• Goal composition, in particular chaining different goals.

The latter two exploration drivers are the ones we study in particular in this project, and are inspired by the Go-Explore 112 framework. Please see Figure 5 for an outline of the architecture of the agent.

Overview of the language-based autotelic agent architecture used for the ScienceWorld experiments. At each time step the agent observes a goal g(e)g^{(e)} and an observation ot(e)o_t^{(e)} and produces an action at(e)a_t^{(e)}. The reward is given by a goal-conditioned reward function and is used to decide whether to terminate the current episode. If the episode is terminated, if the agent is under the seq_compose condition, with probability 0.5 the agent samples another goal. If the agent is under the do_then_explore condition then 5 additional steps of exploration are produced with a random policy. Once the trajectory is over, it is relabelled by the relabeller based on the chosen relevant goals and added to the modular memory buffer. The buffer also receives the reward signal to update the online estimation of competence for the agent. A new episode then starts, and the memory buffer is used to produce a new goal for the agent. While the agent is interacting with the environment, the replay buffer samples minibatches of transitions and uses those to train the policy by minimizing the TD-error.

We place ourselves in the ScienceWorld 182 textual environment, a complex textual environment featuring a series of rooms in a house with many different objects. ScienceWorld implements simplified versions of thermodynamics, biology, newtonian mechanics, chemistry and electronics. The environment was designed with the aim of testing scientific knowledge of text agents in an interactive context. We leverage this environment with complex interactions to define a goal tree for a language-conditioned RL agent (see Figure) with several layers of depth, featuring nested goals. For the agent we build upon the DRRN 124 agent of the original ScienceWorld paper. This agent is a TD-error based agent with a Huber loss. In this work, goals are defined as a part of the textual observation space, and are achieved if the text describing them is present in the environment description (string matching). Only the goals in this goal tree are allowed to be relabeled. We study the impact of the following exploration drivers on final performance over all goals:

• do_then_explore: After the provided goal has been achieved or the maximum allowed number of steps have been reached, we perform an additional 5 steps of random exploration.
• seq_compose: After the provided goal has been achieved or the maximum allowed number of steps have been reached, with probability 0.5 we choose another goal and pursue this one.

We demonstrate that an agent combining do_then_explore with seq_compose and that uses the current estimated performance to bias sampling of the first goal towards the easy ones and sampling of the hard goal towards the hard ones achieved the best final performance and the lowest variance off all considered variants (See Table). This underlines the importance of correctly following the goal curriculum in this setting, targeting the easier goals allowing the agent to first find and then master easier goals first and building upon these goals to achieve the hardest ones.

This is ongoing work, and is done in collaboration with Marc-Alexandre Côté and Eric Yuan from Microsoft Research Montreal.

8.2.4 Autotelic Agents Augmented with Large Language Models

Participants: Cédric Colas [correspondant], Laetitia Teodorescu, Eric Yuan, Marc-Alexandre Côté, Pierre-Yves Oudeyer.

In this project, we take a step towards truly open-ended language-based autotelic agents by leveraging GPT3 90, a Large Language Model (LLM) demonstrating impressive language understanding capabilities. For an autotelic agent to be truly open-ended, it needs to be able to:

• Generate its own goals creatively (Goal generator)
• For an arbitrary goal, decide whether a given trajectory achieved the goal or not (Reward function)
• For a given trajectory, give a list of relevant achieved goals (Relabeller or Social Partner)

In this project, we also place ourselves in a textual environment. CookingWorld has been proposed as part of the First TextWorld Problems competition on text agents, and features a house with ingredients that can be cooked to achieve a recipe. We leverage the textual nature of the trajectories of agents in this environment by using GPT3, with different prompts, as our goal generator, reward function, and relabeller. More precisely:

• Goal generator: we present GPT3 with a prompt composed of a context explanation ("We are in this video game environment"), a list of previously seen rooms, a list of previously seen objects, a list of previously achieved goals, possibly a past trajectory example, and ask it to generate new goals;
• Reward function: given an arbitrary language goal and a trajectory including actions and observations, labelled with the steps index at which those actions and observations were issued, we ask GPT3 to decide whether a given goal is achieved or not in the trajectory, and if it is, at which time step. In this way, we only need to call the GPT3 based reward function once per episode (compared with once per step in a traditional RL setup), significantly speeding up our experiments.
• Relabeller: Once a trajectory is finished, we show GPT-3 the completed trajectory and ask it to provide a list of goals achieved in this trajectory. We extract these goals and we add the relabelled trajectories to the replay buffer.

As in the ScienceWorld project (8.2.3), we use a variant of the DRRN architecture for text worlds (cite). We tried several variants of the architecture with transformers instead of a recurrent architecture, and using a sentencebert (cite) pooling operation. We find nontrivial performance of this truly open-ended textual agent on a set of human-defined evaluation goals, as well as on the set of imagined and relabelled goals. Imagined and relabelled goals have also have a nontrivial stem distribution (see Figure). We also demonstrate the alignment of the GPT3 reward function with a priori human-defined reward functions: (true positive rate of 0.81, false positive fate of 0., the latter being excellent news for use in a reward function).

This is ongoing work, and is done in collaboration with Marc-Alexandre Côté and Eric Yuan from Microsoft Research Montreal.

8.2.5 EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL

Participants: Thomas Carta, Pierre-Yves Oudeyer, Olivier Sigaud [ISIR Sorbonne Université, Paris, France], Sylvain Lamprier [Univ Angers, LERIA].

Reinforcement learning (RL) in long horizon and sparse reward tasks is notoriously difficult and requires a lot of training steps. A standard solution to speed up the process is to leverage additional reward signals, shaping it to better guide the learning process. In the context of language-conditioned RL, the abstraction and generalisation properties of the language input provide opportunities for more efficient ways of shaping the reward. In this paper, we leverage this idea and propose an automated reward shaping method where the agent extracts auxiliary objectives from the general language goal. These auxiliary objectives use a question generation (QG) and question answering (QA) system: they consist of questions leading the agent to try to reconstruct partial information about the global goal using its own trajectory. When it succeeds, it receives an intrinsic reward proportional to its confidence in its answer (as illustrated in figure 7). This incentivizes the agent to generate trajectories which unambiguously explain various aspects of the general language goal. Our experimental study shows that this approach, which does not require engineer intervention to design the auxiliary objectives, improves sample efficiency by effectively directing exploration.

8.2.6 Open-ended recipe crafting through meta-learning

Participants: Gautier Hamon [correspondant], Eleni Nisioti, Clément Moulin-Frier.

As a first step towards studying the evolution of open-ended skill acquisition in artificial agents, we studied the environmental conditions favoring the systematic exploration of combinatorial recipes involving various objects. In this work, the training of an agent uses meta-learning where an outer loop, equivalent to an evolutionary mechanism, meta-learns the parameters of an inner loop which can be seen as a developmental mechanism (where the agent acquire skills during its lifetime by interacting with the environment). In the current setup we use $R{L}^2$ as our meta-learning algorithm 111, 183 which has already been used for acquiring behaviors efficiently balancing exploration and exploitation in a simple navigation task. Other work studied how different conditions in a bandit favor the evolution of innate vs learned behaviors 136.

Our experiments with recipe crafting are inspired by the little alchemy game. The difference with previous works in similar environments (e.g. 8.5.3) is that at every episode the structure of the recipe is randomly chosen. The agent therefore cannot predict what recipes will be rewarding and have to explore different combinations of objects in order to find the rewarding ones. The agent should also memorize the successful and unsuccessful combinations in order to explore and exploit efficiently.

Our preliminary results use both in a vectorized version of the game (where the agents action are only to choose the 2 objects to combine) and an embodied gridworld version (where the agent has to move, grab objects and put them on top of others in order to craft new ones). In both of these cases, the training efficiently meta learns an exploration/exploitation strategy which is to try new recipes (most of the time it does not try non working recipes more than once) until it finds the working ones and then simply exploits them by making them over and over.

Further work will study how we can change the environment/training in order to evolve open-ended exploration strategies where an agent will continuously explore new recipes even if it has already found rewarding ones, as a way to be better prepared for future changes in the recipe structure. We hypothesize that this could be done by introducing drastic changes of recipes which the agent has to anticipate in order to survive.

This work uses JAX python library for both the model/machine learning part and the environment simulation. JAX allows easy parallelization and fast GPU computation and so learning it through this project will be useful for later projects.

8.2.7 Robust Autotelic Learning using Learning Progress in Visual Deep Reinforcement Learning: the GRIMGEP architecture

Participants: Grgur Kovač [correspondant], Adrien Laversanne-Finot, Pierre-Yves Oudeyer.

Autonomous agents, using novelty based goal exploration, are often efficient in environments that require exploration. However, they get attracted to various forms of distracting unlearnable regions. To address this problem, Absolute Learning Progress (ALP) has been used in reinforcement learning agents with predefined goal features and access to expert knowledge. This work extends those concepts to unsupervised image-based goal exploration.

We present the GRIMGEP framework: it provides a learned robust goal sampling prior that can be used on top of current state-of-the-art novelty seeking goal exploration approaches, enabling them to ignore noisy distracting regions while searching for novelty in the learnable regions. It clusters the goal space and estimates ALP for each cluster. These ALP estimates can then be used to detect the distracting regions, and build a prior that enables further goal sampling mechanisms to ignore them.

We construct an image based environment with distractors, on which we show that wrapping current state-of-the-art goal exploration algorithms with our framework allows them to concentrate on interesting regions of the environment and drastically improve performances.

In our experiments shown on figure 9, we compare the performance of two novelty-based exploration approaches: Countbased, and Skewfit (with two different values of its hyperparameter $\alpha$). We can see that wrapping all baselines with the GRIMGEP framework drastically improves their performance.

(a)	(b)	(c)

Most of this project was conducted in the year 2021. In this year we added an ablation study in which we analysed the impact of choosing the cluster by ALP compared to uniformly. We observed that ALP is essential for the CountBased approach, while for Skewfit the clustering part of GRIMGEP is sufficient to escape the noisy distractor. We finalized the paper and published it in 39. The source code is available at.

8.3.1 Towards Automatic Curriculum Learning in Hierarchical Task Spaces

Participants: Rémy Portelas [correspondant], Harm van Seijen, Pierre-Yves Oudeyer.

This work follows a previous team contribution: the TeachMyAgent benchmark 31, which allows researchers to conduct thorough experiments with state-of-the-art Automatic Curriculum Learning methods in various deep reinforcement learning scenarios. Our – still ongoing – objective is to extend the benchmark to feature a new set of ACL challenges: Procgen's 99 challenging pixel-based environments. In addition, considering controllable Procgen environments allows to focus on an understudied ACL problem: how to deal with hierarchical task spaces?, i.e. how to control level generation when encoded as a parametric tree. As a first step towards hierarchical ACL, we propose to adapt ALP-GMM 28 into Hierarchical-ALP-GMM (HALP-GMM). HALP-GMM decomposes the parameter sampling process by using one ALP-GMM teacher per group of parameters, creating a tree of ALP-GMM teachers. After validating our approach in a simulated task space, we are performing experiments on a controllable (hierarchical) Coinrun environment.

8.3.2 Automatic Curriculum Learning for Language Modeling

Participants: Clément Romac [correspondant], Rémy Portelas, Pierre-Yves Oudeyer.

We showed in recent works how Automatic Curriculum Learning (ACL) could help Deep Reinforcement Learning methods by tayloring a curriculum adapted to learner's capabilities 31, 28. Using ACL can lead to sample efficiency, asymptotic performance boost and help in solving hard tasks.

Parallel to this, recent works in Language Modeling using Transformers (e.g. GPT-2) have starting to get more interested in better understanding convergence and learning dynamics of these models. Trained in a supervised setup, these models are fed with hundred of millions of natural language sequences crawled from the web. The current standard way of training these models (i.e. constructing batches of randomly selected sequences) makes the assumption that all sequences have same interest for the model. However, recent works showed that this does not seem to be the case and that datasets can contain outliers harming training. Additionally, some works also showed that hand-designing a curriculum over sequences (e.g. ordered by their length) could speed up and stabilize training.

Building on this, we propose to experiment how ACL could help taylor such a curriculum in an automated way relying on Learning Progress. Our study has several contributions:

• Propose a standardized and more in-depth comparison of current curriculum learning methods used to train language models
• Introduce the first study of ACL in such a training
• Use ACL to propose deeper insights about training dynamics of Transformer models when doing Language Modeling by analysing generated curricula and Learning Progress estimations

We chose to train GPT-2 on the standard OSCAR dataset and use teacher algorithms to select samples that are shown to the model (see fig. 10).

Using ACL, we perform an in-depth analysis of prior methods changing the size of tokens' sequences observed during training following a hand-designed curriculum. Our experiments showed that a Random baseline outperforms these methods. We also provide, thanks to ACL methods based on Learning-Progress Multi-Armed Bandits, hints that while short sequences should not be used as training advances (as Large Language Models quickly learn them), there is no clear evidence that short sequences should be prioritized (and thus long sequences avoided) at the beginning of training.

Additionally, we performed several experiments using more advanced ACL methods on different task spaces and show that these lead to overfitting and underperform in comparison the no-curriculum strategy usually applied in Language Modeling. We hypothesize that, given how large models used in Language Modeling are, it is better to give a huge amount of very diverse and different samples (even though outliers or harmful samples exist) without any curriculum than using a curriculum that restrains the diversity of samples and introduces duplicates (leading to overfitting).

8.4.1 Self-organizing cultural conventions in a new interactive AI paradigm: the Architect-Builder Problem

In this experiment, we are interested in interactive agents that learn to coordinate, namely, a builder – which performs actions but ignores the goal of the task, i.e. has no access to rewards – and an architect which guides the builder towards the goal of the task. We define and explore a formal setting where artificial agents are equipped with mechanisms that allow them to simultaneously learn a task while at the same time evolving a shared communication protocol. Ideally, such learning should only rely on high-level communication priors and be able to handle a large variety of tasks and meanings while deriving communication protocols that can be reused across tasks. We present the Architect-Builder Problem (ABP): an asymmetrical setting in which an architect must learn to guide a builder towards constructing a specific structure. The architect knows the target structure but cannot act in the environment and can only send arbitrary messages to the builder. The builder on the other hand can act in the environment, but receives no rewards nor has any knowledge about the task, and must learn to solve it relying only on the messages sent by the architect. Crucially, the meaning of messages is initially not defined nor shared between the agents but must be negotiated throughout learning. The Architect-Builder problem was initially introduced by Vollmer et. al 180 in an experiment named the CoCo game studying the formation of communication protocol between humans in such a context. Diagrams of interactions in the CoCo game and in our numerical adaptation are given in figure 11.

(a)	(b)

Under these constraints, we propose Architect-Builder Iterated Guiding (ABIG), a solution to ABP where the architect leverages a learned model of the builder to guide it while the builder uses self-imitation learning to reinforce its guided behavior. We analyze the key learning mechanisms of ABIG and test it in 2D tasks involving grasping cubes, placing them at a given location, or building various shapes. ABIG results in a low-level, high-frequency, guiding communication protocol that not only enables an architect-builder pair to solve the task at hand, but that can also generalize to unseen tasks as illustrated in figure 12. These results were published at the International Conference on Representation Learning (ICLR 2022) 53.

8.4.2 Emergence of a Shared Graphical Language from Visual Inputs

Participants: Tristan Karch [correspondant], Yoann Lemesle, Clément Moulin-frier, Pierre-Yves Oudeyer.

For this project, we place ourselves in the framework of Language Games which are vastly used to study the emergence of languages in populations of agents. Recent contributions relying on deep learning methods focused on agents communicating via an idealized communication channel, where utterances produced by a speaker are directly perceived by a listener. This comes in contrast with human communication, which instead relies on a sensory-motor channel, where motor commands produced by the speaker (e.g. vocal or gestural articulators) result in sensory effects perceived by the listener (e.g. audio or visual). Here, we investigate if agents can evolve a shared language when they are equipped with a continuous sensory-motor system to produce and perceive signs, e.g. drawings. To this end, we introduce the Graphical Referential Game (GREG) where a speaker must produce a graphical utterance to name a visual referent object consisting of combinations of MNIST digits while a listener has to select the corresponding object among distractor referents, given the produced message. The utterances are drawing images produced using dynamical motor primitives combined with a sketching library. A schematic of the GREG is provided in figure 13

The Graphical Referential Game: During the game, the speaker's goal is to produce a motor command cc that will yield an utterance uu in order to denote a referent rSr_S sampled from a context R˜S\tilde{R}_S. Following this step, the listener needs to interpret the utterance in order to guess the referent it denotes among a context R˜L\tilde{R}_L. The game is a success if the listener and the speaker agree on the referent (rL≡rSr_L\equiv r_S).

To tackle GREG we present CURVES: a multimodal contrastive deep learning mechanism that represents the energy (alignment) between named referents and utterances generated through gradient ascent on the learned energy landscape. We, then, present a set of experiments showing that our method allows the emergence of a shared, graphical language that generalizes to feature compositions never seen during training. We also propose a topographic metric to investigate the compositionality of emergent graphical symbols. Finally, we conduct an ablation study illustrating that sensory-motor constraints are required to yield interpretable lexicons.

(a)	(b)

8.4.3 Intrinsically motivated agents in multi-agent environments

Participants: Eleni Nisioti [correspondant], Elias Masquil, Gautier Hamon, Clément Moulin-Frier.

The intrinsically-motivated goal-conditioned learning paradigm is well-established in single-agent settings: by setting its own goals and pursuing them in an environment without external supervision, such an agent is able to acquire a wide diversity of skills 103. Such agents are called autotelic, from the Greek words auto (self) and telos (end). What happens when you transfer the autotelic paradigm to multi-agent environments, where some skills may require the cooperation of multiple agents (lifting a heavy box for example)? This is the question we aimed at addressing in this project. We believe that multi-agent applications will benefit from agents that can autonomously discover and learn cooperative skills, but entail additional challenges to the ones found in single-agent settings: agents that independently set their own goals will have a very low probability of simultaneously sampling the same cooperative goal, which will make solving these goals difficult.

To explore this question, we implemented what we call cooperative navigation tasks in the Simple Playgrounds environments. This 2-D environment, illustrated on the left of figure 15, consists of a room with 6 landmarks on its walls and two agents that receive continuous-valued observations about the distance and angle to all landmarks and other agents and perform discrete-valued actions that control their angular velocity and longitudinal force. A navigation task is a description of the landmarks that need to be reached: some tasks are individual (for example "at least one agent reaches the red landmark") and some are cooperative (for example "at least one agent reaches the red landmark and at least one agent reaches the blue landmark"). Each autotelic agent is learning using the RL algorithm PPO and, at each training episode, chooses which goal to pursue by random sampling within the goal distribution. In addition to a policy conditioned on its goals, the agent also needs a reward function that indicates whether a goal is achieved (see the schematic on the right of figure 15 for an illustration of the main algorithmic components of autotelic agents). In this project, we assume that the two agents already know this reward function and focus on examining the process of decentralized goal selection.

(a)	(b)

Our empirical analysis of this set-up showed that goal alignment is important for solving the cooperative tasks we considered: agents that were independently sampling the goals failed to solve all tasks (see orange curve in figure 16) while agents whose goals were determined by a centralized process (blue curve) that guaranteed that the two agents are always pursuing the same goal performed optimally. We then wondered: can we achieve the same performance without requiring centralization? To achieve this we designed a communication-based algorithm that enables a group to align its goals while remaining decentralized: at the beginning of an episode and before determining their goals the two agents exchange messages and then use these messages to condition their goal-selection (see dashed arrows in the schematic on the right of figure 15). This communication is asymmetric: one randomly chosen agent, the leader, uses its goal generator to choose which goal to pursue and then decides what message to transmit to the follower, which conditions its goal selection on the received message. We observed that the agents learn a communication protocol that leads to the alignment of cooperative goals, even though they were not directly incentivised to do so. They were both independently learning a protocol that maximised their individual rewards but, as we show in our experiments corresponding to figure 16, goal alignment was able to emerge from such decentralized learning. We called this algorithm the Goal-coordination game, as it was inspired from another emergent communication algorithm called the Naming game 171.

To get a better understanding of how alignment helps solve this task we measured specialization, which is the tendency of agents to always go the same landmark when there are two options. For example, if for the goal "at least one agent reaches the red landmark and at least one agent reaches the blue landmark" the first agent always go to red and the second goes to blue, then specialization is maximum. We empirically observed that specialization correlates with alignment and is an optimal strategy in our tasks (see right plot of figure 16).

This work was submitted at the AAMAS 2023 conference and a preprint is available on HAL 66. The source code for reproducing the experiments is available at.

8.4.4 The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents

Participants: Grgur Kovač [correspondant], Remy Portelas, Peter Ford Dominey, Pierre-Yves Oudeyer.

Developmental psychologists have long-established socio-cognitive abilities as a core aspect of human intelligence and development 18191. Those abilities enable us to enter, participate in and benefit from human culture. Humans are then able to push our culture forward by improving on already existing cultural artifacts. This is referred to as the cumulative cultural evolution, and it has been argued that the most impressive of human achievements are the product of it 176. It seems clear that to construct artificial agents capable of interacting with and learning from humans one must equip them with socio-cognitive abilities enabling them to enter the human culture. This would enable artificial agents to benefit from our culture, and also push it forwards, i.e. to participate in the cumulative cultural evolution.

Current AI research mostly studies asocial settings or, in the case of Multi-Agent Reinforcement Learning, the emergence of culture (how culture emerges in the first place, rather than how one enters an already existing culture). Furthermore, this research is often done without a strong grounding in developmental psychology.

In this project, we follow the work of Michael Tomasello and Jerome Bruner who outlined a set of core socio-cognitive skills, e.g. social cognition (joint attention, theory of mind), referential and conventionalized communication, imitation, role-reversal, scaffolding, and many more 175, 91. Importantly, they also showed the importance the (cultural) environment for cognitive development.

To introduce some of those concepts to the AI community, we constructed a procedurally generated suite of environments - The SocialAI school. With The SocialAI school, experiments studying those socio-cognitive abilities can be easily conducted and cognitive-science experiments reconstructed. An example of a SocialAI school environment is shown in figure 17. In it the peer is pointing towards the red box. The agent (the red triangle) has to infer this to mean that the apple is hidden inside the red box.

We conducted many experiments, here we outline a few more important ones. In the following experiments we experimented with multimodal RL agents. We tested generalization of inferring the meaning of referential communication (ex. the pointing gesture) to new scenarios/objects. We found that such a generalization is very hard for standard reinforcement learning agents. We show how a scaffolded environment helps with learning complex interaction sequences (formats). To show how cognitve science experiments can be recreated - we reconstruct a study of role reversal from 115. Furthermore, we conducted experiments regarding other aspects of social-cognition: joint attention, imitation, perspective taking, etc. The SocialAI school is not limited to RL agents. For example, these environments can easily be transformed into pure text. We created one case study to demonstrate how this feature can be used to study large-language models as interactive agents.

We stared working on this project in the year 2021, in which we created only a few small experiments 135. In this year, we significantly extended the project. We grounded it more explicitly in cognitive-science (especially in the work of Tomasello and Bruner). We created a procedural generation engine using which we constructed many additional environments and conducted many additional experiments (discussed above).

8.5.1 Research perspective: The Ecology of Open-Ended skill Acquisition

Participants: Clément Moulin-Frier [correspondant], Eleni Nisioti, Pierre-Yves Oudeyer.

An intriguing feature of the human species is our ability to continuously invent new problems and to proactively acquiring new skills in order to solve them: what is called Open-Ended Skill Acquisition (OESA). Understanding the mechanisms underlying OESA is an important scientific challenge in both cognitive science (e.g. by studying infant cognitive development, see section 8.1) and in artificial intelligence (aiming at computational architectures capable of open-ended learning, see section 8.2). Both fields, however, mostly focus on cognitive and social mechanisms at the scale of an individual’s life. It is rarely acknowledged that OESA, an ability that is fundamentally related to the characteristics of human intelligence, has been necessarily shaped by ecological, evolutionary and cultural mechanisms interacting at multiple spatiotemporal scales.

We have recently initiated a new research direction aiming at understanding, modeling and simulating the dynamics of OESA in artificial systems, grounded in theories studying its eco-evolutionary bases in the human species. For this aim, we have proposed a conceptual framework, called ORIGINS (illustrated Fig. 18 and developed in 60), expressing the complex interactions between environmental, adaptive, multi-agent and cultural dynamics. This framework raises three main research questions:

• What are the ecological conditions favoring the evolution of autotelic agents?
• How to bootstrap the formation of a cultural repertoire in populations of adaptive agents?
• What is the role of cultural feedback effects in the open-ended dynamics of human skill acquisition?

The contributions described below are addressing some aspects of these research questions.

8.5.2 Evolution of plasticity and evolvability in variable environments

Participants: Eleni Nisioti [correspondant], Clément Moulin-Frier.

The diversity and quality of natural systems have been a puzzle and inspiration for communities studying artificial life. It is now widely admitted that the adaptation mechanisms enabling these properties are largely influenced by the environments they inhabit. Organisms facing environmental variability have two alternative adaptation mechanisms operating at different timescales: plasticity, the ability of a phenotype to survive in diverse environments and evolvability, the ability to adapt through mutations. Although vital under environmental variability, both mechanisms are associated with fitness costs hypothesized to render them unnecessary in stable environments. In this work, we aimed at studying the interplay between environmental dynamics and adaptation in a minimal model of the evolution of plasticity and evolvability.

To achieve this we designed a simulation environment that attempts to capture the spatial and temporal heterogeneity of real-world environments while keeping the computational complexity low: the user can choose the number of niches, which are arranged in a simple longitudinal model, and a climate function that captures the temporal variation of environmental conditions, which are arranged based on a simple longitudinal model (see left of figure 19 for an illustration of the environment). We defined the evolvability of an agent as its mutation rate and capture plasticity using tolerance curves, a tool developed in ecology 120. Tolerance curves (which we visualize on the right of Figure 19) have the form of a Gaussian whose mean shows the preferred environmental state of an individual and the variance its plasticity, i.e., its ability to survive under different environmental conditions. This figure also illustrates the cost and benefit of plasticity. If both individuals are at their preferred niche, which coincides with the environmental state, then the plastic individual has lower fitness that the non-plastic (cost of plasticity). If the actual environmental state differs significantly from the preferred one, the plastic individual has higher fitness (benefit of plasticity).

 

We conducted an extensive empirical study in this environment that aimed at disentangling the effects of different mechanisms: we studied three types of climate functions (stable, sinusoid, noisy) and two types of evolutionary selection pressures (survival of the fittest and niche-limited competition) and environments were the number of niches varies from 1 to 100. Through these simulations we showed that environmental dynamics affect plasticity and evolvability differently and that the selection mechanism matters: a) in stable environments with a large number of niches when both selection-based fitness and niche-limited competition (we call this method NF-selection) are activated, plasticity remains high despite its cost (see left plot in Figure 20) ; b) in a noisy environment introducing niche-limited competition (N-selection and NF-selection) makes populations capable of resisting larger amounts of noise (see right plot in Figure 20). We presented our work at GECCO 2022 55 and open-sourced the software for reproducing our simulations at.

 

8.5.3 The effect of social network structure on collective innovation

Participants: Eleni Nisioti [correspondant], Mateo Mahaut, Pierre-Yves Oudeyer, Ida Momennejad, Clément Moulin-Frier.

Innovations are a central component of open-ended skill acquisition: they denote the emergence of new solutions by the recombination of existing ones and their presence is necessary to ensure a continuous complexification of an agent's cultural repertoire. While we often tend to attribute discoveries to certain innovative individuals, if we shed a broad perspective at the history of our species we see that human innovation is primarily a collective process. Fields such as psychology and anthropology have been studying the ability of human groups to innovate for some time, with studies indicating that the social network structure has a significant impact: fully-connected structures are better suited for quick convergence in easy problems with clear global optima, while partially-connected structures perform best in difficult tasks where local optima may lure agents away from the globally optimal solution 110. At the same time a parallel story is unfolding in reinforcement learning (RL): distributed RL is a sub-field where multiple agents solve a task collectively 150. Compared to the single-agent paradigm, distributed RL algorithms converge quicker and often achieve superior performance. However, these algorithms have only considered full connectivity. In this inter-disciplinary project, we presented a novel learning framework that augments distributed RL with the notion of a social network structure and employed it to study the hypothesis from human studies that partial connectivity performs best in innovation tasks.

We implemented such innovation tasks using Wordcraft, a recently introduced RL playground inspired from the Little Alchemy 2 game (see left of figure 21 for an illustration of how this task works). We considered a wide diversity of social network structures: static structures that remain constant throughout learning (fully-connected, ring, small-world) and a dynamic structure where the group oscillates between phases of low and high connectivity (we illustrate this dynamic structure on the right of figure 21). Each agent in our implementation employs the DQN learning algorithm and exchanges experiences that have the form of sequences of state-action combinations with its neighbors.

A central conclusion of our empirical analysis was that the dynamic social network structure performs best. In addition to the performance groups achieve we measured behavioral and mnemonic metrics such as behavioral conformity and mnemonic diversity. Such metrics were inspired from human studies and helped us further analyze the behavior of groups. For example, one empirical observation was that sharing experiences did not help the group learn quicker in a very simple innovation task; instead the fully-connected group was the slowest. By looking at the diversity in the memories of the agents we observed that the fully-connected structure had the highest individual diversity (left of figure 22 ) and the lowest group diversity (right of figure 22): sharing experiences with others diversifies an individual's experiences but also homogenizes the group, which is bad for its performance.

We see the contribution of this project as two-fold. From the perspective of fields studying human intelligence, we have shown that using RL algorithms as computational tool is a promising direction towards increasing the verisimilitude of simulations and analyzing both behavior and memory. From the perspective of RL, we have shown that distributed RL algorithm should move beyond the fully-connected architecture and explore groups with dynamic topologies. This work is currently a preprint 67 and under review at ICLR 2023 and we open-source the code at.

8.5.4 Socially Supervised Representation Learning: the Role of Subjectivity in Learning Efficient Representations

Participants: Julius Taylor, Eleni Nisioti [correspondant], Clément Moulin-Frier.

Results

We show that our proposed architecture allows the emergence of aligned representations. This means that different agents find similar encodings for the same sensory inputs. The subjectivity introduced by presenting agents with distinct perspectives of the environment state contributes to learning abstract representations that outperform those learned by a single autoencoder and a population of autoencoders, presented with identical perspectives of the environment state, which is shown in the left column of Fig. 23. Furthermore, in Fig. 23 (right) we show that the learned representations are data-efficient, i.e. they enjoy the most benefit when evaluated on small testing splits. This is important, because good representations should allow agents to adapt to downstream tasks quickly and with few samples. Altogether, our results demonstrate how communication from subjective perspectives can lead to the acquisition of more abstract representations in multi-agent systems, opening promising perspectives for future research at the intersection of representation learning and emergent communication.

8.6.1 Machine Learning for Adaptive Personalization in Intelligent Tutoring Systems

Participants: Pierre-Yves Oudeyer [correspondant], Benjamin Clément, Didier Roy, Hélène Sauzeon.

8.6.2 Machine learning for adaptive cognitive training

Participants: Pierre-Yves Oudeyer, Hélène Sauzéon [correspondant], Masataka Sawayama, Benjamin Clément, Maxime Adolphe.

Because of its cross-cutting nature to all cognitive activities such as learning tasks, attention is a hallmark of good cognitive health throughout life and more particularly in the current context of societal crisis of attention. Recent works have shown the great potential of computerized attention training for an example of attention training, with efficient training transfers to other cognitive activities, and this, over a wide spectrum of individuals (children, elderly, individuals with cognitive pathology such as Attention Deficit and Hyperactivity Disorders). Despite this promising result, a major hurdle is challenging: the high inter-individual variability in responding to such interventions. Some individuals are good responders (significant improvement) to the intervention, others respond variably, and finally some respond poorly, not at all, or occasionally. A central limitation of computerized attention training systems is that the training sequences operate in a linear, non-personalized manner: difficulty increases in the same way and along the same dimensions for all subjects. However, different subjects require in principle a progression at a different, personalized pace according to the different dimensions that characterize attentional training exercises.

To tackle the issue of inter-individual variability, the present project proposes to apply some principles from intelligent tutoring systems (ITS) to the field of attention training. In this context, we have already developed automatic curriculum learning algorithms such as those developed in the KidLearn project, which allow to customize the learner's path according to his/her progress and thus optimize his/her learning trajectory while stimulating his/her motivation by the progress made. ITS are widely identified in intervention research as a successful way to address the challenge of personalization, but no studies to date have actually been conducted for attention training. Thus, whether ITS, and in particular personalization algorithms, can optimize the number of respondents to an attention training program remains an open question.

To investigate this question, an ongoing work on systematically reviewing the literature of the use of ITS in the field of cognitive training has been started. In parallel to this, a web platform has been designed for planning and implementing remote behavioural studies. This tool provides means for registering recruited participants remotely and executing complete experimental protocols: from presenting instructions and obtaining informed consents, to administering behavioural tasks and questionnaires, potentially throughout multiple sessions spanning days or weeks. In addition to this platform, a cognitive test battery composed of seven classical behavioural tasks has been developed. This battery aims to evaluate the evolution of the cognitive performance of participants before and after training. Fully open-source, it mainly targets attention and memory. A preliminary study on a large sample of 50 healthy participants showed that the developed tasks reproduced the results of previous studies, that there were large differences between individuals (no ceiling effect) and that the results were significantly reliable between two measurements taken on two days separated by one night 35.

Utilizing these tools, a pilot study campaign was conducted to evaluate the impact of our AI-based personalized cognitive training program. The first experiment involved n=27 participants and aimed to compare the effectiveness of a cognitive training program using a linear difficulty management procedure (staircase procedure) to a program using an ITS for difficulty manipulation. The online training lasted for 10 hours over a period of 2 weeks. The results indicated that the ITS-based intervention produced diverse learning trajectories compared to the linear procedure 24, leading to broader improvements in pre-post cognitive assessment. However, no significant differences were observed in subjective measures of motivation and engagement between the two groups.

Subsequent to this initial experiment, two pilot studies (n=11 and n=10, respectively) were conducted with the goal of enhancing motivation and engagement in the game. The first study implemented gamified components such as scores and feedback, while the second study examined hyperparameter updates to the ITS. The analysis of learning trajectories, learning outcomes, and subjective measures yielded promising results in favor of the AI-based personalized procedure. However, the optimal hyperparameter configuration for the ITS algorithm remains uncertain due to the multiple mechanisms involved. To address this issue, a toy environment is being developed with the objective of identifying an a priori optimal configuration of the ITS. The next step will be to test this optimal configuration in a new study with human participants.

Different learning trajectories for a selected participant in the staircase group (left) and the ITS group (right). The color of a dot indicates the initial presentation of the parameter value, while the size of the dot represents the frequency of the parameter value.

8.6.3 Fostering curiosity and meta-cognition in children using conversational agents

Participants: Pierre-Yves Oudeyer, Hélène Sauzéon [correspondant], Mehdi Alami, Rania Abdelghani, Didier Roy, Edith Law, Pauline Lucas.

Since 2019 via the renewal of the Idex cooperation fund (between the University of Bordeaux and the University of Waterloo, Canada) led by the Flowers team and also involving F. Lotte from the Potioc team, we continue our work on the development of new curiosity-driven interaction systems. Although experiments have been slowed down by sanitary conditions, progress has been made in this area of application of FLOWERS works. In particular, three studies have been completed.

The first study regards a new interactive educational application to foster curiosity-driven question-asking in children. This study has been performed during the Master 2 internship of Mehdi Alaimi co-supervised by H. Sauzéon, E. Law and PY Oudeyer. It addresses a key challenge for 21st-century schools, i.e., teaching diverse students with varied abilities and motivations for learning, such as curiosity within educational settings. Among variables eliciting curiosity state, one is known as « knowledge gap », which is a motor for curiosity-driven exploration and learning. It leads to question-asking which is an important factor in the curiosity process and the construction of academic knowledge. However, children questions in classroom are not really frequent and don’t really necessitate deep reasoning. Determined to improve children’s curiosity, we developed a digital application aiming to foster curiosity-related question-asking from texts and their perception of curiosity. To assess its efficiency, we conducted a study with 95 fifth grade students of Bordeaux elementary schools. Two types of interventions were designed, one trying to focus children on the construction of low-level question (i.e. convergent) and one focusing them on high-level questions (i.e. divergent) with the help of prompts or questions starters models. We observed that both interventions increased the number of divergent questions, the question fluency performance, while they did not significantly improve the curiosity perception despite high intrinsic motivation scores they have elicited in children. The curiosity-trait score positively impacted the divergent question score under divergent condition, but not under convergent condition. The overall results supported the efficiency and usefulness of digital applications for fostering children’s curiosity that we need to explore further. The overall results are published in CHI'20 80. In parallel to these first experimental works, we wrote this year a review of the existing works on the subject 93.

The second study investigates the neurophysiological underpinnings of curiosity and the opportunities of their use for Brain-computer interactions 82. Understanding the neurophysiological mechanisms underlying curiosity and therefore being able to identify the curiosity level of a person, would provide useful information for researchers and designers in numerous fields such as neuroscience, psychology, and computer science. A first step to uncovering the neural correlates of curiosity is to collect neurophysiological signals during states of curiosity, in order to develop signal processing and machine learning (ML) tools to recognize the curious states from the non-curious ones. Thus, we ran an experiment in which we used electroencephalography (EEG) to measure the brain activity of participants as they were induced into states of curiosity, using trivia question and answer chains. We used two ML algorithms, i.e. Filter Bank Common Spatial Pattern (FBCSP) coupled with a Linear Discriminant Algorithm (LDA), as well as a Filter Bank Tangent Space Classifier (FBTSC), to classify the curious EEG signals from the non-curious ones. Global results indicate that both algorithms obtained better performances in the 3-to-5s time windows, suggesting an optimal time window length of 4 seconds to go towards curiosity states estimation based on EEG signals. These results have been published 82

Finally, the third study investigates the role of intrinsic motivation in spatial learning in children (paper in progress). In this study, the state curiosity is manipulated as a preference for a level of uncertainty during the exploration of new environments. To this end, a series of virtual environments have been created and is presented to children. During encoding, participants explore routes in environments according the three levels of uncertainty (low, medium, and high), thanks to a virtual reality headset and controllers and, are later asked to retrace their travelled routes. The exploration area and the wayfinding. ie the route overlap between encoding and retrieval phase, (an indicator of spatial memory accuracy) are measured. Neuropsychological tests are also performed. Preliminary results showed that there are better performances under the medium uncertainty condition in terms of exploration area and wayfinding score. These first results supports the idea that curiosity states are a learning booster (paper in progress).

At the end of 2020, we started an industrial collaboration project with EvidenceB on this topic (CIFRE contract of Rania Abdelghani validated by the ANRT). The overall objective of the thesis is to propose new educational technologies driven by epistemic curiosity, and allowing children to express themselves more and learn better. To this end, a central question of the work will be to specify the impact of self-questioning aroused by states of curiosity about student performance. Another objective will be to create and study the pedagogical impact of new educational technologies in real situations (schools) promoting an active education of students based on their curiosity. To this end, a web platform called 'Kids Ask' has been designed, developed and tested in three primary schools. The tool offers an interaction with a conversational agent that trains children's abilities to generate curiosity-driven questions and use these questions to explore a learning environment and acquire new knowledge. The results suggest that the configuration helped enhance children's questioning and exploratory behaviors; they also show that learning progress differences in children can be explained by the differences in their curiosity-driven behaviors 34.

Illustration of a conversational agent's strategies in the different work spaces of the "Kids Ask" platform

Despite showing pedagogical efficiency, the method used in the first study of this PhD is still very limited since it relies on generating curiosity-prompting cues by hand for each educational resource in order to feed the "discussion" with the agent, which can be a very long and costly process. For this reason, a logical follow-up to scale-up and generalize this study was to explore ways to automate the said conversational agents' behaviors in order to facilitate their implementation on a larger scale and for different learning tasks. More particularly, we move towards the natural language processing (NLP) field and the large language models (LLMs) that showed an impressive ability in generating text that resembles the way people write.

In this context, we study using the recent LLM GPT-3 to implement conversational agents that can prompt children's curiosity about a given text-based educational content, by proposing some specific cues. We investigate the validity of this automation method by comparing its impact on children's divergent question-asking skills with respect to the hand-crafted condition we had in our previous work. In a second step, we explore using GPT-3 to propose a new curiosity-prompting behavior for our agent that aims to better support the children's needs of competence, autonomy and relatedness during the question-asking training.

The study was conducted in two primary schools with 75 children aged between 9 and 11. Our first results suggest the validity of using GPT-3 to facilitate the implementation of curiosity-stimulating learning technologies. Indeed, children's performance was similar between the conditions where they had hand-generated or GPT-3-generated cues. In a second step, we also found that GPT-3 can be efficient in proposing the relevant cues that leave children with more autonomy to express their curiosity  63 (publication in process).

Finally, as a follow-up direction to this line of work, we design new digital interventions that focus on eliciting the metacognitive mechanisms involved in the stimulation and continuity of curiosity, and not just giving the tools to pursue it as done in the previous studies. For this, we use findings from the theories explaining curiosity in order to break this latter up into a set of relevant metacognitive skills. We then take an operational approach to propose adequate digital metacognitive exercises for each one of the said skills (i.e. exercises to identify uncertainty, generate hypotheses etc). We aim to implement this set of metacognitive exercises and investigate its impact on children's abilities to initiate and maintain curious behaviors. We would also be interested in investigating the impact of such training on the learning progress children can achieve. A pilot study has been conducted to evaluate the accessibility of this new training and we are working to recruit more participants to investigate the desired effects in the upcoming months.

8.6.4 ToGather : Interactive website to foster collaboration among stakeholders of school inclusion for pupils with neurodevelopmental disorders

Participants: Hélène Sauzéon [correspondant], Cécile Mazon, Eric Meyer, Isabeau Saint-Supery, Christelle Maillart [Uni. Liège, Belgium], Kamélia Belassel, Mathieu Périé.

Technology-based interventions to improve school inclusion of children with neurodevelopmental disorders have mostly been individual centered, focusing on their socio-adaptive and cognitive impairments and implying they have to adapt themselves in order to fit in our society's expectations. Although this approach centered on the normalization of the person has some advantages (reduction of clinical symptoms), it carries social stereotypes and misconceptions of cognitive disability that are not respectful of the cognitive diversity and intrinsic motivations of the person, and in particular of the student's wishes in terms of school curriculum to achieve his or her future life project.

The "ToGather" project aims at enlightening the field of educational technologies for special education by proposing an approach centered on the educational needs of the students and by bringing a concerted and informed answer between all the stakeholders including the student and all their support spheres (family, school, medico-social care). To this end, ToGather project that emanates from participatory design methods, primarily consists of having developed a pragmatic tool (interactive website) to help students with cognitive disability and their caregivers to formalize and to visualize the repertoire of academic skills of the student and to make it evolve according to his or her proximal zone of development (in the sense of Vygotsky) on the one hand, and to the intrinsic motivations of the student (his or her own educational and life project) on the other 145.

Usability validation in France

To validate its usability (interaction data, user experience, motivations of different users, etc. ), user testing were performed on the mock-up of the website with 7 parents, 7 teachers and 7 social services professionals (n=21 in total).

Participants were given a scenario with tasks to perform on the mock-up, then they were asked to answer some questionnaires to collect subjective measures, followed by an interview to debrief on the experiment. It was conducted online so the participants' screen were recorded and a video analysis was performed according to a grid to collect objective measures (effectiveness and efficiency). Using the mock-up elicited low cognitive load and showed low average perceived difficulty of usability with an excellent user experience, as asked per the users who participated in the design process. The high level of elicited self-determination feeling underlines the need for a collaborative tool to gather information on the pupil, empower each stakeholder by inviting them to participate in every collective decision and giving them the means to coordinate actions led towards the pupil.

Perceived difficulty answers’ distribution depending on the role, the assessed tab and the test instance on a scale from 0 (very easy) to 10 (very difficult)

From those results, user testing for the new release of ToGather appears conclusive   69 (under review).

8.6.5 Curious and therefore not overloaded : Study of the links between curiosity and cognitive load in learning mediated by immersive technologies

Participants: Hélène Sauzéon [correspondant], Matisse Poupard, André Tricot [Cosupervisor - Univ. Montpellier], Florian Larrue [Industrialist - Le Catie].

With the ever-increasing interest in digital technologies in education, many questions about their use and effectiveness are emerging. In this context, this thesis will focus on the relationships between three key dimensions of technology-mediated learning: the learner's internal learning processes, the instructional design, and the educational technology used. In partnership with CATIE (industrial partner) and the EPSYLON laboratory of the University of Montpellier (PR. André Tricot), two main objectives are targeted in this research program started in April 2022:

• To relate the theory of cognitive load to models of curiosity -driven learning;
• To experiment the impact of the choice of educational technology on the links between pedagogical choice (guided instruction vs. exploration) and learner expertise, by studying the reversal effect due to expertise.

To this end, the program includes 3 main phases of study:

8.7.1 Design of an Interactive Software for Automated Discovery in Complex Systems

Participants: Clément Romac [correspondant], Jesse Lin, Mathieu Périé, Mayalen Etcheverry, Clément Moulin-Frier, Pierre-Yves Oudeyer.

Secondly, this year was mostly focused on moving our software towards large-scale experiments and fostering human-in-the-loop interactions with automated discovery algorithms. For the former, we now allow users to run experiments on remote servers through the use of simple configuration files. Our software then seamlessly communicates with the distant server, sends the code necessary for the experiment, launches the experiment and retrieves results while the experiment is running allowing users to still monitor in quasi-real-time. Our method supports any server, including SLURM clusters commonly used for large-scale experiments.

We also focused our efforts on implementing tools and interfaces for users to give feedback or instructions to the automated discovery algorithm that explores the complex system. As identified by 16, empowering experiments to collaborate with automated discovery methods can be key to obtain interesting discoveries. Integrating such a collaborative process in our tool came with several engineering challenges and we are currently experimenting with our solution and working on making it user-friendly for non-experts end users.

8.7.2 Learning Sensorimotor Agency in Cellular Automata

Participants: Gautier Hamon [correspondant], Mayalen Etcheverry, Bert Chan, Clément Moulin-Frier, Pierre-Yves Oudeyer.

We use Lenia continuous cellular automaton as our artificial "world"  94. We introduce a novel method based on gradient descent and curriculum learning combined within an intrinsically-motivated goal exploration process (IMGEP) to automatically search parameters of the CA rule that can self-organize spatially localized 1 and moving patterns 2 within Lenia. The IMGEP defines an outer exploratory loop (generation of training goal/loss) and an inner optimization loop (goal-conditioned). We use a population-based version of IMGEP 17, 104 but introduce two novel elements compared to previous papers in the IMGEP literature. First, whereas previous work in 30 and 16 used a very basic nearest-neighbor goal-achievement strategy, our work relies on gradient descent for the local optimization of the (sensitive) parameters of the complex system, which has shown to be very powerful. To do so we made a differentiable version of the Lenia framework, which is also a contribution of this work. Secondly, we propose to control subparts of the environmental dynamics with functional constraints (through predefined channels and kernels in Lenia) to build a curriculum of tasks; and to integrate this stochasticity in the inner optimization loop. This has shown central to train the system to emerge sensorimotor agents that are robust to stochastic perturbations in the environment. In particular, we focus on modeling obstacles in the environment physics and propose to probe the agent sensorimotor capability as its performance to move forward under a variety of obstacle configurations. We also provide in this work tests and metrics to measure the robustness of the obtained agents.

While many complex behaviors have already been observed in Lenia, among which some could qualify as sensorimotor behaviors, they have so far been discovered "by chance" as the result of time-consuming manual search or with simple evolutionary algorithms. Our method provides a more systematic way to automatically learn the CA rules leading to the emergence of basic sensorimotor structures, as shown in Figure 32. Moreover, we investigated and provided ways to measure the (zero-shot) generalization of the discovered sensorimotor agents to several out-of-distribution perturbations that were not encountered during training. Impressively, even though the agents still fail to preserve their integrity in certain configurations, they show very strong robustness to most of the tested variations. The agents are able to navigate in unseen and harder environmental configurations while self-maintaining their individuality (Figure 30). Not only the agents are able to recover their individuality when subjected to external perturbations but also when subjected to internal perturbations: they resist variations of the morphogenetic processes such that less frequent cell updates, quite drastic changes of scales as well as changes of initialization (Figure 31). Furthermore, when tested in a multi-entity initialization and despite having been trained alone, not only the agents are able to preserve their individuality but they show forms of coordinated interactions (attractiveness and reproduction). Our results suggest that, contrary to the (still predominant) mechanistic view on embodiment, biologically-inspired embodiment could pave the way toward agents with strong coherence and generalization to out-of-distribution changes, mimicking the remarkable robustness of living systems to maintain specific functions despite environmental and body perturbations 133. Searching for rules at the cell-level in order to give rise to higher-level cognitive processes at the level of the organism and at the level of the group of organisms opens many exciting opportunities to the development of embodied approaches in AI in general.

Scatter plot of the agents as their measured performances of robustness to obstacles (y axis) and speed in obstacles (x axis) obtained by IMGEP (red), random search with the same compute resources as IMGEP(blue) and the one from the original lenia paper (green)

The work has been released as a distill-like article which is currently hosted at. This article contains an interactive demo in webGL and javascript, as well as many videos and animations of the results. A colab notebook with the source code of the work is publicly available at. We are currently working on a journal publication.

8.7.3 Flow lenia: Mass conservation for the study of virtual creatures in continuous cellular automata

Participants: Erwan Plantec, Gautier Hamon [correspondant], Mayalen Etcheverry, Pierre-Yves Oudeyer, Clément Moulin-Frier, Bert Chan.

Following our work on trying to find sensorimotor cabapabilities in cellular automata such as Lenia   95, 94, we kept exploring the search for low level cognition in continuous cellular automata. This led to preliminary search on trying to emerge memory in self-organizing agents as well as work on trying to implement other environmental constraints in the CA in order to emerge interesting behavior. To implement more easily those environmental constraints as well as to ease the emergence of spatially localized patterns (and thus have the optimization/search to focus more on the cognitive ability, removing the need to optimize to prevent uncontrollable growth/explosion of the pattern), we worked on adding mass conservation to the Lenia system.

We propose in this work a mass-conservative (i.e the sum of the CA’s activations remains constant over time) extension to Lenia called Flow Lenia 68. We hypothesize that such conservation laws will help in the search for artificial life-forms by constraining emerging patterns to spatially localized ones. It also allows to implement more easily environmental constraints on the self-organizing agents such as a need for food to grow etc.

Furthermore, we show that this new model allows for the integration of the update rule parameters within the CA dynamics enabling the emergence of creatures with different parameters and so different properties in the same environment/grid. This leads to multi-species simulation where the grid is filled with agents with different behaviors and properties 33. Such a feature opens up research perspectives towards the achievement of open-ended intrinsic evolution inside continuous CAs, which means that all the evolutionary part would be a result of the dynamic of the CA (without any external loop/system). We hypothesize that this open-ended instrinsic evolution could, through the competition/cooperation, lead to the emergence of interesting low level cognition in those system.

Multi-species simulation in Flow Lenia where each colour represents different parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.

Multi-species simulation in Flow Lenia where each colour represents different values of parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.: Multi-species simulation in Flow Lenia where each colour represents different parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.

Multi-species simulation in Flow Lenia where each colour represents different values of parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.: Multi-species simulation in Flow Lenia where each colour represents different parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.

Multi-species simulation in Flow Lenia where each colour represents different values of parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.: Multi-species simulation in Flow Lenia where each colour represents different parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.

Multi-species simulation in Flow Lenia where each colour represents different values of parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.: Multi-species simulation in Flow Lenia where each colour represents different parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.

Simple evolutionary strategy (with an evolutionary loop outside the system) was also used to optimized for pattern with directional and rotational movement.

You can find some examples of the system and pattern in this companion website, including the ones trained for movement, random parameters, food in flow Lenia, and multi species simulations: see. Notebook with the system can be found : here.

This work led to an oral presentation to the WIVACE 2022, 15th International Workshop on Artificial Life and Evolutionary Computation.

8.7.4 Use of Automated Discovery Tools for the study of Goal-Directedness and Navigation Competency in Models of Gene Regulatory Networks

Participants: Mayalen Etcheverry [correspondant], Clément Moulin-Frier, Pierre-Yves Oudeyer, Michael Levin.

Previous work around project 8.7 has shown that modern tools that leverage computational models of curiosity developed in the Flowers team can be transposed to form efficient AI-driven “discovery assistants” that assist scientists in mapping and navigating the space of possible outcomes in complex systems 30, 16, 78. In particular, recent work from project 8.7.2 has shown how these tools can be very useful to study the origins of sensorimotor agency in continuous cellular automata models. This work is part of the emerging effort to study the fundamental mechanisms that enable the emergence of agency and cognition across all (and unconventional) domains of biological and synthetic life, which is known as the field of basal cognition  141. In particular, the lab of Dr. Michael Levin at Tufts University is pioneer in the field of basal cognition and has shown that it holds great promises to open pioneering opportunities for top down control in synthetic and regenerative biology  138. Discussions with Michael Levin has highlighted the great opportunity to leverage curiosity-driven exploration algorithms as tools to empower scientific exploration and analysis of basal cognition in numerical models of biological systems. Hence we have started a collaboration with his Lab, under the form of a 5 month academic exchange of Mayalen Etcheverry in their lab in Boston.

The objective of this collaboration is to develop a new machine-learning-based toolkit to study goal-directedness and navigation competency in self-organizing systems, more particularly focusing the research program on the analysis of navigation competency  114 in numerical models of gene regulatory networks (GRN).

GRN are molecular network that describe how genes in a cell influence the activation/inhibition of others by means of their end product. Those network of interactions govern protein synthesis and hence cellular functions. Understanding the dynamics of these pathways is a central challenge for the design of effective strategies for bio-medicine and bio-engineering.

In this project, we aim to study the behavioral competencies of these molecular networks in navigating the transcriptional space, i.e. the space of gene expression. In particular we are interesting in evaluating the goal-directedness of these molecular networks and their robustness to a distribution of perturbations (dynamical noise, pushes, energetical barrier, structural changes in the network, etc). To do so, we propose to use intrinsically motivated goal exploration processes (IMGEPs) to automatically organize the exploration of initial GRN states that will lead to diverse final states (goals) in the transcriptional space.

To that end, we have worked on the design of two software packages: AutoDiscJax and SBMLtoODEJax. AutoDiscJax is a python software built upon jax, that allows to perform automated discovery and exploration of complex systems and that we use to organize the exploration of computational models of biological GRNs. SBMLtoODEJax is a python package which allows to convert Systems Biology Markup Language (SBML) models into python classes that can then easily be simulated and manipulated in python projects. SBMLtoODEJax makes use of jax mains features which allows: just-in-time compilation and automatic vectorization, hence parallel computing and speed-ups, as well as automatic differentiation and hence gradient-descent based optimization. Those codebases are at the moment private but will be publicly released in 2023.

After having defined an experimental framework to quantitatively study the navigation competency and goal-directedness in gene regulatory networks, we have preliminary results on the ability of certain networks to reach goals in the transcriptional space under a variety of perturbations. Future work will consist in running experimental campaign on a larger database of GRN models and will hopefully lead to a scientific publication.

8.8.1 Language-biased image classification: evaluation based on semantic representations

Participants: Yoann Lemesle, Masataka Sawayama, Guillermo Valle-Perez, Maxime Adolphe, Hélène Sauzéon, Pierre-Yves Oudeyer.

Humans show language-biased image recognition for a word-embedded image, known as picture-word interference. Such interference depends on hierarchical semantic categories and reflects that human language processing highly interacts with visual processing. Similar to humans, recent artificial models jointly trained on texts and images, e.g., OpenAI CLIP, show language-biased image classification. Exploring whether the bias leads to interference similar to those observed in humans can contribute to understanding how much the model acquires hierarchical semantic representations from joint learning of language and vision 34.

In this work 48, 75, we introduced methodological tools from the cognitive science literature to assess the biases of artificial models. Specifically, we introduce a benchmark task to test whether words superimposed on images can distort the image classification across different category levels and, if it can, whether the perturbation is due to the shared semantic representation between language and vision. Our dataset is a set of word-embedded images and consists of a mixture of natural image datasets and hierarchical word labels with superordinate/basic category levels. Using this benchmark test, we evaluate the CLIP model. We show that presenting words distorts the image classification by the model across different category levels, but the effect does not depend on the semantic relationship between images and embedded words. This suggests that the semantic word representation in the CLIP visual processing is not shared with the image representation, although the word representation strongly dominates for word-embedded images.

8.8.2 Towards locally adaptive model based deep reinforcement learning

Participants: Rémy Portelas [correspondant], Harm van Seijen, Ali-Rahimi Kalahroudi, Pierre-Yves Oudeyer.

In this project in collaboration with Harm van Seijen and Ali-Rahimi Kalahroudi from Microsoft Research Montréal, we focused on “local adaptivity”, which is a measure defining how quickly an RL method adapts to a local change in the environment. This metric, presented by Harm van Seijen and colleagues in 178 enables to identify model-based behavior and provides insight in how close a method's behavior is from optimal model-based behavior. To enable the study of their proposed Deep RL method showed to be adaptive in simple settings in more complex environments, we created LoCa Coinrun, a modified coinrun environment well-suited for local adaptivity tests. We then performed baseline experiments with PPO, a model-free agent, and EfficientZero, a model-based agent without adaptivity mechanisms, and showed that they both failed to learn in LoCa Coinrun. Next experiments will aim to scale their adaptive approach to LoCa Coinrun.

Research on lifelong Deep Reinforcement Learning of multiple tasks (Microsoft

Participants: Pierre-Yves Oudeyer [correspondant], Laetitia Teodorescu.

Financing of the PhD grant of Laetitia Teodorescu.

Automated Discovery of Self-Organized Structures (Poïetis)

Participants: Pierre-Yves Oudeyer [correspondant], Mayalen Etcheverry.

Financing of the CIFRE PhD grant of Mayalen Etcheverry by Poietis.

Machine learning for adaptive cognitive training (OnePoint)

Participants: Hélène Sauzéon [correspondant], Pierre-Yves Oudeyer, Maxime Adolph.

Financing of the CIFRE PhD grant of Maxime Adolphe by Onepoint.

Curiosity-driven interaction system for learning (evidenceB)

Participants: Hélène Sauzéon [correspondant], Pierre-Yves Oudeyer, Rania Abdelghani.

Financing of the CIFRE PhD grant of Rania Abdelghani by EvidenceB.

Curious and therefore not overloaded : Study of the links between curiosity and cognitive load in learning mediated by immersive technologies (CATIE)

Participants: Hélène Sauzéon [correspondant], Matisse Poupard, André Tricot [Cosupervisor - Univ. Montpellier], Florian Larrue [Industrialist - Le Catie].

Financing of a PhD grant of Matisse Poupard with CATIE and EPSYLON Lab (Univ. Montpellier).

Augmenting curiosity-driven exploration with very large language models in deep reinforcement learning agents (Hugging Face)

Participants: Pierre-Yves Oudeyer [correspondant], Clément Romac, André Tricot [Cosupervisor - Univ. Montpellier], Florian Larrue [Industrialist - Le Catie].

Financing of the PhD grant of Clément Romac by Hugging Face.

Autonomous Driving Commuter Car (Renault)

Participants: David Filliat [correspondant], Emmanuel Battesti.

We developed planning algorithms for a autonomous electric car for Renault SAS in the continuation of the previous ADCC project. We improved our planning algorithm in order to go toward navigation on open roads, in particular with the ability to reach higher speed than previously possible, deal with more road intersection case (roundabouts), and with multiple lane roads (overtake, insertion...).

School+ /ToGather project (FIRAH and Region Nouvelle-Aquitaine)

Participants: Hélène Sauzéon [correspondant], Cécile Mazon, Isabeau Saint-supery, Eric Meyer.

Financing of one year-postdoctoral position and the app. development by the International Foundation for Applied Research on Disability (FIRAH). The School+ project consists of a set of educational technologies to promote inclusion for children with Autism Spectrum Disorder (ASD). School+ primary aims at encouraging the acquisition of socio-adaptive behaviours at school while promoting self-determination (intrinsic motivation), and has been created according to the methods of the User-Centered Design (UCD). Requested by the stakeholders (child, parent, teachers, and clinicians) of school inclusion, Flowers team works to the adding of an interactive tool for a collaborative and shared monitoring of school inclusion of each child with ASD. This new app will be assessed in terms of user experience (usability and elicited intrinsic motivation), self-efficacy of each stakeholder and educational benefit for child. This project includes the Academie de Bordeaux –Nouvelle Aquitaine, the CRA (Health Center for ASD in Aquitania), and the ARI association.

10.1.1 International Programs

10.1.2 Other

The FLOWERS has initiated or is continuing international collaborations, including:

• Collaboration with Bert Chan (Google Brain Tokyo, Japan) on curiosity-driven algorithms applied to continuous cellular automata (sections 8.7.2, 8.7 and 8.7.3).
• Collaboration with Michael Levin (Tufts and Harvard University, USA) on curiosity-driven algorithms applied to the control of gene regulatory networks (section 8.7.4)
• Collaboration with Ida Momennejad (Microsoft Research NYC, USA) on the role of social network structures in collective innovation (section 8.5.3)
• Clément Moulin-Frier is collaborating with Marti Sanchez-Fibla and Ricard Solé (University Pompeu Fabra, Barcelona, Spain) on the cooperative control of environmental extremes by artificial intelligent agents. A preprint is available on Arxiv 163 (soon on HAL) and the paper will be soon submitted to a journal.

10.1.3 Visits to international teams

10.2.1 Horizon Europe

ANR Chaire Individuelle Deep Curiosity

- PY Oudeyer continued to work on the research program of this Chaire, funding 2 PhDs and 3 postdocs for five years (until 2025).

ANR JCJC ECOCURL

- C. Moulin-Frier obtained an ANR JCJC grant. The project is entitled "ECOCURL: Emergent communication through curiosity-driven multi-agent reinforcement learning". The project starts in Feb 2021 for a duration of 48 months. It will fund a PhD student (36 months) and a Research Engineer (18 months) as well as 4 Master internships (one per year).

Inria Exploratory Action ORIGINS

- Clément Moulin-Frier obtained an Exploratory Action from Inria. The project is entitled "ORIGINS: Grounding artificial intelligence in the origins of human behavior". The project starts in October 2020 for a duration of 24 months. It funds a post-doc position (24 months). Eleni Nisioti has been recruited on this grant.

Inria Exploratory Action AIDE

- Didier Roy is collaborator of the Inria Exploratory Action AIDE "Artificial Intelligence Devoted to Education", ported by Frédéric Alexandre (Inria Mnemosyne Project-Team), Margarida Romero (LINE Lab) and Thierry Viéville (Inria Mnemosyne Project-Team, LINE Lab). The aim of this Exploratory Action consists to explore to what extent approaches or methods from cognitive neuroscience, linked to machine learning and knowledge representation, could help to better formalize human learning as studied in educational sciences. AIDE is a four year project started middle 2020 until 2024 see.

10.3.1 Adaptiv'Math

• Adaptiv'Math
• Program: PIA
• Duration: 2019 - 2020
• Coordinator: EvidenceB
• Partners:
  • EvidenceB
  • Nathan
  • APMEP
  • LIP6
  • INRIA
  • ISOGRAD
  • Daesign
  • Schoolab
  • BlueFrog

The solution Adaptiv'Math comes from an innovation partnership for the development of a pedagogical assistant based on artificial intelligence. This partnership is realized in the context of a call for projects from the Ministry of Education to develop a pedagogical plateform to propose and manage mathematical activities intended for teachers and students of cycle 2. The role of Flowers team is to work on the AI of the proposed solution to personalize the pedagogical content to each student. This contribution is based on the work done during the Kidlearn Project and the thesis of Benjamin Clement 97, in which algorithms have been developed to manage and personalize sequence of pedagogical activities. One of the main goal of the team here is to transfer technologies developed in the team in a project with the perspective of industrial scaling.

SNPEA (RNA and Inria)

• Associated researcher : Hélène Sauzéon
• Duration: 2020 - 2025
• Amount: 150 000€
• Participants: Pierre-Yves Oudeyer, Masataka Sawayama, Maxime Adolphe
• Description: the project's objective is twofold: 1) to adapt and develop new algorithms in machine learning to the field of attention training and 2) to evaluate with the help of experimental methods from psychology whether automated personalization according to progress generates more responders in elderly and young people with attentional disorders.

Evaluation TousEnsemble ; ToGatherAssessment project (RNA)

• Associated researcher : Hélène Sauzéon
• Duration: 2021 - 2026
• Amount: 57 000€
• Participants: Cécile Mazon; Eric Meyer; Isabeau Saint-Supery; Christelle Maillart (Univ. Liège); Bordeaux Academy of National French education; Centre Ressources Autisme d'Aquitaine; ARI Association.
• Description: this project is the continuation of the Togather app. project. To validate the effectiveness of this new tool, a controlled study (control group vs. equipped group) is planned over 2 to 3 quarters with 60 students (ASD and/or ID). In addition, a study of applicability to the Walloon context will be conducted.

11.1.1 Scientific events: organisation

11.1.2 Scientific events: selection

11.1.3 Journal

11.1.4 Invited talks

Clément Moulin-Frier gave two invited talks:

• November 2022. “Language, Curiosity, Social Networks And Cultural Innovation”. Workshop Reservoir-SMILES at Inria Bordeaux Sud-Ouest, France.
• October 2022. “Grounding Artificial Intelligence In The Origins Of Human Behavior: The Origins Project”. Annual joint workshop Inria-DFKI at Inria Bordeaux Sud-Ouest, France.

Hélène Sauzéon gave four invited talks:

• HomeAssist technology for aging in place: From technology-design to clinical assessment amongst frail older individuals. INNOVCARE 2nd Annual Forum “Care-led innovation: The case of elderly care in France and Japan”, Fondation France Japon de l'EHESS, December, 7th, Paris, France.
• IA and Education. Annual joint workshop Inria-DFKI at Inria Bordeaux Sud-Ouest, 18- October, 17-19 th, Talence, France.
• Participation in the round table "General state of digital health at the University of Bordeaux", 6th of July, Bordeaux.
• Presentation of scientific advances in the field technologies for mental disability, Audition « Les progrès récents des technologies au service de la prise en charge du handicap à l’Office Parlementaire d’Evaluation des Choix Scientifique et Technologique, vidéos, 27 janvier 2022.

PY Oudeyer gave 2 invited talks:

• Developmental artificial intelligence, at the Life, Structure and Cognition symposium 2022,
• Language and culture internalization in developmental AI, Dagstuhl seminar on Developmental Learning.

Rémy Portelas gave one invited talk at Microsoft Research Montréal on Automatic Curriculum Learning and Deep Reinforcement Learning, on April 24th.

Tristan Karch gave an invited talk at the RL Sofa at Mila (Montreal) in January to present the paper Learning to Guide and to be Guided in the Architect-Builder Problem.

Cédric Colas gave an invited talk at Facebook AI Research Paris, on Vygotskian Autotelic AI in July.

Mayalen Etcheverry gave an invited talk at the ICLR 2022 From Cells to Societies workshop, as well as a panel discussion about self-organization in natural and artificial systems with Alex Mordvintsev & Richard A. Watson.

Cécile Mazon gave an invited talk at the Brain conference NeuroDev (Bordeaux) in May, on digital technologies for the schooling of children with ASD.

11.1.5 Leadership within the scientific community

Flowers’ team members have been highly active withing the scientific community, including the organisation of events, editing journals, reviewing or giving invited talks.

11.1.6 Scientific expertise

• Hélène Sauzéon was a reviewer for the ANRT (2 PhD projects)
• Hélène Sauzéon participated to several Selection committees for permanent positions (e.g., Chaire de professeur junior en Santé & Médecine digitale - solutions digitales pour la sante mentale at the Univ. of Bordeaux).
• PY Oudeyer was a member of the jury selecting grants for PhDs in AI in the context of Plan IA at University of Bordeaux (April 21).
• PY Oudeyer was a member of the advisory board of the BigScience project
• Pierre-Yves Oudeyer and Hélène Sauzéon participated to several Selection committees for permanent positions as researcher (e.g., inria) or assistant professor at the university (2 committee organization for Assistant Professors positions at the Univ. of Bordeaux) , and for young and senior non permanent researchers position at inria (SRP and ARP).
• PY Oudeyer was a member of the hiring committee for the Moex Inria team.

11.1.7 Research administration

• Hélène Sauzéon was the head of Flowers team during the scientific stay of Pierre-Yves Oudeyer, and thus was member of the Project-teams’s committee of the center of Inria of the university of Bordeaux.
• Hélène Sauzéon and Cécile Mazon are members of committee of LILLAB which is a living and learning lab funded by the “délégation interministérielle à la stratégie nationale à l’autisme et troubles neurodéveloppementaux” and aiming the dissemination of knowledge in connection with the 3 centers of excellence for autism and Neurodevelopmental syndromes; since 2020.
• Hélène Sauzéon is member of directory committee of IFHR which is a national institute on disability funded by Inserm aiming the researcher networking and dissemination of knowledge on multidisciplinary research on disability; since 2018.
• Hélène Sauzéon is the head of the Innovations and Transfer Committee of the BIND Center of Excellence in Bordeaux and member of the BIND directory Committee since 2018
• Hélène Sauzéon is a member of extended office of Project-team committee of the centre of Inria of university of Bordeaux, since 2020.
• PY Oudeyer has been head of the team in the september-december period
• PY Oudeyer was member of piloting committees of consortium projects Adaptiv'Maths on educational technologies.
• Cécile Mazon is co-responsible of the workpackage "Digital technologies" of the PIA AspieFriendly
• H. Sauzéon is responsible of Education topic into the centre Inria of the university of Bordeaux

11.2.1 Teaching

Teaching Responsibilities:

• Hélène Sauzéon was director of the curriculum in Technology, Ergonomics, Cognition and handicap (First and Second years of master degree in cognitive Science - University of Bordeaux) until sept. 2022.
• Hélène Sauzéon was in charge of the "Autonomy & Digital" axis of the Bordeaux instanciation of the PIA3 project (2018-22) Aspie-Friendly (P. Monthubert, Univ. Toulouse) aiming to develop digital tools to prepare, support the university inclusion of students with ASD.
• David Filliat is in charge since 2012 of the "Robotics and autonomous systems" third year speciality at ENSTA Paris.
• Sao Mai Nguyen is in charge of the "Robot Learning" third year course at ENSTA Paris.
• Clément Moulin-Frier is responsible professor of the "System Design, Integration and Control" course at the University Pompeu Fabra in Barcelona, Spain.
• Cécile Mazon is responsible of the second year of the curriculum in Technology, Ergonomics, Cognition and Handicap (Cognitive Sciences - University of Bordeaux) since sept. 2021.
• Cécile Mazon is responsible of the curriculum in Technology, Ergonomics, Cognition and Handicap (Cognitive Sciences - University of Bordeaux) since sept. 2022.

Teaching Involvement in Computer / Engineer science or in cognitive science:

• Université de Bordeaux - MIASHS Bachelor: Introductory course on assistive technology, disability and its management in the workplace, 7.5h (Isabeau Saint-Supery)
• ENSC/ENSEIRB Presentation of developmental artificial intelligence and the Flowers Lab, 2h, Option Robot (Laetitia Teodorescu)
• ENSC Introduction to bayesian analysis, 8h, Option AI (Adolphe Maxime)
• BS & Master: Cognitive Science, Univ. of Bordeaux- , 96h, Hélène Sauzéon
• BS & Master: Cognitive Science, Univ. of Bordeaux- , 192h, Cécile Mazon
• Master: Navigation for Robotics, 21 h, M2, ENSTA Paris, David Filliat
• Master: Navigation for Robotics, 24 h, M2 DataAI, IP Paris - Paris, David Filliat
• 2nd year : Deep Learning, 12h, IMT Atlantique (Sao Mai Nguyen).
• Université de Bordeaux - TCEH Master: IT Project Management, 18h (Isabeau Saint-Supery)
• Master UPF-Barcelona: Robotics and AI, 10h (Clément Moulin-Frier)
• Master : PY Oudeyer taught a course (3h) on Developmental Machine Learning at CogMaster, University Paris-Sorbonne (Jan 21)
• Industry : PY Oudeyer gave a course (1h) on Developmental AI at the AI4Industry event, Bordeaux (Jan 21)
• Academic : PY Oudeyer gave an invited tutorial (1h) at the CoRL robot learning conference (Nov 21)
• Master: Cognitive Sience, 24h (Eric Meyer)
• Teacher training at the Rectorat of Bordeaux : digital technologies for students with special educational needs , 6h (Eric Meyer)
• 2nd year Master in cognitive science : Assistive technologies (20h), Rania Abdelghani.
• PY Oudeyer gave a course on developmental reinforcement learning at ENSEIRB (2h), dec. 2022.
• PY Oudeyer gave a course on interaction grounded learning at the Microsoft Research machine learning school (1h30), april. 2022.
• PY Oudeyer gave a course on developmental AI at the Seminaire Numerics, for the EDMI graduate school, University of Bordeaux (1h30), oct. 2022.

11.2.2 Supervision

• PhD in progress: Maxime Adolphe, "Adaptive personalization in attention training systems", beg. in sept. 2020 (supervisors: H. Sauzéon and PY. Oudeyer)
• PhD in progress: Rania Abdelgani, "Fostering curiosity and meta-cognitive skills in educational technologies", beg. in dec. 2020 (supervisors: H. Sauzéon and PY. Oudeyer).
• PhD in progress: Isabeau Saint-Supery, "Designing and Assessing a new interactive tool fostering stakeholders' cooperation for school inclusion", (supervisors: Sauzéon and C. Mazon.)
• PhD in progress : Matisse Poupard "Optimize learning in a digital environment according to learners' level of expertise, epistemic curiosity and mode of instruction", (supervisors : H. Sauzéon and A. Tricot, Univ. Montpellier).
• PhD in progress : Gautier Hamon, "Environmental, adaptive, multi-agent and cultural dynamics of open-ended skill acquisition" '(supervisor: C. Moulin-Frier)
• PhD in progress: Mayalen Etcheverry, "Automated discovery with intrinsically motivated goal exploration processes", beg. in sept. 2020 (supervisors: PY. Oudeyer and C. Moulin-Frier)
• PhD in progress: Laetitia Teodorescu, "Graph Neural Networks in Curiosity-driven Exploring Agents", beg. in sept. 2020 (supervisors: PY. Oudeyer and K. Hoffman)
• PhD in progress: Tristan Karch, "Language acquisition in curiosity-driven Deep RL", beg. in sept. 2019 (supervisors: PY. Oudeyer and C. Moulin-Frier)
• PhD defended : Rémy Portelas, "Teacher algorithms for curriculum learning in Deep RL", beg. in sept. 2018 (supervisors: PY. Oudeyer and K. Hoffmann)
• PhD defended: Alexander Ten, "Models of human curiosity-driven learning and exploration", beg. in sept. 2018 (supervisors: PY. Oudeyer and J. Gottlieb)
• Master Thesis Defended : Elias Masquil "Intrinsically Motivated Goal-Conditioned Reinforcement Learning in Multi-Agent Environments", co-supervised by Eleni Nisioti, Gautier Hamon and Clément Moulin-Frier
• Master Thesis Defended : Yoann Lemesle "Self-organization of shared graphical languages in groups of agents using multimodal contrastive deep learning mechanisms", co-supervised by Tristan Karch, Pierre-Yves Oudeyer, Clément Moulin-Frier and Romain Laroche
• Master Thesis Defended : Erwan Plantec, Mass conservation for the study of virtual creatures in continuous cellular automata", co-supervised by Gautier Hamon, Mayalen Etcheverry, Pierre-Yves Oudeyer, Clément Moulin-Frier and Bert Chan
• Master Thesis Defended: Pauline Lucas " Designing of a conversational agent for fostering the curiosity states in children : Assessment of the use of GPT3- LLM ", co-supervised by R. Abdelghani and H. Sauzéon

11.2.3 Juries

• H. Sauzéon has been president and mentor of HDR degree of E. Altena
• H. Sauzéon was a member of PhD jury of Alexander Ten (Title : "The Role of Progress-Based Intrinsic Motivation in Learning : Evidence from Human Behavior and Future Directions") and was in the PhD "comité de suivi" of Charlotte Bettencourt (Univ. of Paris/ Sorbone University), Hugo Fournier (Univ. of Bordeaux), and Marion Pèch (Univ. of Bordeaux).
• H. Sauzéon and C. Mazon are permanent members of jury of Master degree in cognitive science at the university of Bordeaux.
• C. Colas has been a member of the jury of the undergrad thesis defence of Natan de Almeida Junges from Universidade Tecnológica Federal do Paraná. Title: Learning Pragmatic Frames in a Computational Architecture.
• PY Oudeyer was a member of the admissibility jury of the CR2 competition at Inria Bordeaux Sud-Ouest
• PY Oudeyer was a reviewer in the PhD juries of Leonard Hussenot (Title: "Apprenticeship learning: transferring human motivation to artificial agents", University of Lille), and an examiner in the juries of Ahmed Akakzia (Title: "Teaching Predicate-based Autotelic Agents", University Paris-Sorbonne) and Valentin Guillet (Title: "Distillation de réseaux de neurones pour la généralisation et le transfert en apprentissage par renforcement", University of Toulouse).
• PY Oudeyer was in the PhD "comité de suivi" of Alexandre Chenu (Univ. Paris VI), Marc Welter (Univ. Bordeaux), Jean-Baptiste Gaya (Université Paris-Sorbonne),

11.3.1 Internal or external Inria responsibilities

NA

11.3.2 Articles and contents

• Meyer, E., Saint-Supery, I., Mazon, C., Sauzeon, H. (à paraître) L’inclusion scolaire des enfants ayant un TSA en favorisant la collaboration entre la famille, l’école et les professionnels médicosociaux : l’intérêt des outils numériques. Revue de littérature, Ressources documentaires de la FIRAH.
• R. Abdelghani led writing a blog post on training curiosity in children with conversational agents,
• M. Sawayama led writing a blog post on studying AI systems through the lens of cognitive science methods,
• T. Karch and P. Barde led writing a blog post on self-organization of communication protocols in groups of agents,
• D. Roy and PY. Oudeyer published the second edition of their popular science book for children on AI and robotics, Nathan,

11.3.3 Education

• Sauzéon, H. (2022) L'IA au service d'un apprentissage personnalisé basé sur la curiosité. Colloque InFine - Sens et finalités du numérique pour l’éducation, 13 octobre, Futuroscope Poitiers.
• H. Sauzéon, P.Y. Oudeyer, C. Moulin-Frier, C. Mazon, I. Saint-Supery and M. Etcheverry have received Eleonore, for a one week internship (stage de 3eme) in the team as part of the actions at Inria to promote the inclusion of people with disabilities.
• H. Sauzéon, M. Perie, C. Romac, T. Carta, G. Hamon have received 3 interns, for a day as part of their internship at inria (stage de 3eme). They were introduced to the research topic of the team and did some exploratory play with machine learning tools.

11.3.4 Interventions

• Members of the Flowers team participated to many interviews and documentaries for the press, the radio and television.
• March 2022. Conférence UniThé ou Café - Hélène Sauzéon & Clément Moulin-Frier - Des machines curieuses pour des humains curieux.
• June 2022. Webinaire IMIND - Cécile Mazon presented the study about the pilot study of an ITS-based intervention in specialized classrooms with second degree students with ASD.
• July 2022. Clément Moulin-Frier was interviewed for the Inria Podcast "Désassemblons le numérique".
• September 2022. Webinaire organized by Lorraine University and INSPE Nancy - Cécile Mazon gave a talk about digital technology for children with ASD and the interest of a systemic approach of their support
• November, 2022. G. Hamon and PY. Oudeyer presented the team and the internships available in the team to students of the MVA master (mathématiques vision apprentissage) at their "forum des stages", ENS Paris-Saclay.
• December, 2022. I. Saint-Supery and PY. Oudeyer welcomed and presented the Flowers team to L3 students of Ecole Normale Supérieure de Lyon.
• PY. Oudeyer was interviewed by magazine Polytechnic Insights on modeling curiosity-driven learning in AI and humans

International journals

• 34 articleR.Rania Abdelghani, P.-Y.Pierre-Yves Oudeyer, E.Edith Law, C.Catherine de Vulpillières and H.Hélène Sauzéon. Conversational agents for fostering curiosity-driven learning in children.International Journal of Human-Computer Studies167November 2022, 102887
  HALDOIback to textback to text
• 35 articleM.Maxime Adolphe, M.Masataka Sawayama, D.Denis Maurel, A.Alexandra Delmas, P.-Y.Pierre-Yves Oudeyer and H.Helene Sauzeon. An Open-Source Cognitive Test Battery to Assess Human Attention and Memory.Frontiers in Psychology13June 2022
  HALDOIback to textback to text
• 36 articleC.Cédric Colas, T.Tristan Karch, C.Clément Moulin-Frier and P.-Y.Pierre-Yves Oudeyer. Language and Culture Internalisation for Human-Like Autotelic AI.Nature Machine Intelligence2022
  HAL
• 37 articleC.Cédric Colas, T.Tristan Karch, O.Olivier Sigaud and P.-Y.Pierre-Yves Oudeyer. Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: A Short Survey.Journal of Artificial Intelligence ResearchJuly 2022
  HAL
• 38 articleS.Sébastien Forestier, R.Rémy Portelas, Y.Yoan Mollard and P.-Y.Pierre-Yves Oudeyer. Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning.Journal of Machine Learning ResearchApril 2022
  HAL
• 39 articleG.Grgur Kovač, A.Adrien Laversanne-Finot and P.-Y.Pierre-Yves Oudeyer. GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning.IEEE Transactions on Cognitive and Developmental Systems2022
  HALDOIback to textback to text
• 40 articleA.Audrey Landuran, H.Helene Sauzeon, C.Charles Consel and B.Bernard N'Kaoua. Evaluation of a smart home platform for adults with Down syndrome.Assistive Technology: The Offical Journal of RESNAMay 2022
  HALDOI
• 41 articleC.Cécile Mazon, B.Benjamin Clément, D.Didier Roy, P.-Y.Pierre-Yves Oudeyer and H.Hélène Sauzéon. Pilot study of an intervention based on an intelligent tutoring system (ITS) for instructing mathematical skills of students with ASD and/or ID.Education and Information Technologies2022
  HALDOIback to textback to text
• 42 articleÉ.Éric Meyer, J.Jérôme Cuadrado, J.Jenna Maire and G.Grégory Michel. Semantic long-term memory dysmnesia or diffuse academic difficulties: A clinical case.Annales Médico-Psychologiques, Revue Psychiatrique1806June 2022, 579-587
  HALDOI
• 43 articleS.Sébastien Mick, A.Arnaud Badets, P.-Y.Pierre-Yves Oudeyer, D.Daniel Cattaert and A.Aymar de Rugy. Biological Plausibility of Arm Postures Influences the Controllability of Robotic Arm Teleoperation.Human Factors642March 2022, 372-384
  HALDOI
• 44 articleM.Marion Pech, A.Antoine Gbessemehlan, L.Lucile Dupuy, H.Hélène Sauzéon, S.Stéphane Lafitte, P.Philippe Bachelet, H.Hélène Amieva and K.Karine Pérès. Lessons Learned From the SoBeezy Program for Older Adults During the COVID-19 Pandemic: Experimentation and Evaluation.JMIR Formative Research611November 2022
  HALDOI
• 45 articleM.Mathieu Ribolet and B.Benjamin Clément. Des péristyles corinthiens dans les domus de Vienna : note sur des fragments d’architecture récemment découverts à Sainte-Colombe (69).Revue archéologique de Narbonnaise53-20202022, 211-228
  HAL
• 46 articleH.Hélène Sauzéon, A.Arlette Edjolo, H.Hélène Amieva, C.Charles Consel and K.Karine Pérès. Effectiveness of an Ambient Assisted Living (HomeAssist) Platform for Supporting Aging in Place of Older Adults With Frailty: Protocol for a Quasi-Experimental Study.JMIR Research Protocols1110October 2022, e33351
  HALDOI

International peer-reviewed conferences

• 47 inproceedingsT.Thomas Carta, S.Sylvain Lamprier, P.-Y.Pierre-Yves Oudeyer and O.Olivier Sigaud. EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL.NeurIPS 2022 - Thirty-sixth Conference on Neural Information Processing SystemsNouvelle-Orléans, United StatesNovember 2022
  HALback to text
• 48 inproceedingsY.Yoann Lemesle, M.Masataka Sawayama, G.Guillermo Valle-Perez, M.Maxime Adolphe, H.Hélène Sauzéon and P.-Y.Pierre-Yves Oudeyer. Language-biased image classification: evaluation based on semantic representations.International Conference on Learning Representations (ICLR) 2022Online (virtuel), FranceApril 2022
  HALback to textback to text
• 49 inproceedingsJ.-L.Jean-Luc Schwartz, M.-L.Marie-Lou Barnaud, P.Pierre Bessière, M.-A.Marc-Antoine Georges, R.Raphaël Laurent, C.Clément Moulin-Frier, M.Mamady Nabé, J.-F.Jean-François Patri, P.Pascal Perrier and J.Julien Diard. COSMO : un modèle bayésien des fondements sensorimoteurs de la perception et de la production de la parole.JEP 2022 - 34e Journées d'Études sur la ParoleNoirmoutier-en-l'Ile, FranceJune 2022
  HAL
• 50 inproceedingsJ.Julius Taylor, E.Eleni Nisioti and C.Clément Moulin-Frier. Socially Supervised Representation Learning: the Role of Subjectivity in Learning Efficient Representations.Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems(AAMAS 2022)International Conference on Autonomous Agents and Multi-Agent SystemsAuckland, New ZealandMay 2022
  HALback to text
• 51 inproceedingsK.Kai Zhang, E.Eric Lucet, J.Julien Alexandre Dit Sandretto, S.Selma Kchir and D.David Filliat. Task and motion planning methods: applications and limitations.ICINCO 2022 - 19th International Conference on Informatics in Control, Automation and RoboticsProceedings of the 19th International Conference on Informatics in Control, Automation and Robotics - ICINCOLisbonne, PortugalSCITEPRESS - Science and Technology PublicationsJuly 2022, 476-483
  HALDOI

National peer-reviewed Conferences

• 52 inproceedingsI.Isabeau Saint-Supery, C.Cécile Mazon, E.Eric Meyer and H.Hélène Sauzéon. Design of an application to support co-education for the school inclusion of students with Autism Spectrum Disorders (ASD).ISATT 2022 - L'éthique inclusive comme nouvel horizon éducatif pour les enseignants et pour l'enseignementMérignac, FranceOctober 2022
  HAL

Conferences without proceedings

• 53 inproceedingsP.Paul Barde, T.Tristan Karch, D.Derek Nowrouzezahrai, C.Clément Moulin-Frier, C.Christopher Pal and P.-Y.Pierre-Yves Oudeyer. Learning to Guide and to Be Guided in the Architect-Builder Problem.International Conference on Learning RepresentationsVirtual, FranceApril 2022
  HALback to textback to text
• 54 inproceedingsI.-J.Iou-Jen Liu, X.Xingdi Yuan, M.-A.Marc-Alexandre Côté, P.-Y.Pierre-Yves Oudeyer and A. G.Alexander G Schwing. Asking for Knowledge : Training RL Agents to Query External Knowledge Using Language.ICML 2022 - 39th International Conference on Machine LearningBaltimore, United StatesJuly 2022
  HALback to text
• 55 inproceedingsE.Eleni Nisioti and C.Clément Moulin-Frier. Plasticity and evolvability under environmental variability: the joint role of fitness-based selection and niche-limited competition.GECCO 2022 - The Genetic and Evolutionary Computation ConferenceBoston / Hybrid, United StatesJuly 2022
  HALback to textback to text
• 56 inproceedingsM.Mehdi Zadem, S.Sergio Mover, S. M.Sao Mai Nguyen and S.Sylvie Putot. Towards Automata-Based Abstraction of Goals in Hierarchical Reinforcement Learning.Intrinsically Motivated Open-ended Learning IMOL 2022Tübingen, GermanyApril 2022
  HAL

Scientific book chapters

• 57 inbookC.Cécile Mazon and H.Hélène Sauzéon. Use of mobile technologies with children with ASD.Autisme et usages du numériques en éducation2022
  HALback to text
• 58 inbookH.Hélène Sauzéon and L.Lucile Dupuy. Assistances numériques domiciliaires pour les personnes âgées fragiles : Etudes de conception et d’évaluation pilote d’une technologie ambiante d’assistance domiciliaire basée sur l’orchestration d’objets connectés..Neuropsychologie Clinique et TechnologiesDe Boeck superieurApril 2022, 480
  HAL
• 59 inbookA.Alexandr Ten, P.-Y.Pierre-Yves Oudeyer and C.Clément Moulin-Frier. Curiosity-driven exploration: Diversity of mechanisms and functions.The Drive for Knowledge: The Science of Human Information Seeking2022
  HALDOI

Doctoral dissertations and habilitation theses

• 60 thesisC.Clément Moulin-Frier. The Ecology of Open-Ended Skill Acquisition: Computational framework and experiments on the interactions between environmental, adaptive, multi-agent and cultural dynamics.Université de Bordeaux (UB)December 2022
  HALback to textback to textback to text
• 61 thesisR.Rémy Portelas. Automatic Curriculum Learning for Developmental Machine Learners.Université de BordeauxFebruary 2022
  HALback to text
• 62 thesisA.Alexandr Ten. The Role of Progress-Based Intrinsic Motivation in Learning : Evidence from Human Behavior and Future Directions.Université de BordeauxApril 2022
  HALback to textback to text

Reports & preprints

• 63 miscR.Rania Abdelghani, Y.-H.Yen-Hsiang Wang, X.Xingdi Yuan, T.Tong Wang, H.Hélène Sauzéon and P.-Y.Pierre-Yves Oudeyer. GPT-3-driven pedagogical agents for training children's curious question-asking skills.December 2022
  HALback to text
• 64 reportA.Anne Lehmans, E.Estelle Blanquet, F.Fabrice Vandebrouck, G.Grégory Train, V.Vincent Liquete, C.Camille Capelle, C.Clément Dussarps, A.Anne Cordier, C.Coulibaly Mohamed, K.Karel Soumagnac-Colin, O.Olivier Ly, P.Patxi Laborde-Zubieta, G.Grégoire Passault, B.Benjamin Clément, T.Thibault Desprez, S.Stephanie Noirpoudre, D.Didier Roy, L.Laurent Gallon, A.Angel Abénia, S.Samia Bachir, V.Vanea Chiprianov, T.Timothée Duron, S.Sébastien Gouleau and J.Julien Allali. Rapport scientifique final du projet E-Fran PERSEVERONS.Université de Bordeaux2022
  HAL
• 65 miscY.Yoann Lemesle, T.Tristan Karch, R.Romain Laroche, C.Clément Moulin-Frier and P.-Y.Pierre-Yves Oudeyer. Emergence of Shared Sensory-motor Graphical Language from Visual Input.December 2022
  HAL
• 66 miscE.Elías Masquil, G.Gautier Hamon, E.Eleni Nisioti and C.Clément Moulin-Frier. Intrinsically-Motivated Goal-Conditioned Reinforcement Learning in Multi-Agent Environments.December 2022
  HALback to textback to text
• 67 miscE.Eleni Nisioti, M.Mateo Mahaut, P.-Y.Pierre-Yves Oudeyer, I.Ida Momennejad and C.Clément Moulin-Frier. Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS.July 2022
  HALback to textback to text
• 68 miscE.Erwan Plantec, G.Gautier Hamon, M.Mayalen Etcheverry, P.-Y.Pierre-Yves Oudeyer, C.Clément Moulin-Frier and B.-C. W.Bert Wang-Chak Chan. Flow Lenia: Mass conservation for the study of virtual creatures in continuous cellular automata.December 2022
  HALback to textback to text
• 69 miscI.Isabeau Saint-Supery, C.Cécile Mazon, E.Eric Meyer and H.Hélène Sauzéon. ToGather, an interactive website for the stakeholders of school inclusion of children with ASD: an iterative design including user testing.2022
  HALback to text
• 70 miscX.Xingdi Yuan, T.Tong Wang, Y.-H.Yen-Hsiang Wang, E.Emery Fine, R.Rania Abdelghani, P.Pauline Lucas, H.Hélène Sauzéon and P.-Y.Pierre-Yves Oudeyer. Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation.December 2022
  HAL

Other scientific publications

• 71 miscR.Rania Abdelghani, P.-Y.Pierre-Yves Oudeyer, E.Edith Law, C.Catherine de Vulpillières and H.Hélène Sauzéon. Conversational agents for fostering curiosity-driven learning.October 2022
  HAL
• 72 inproceedingsE.Eric Meyer, I.Isabeau Saint-Supery, M.Miliana Rahouadj, C.Caroline Simonpietri, C.Cécile Mazon and H.Hélène Sauzéon. User experience assessment of an interactive website to support family-professional and interprofessional collaboration for the schooling of middle school pupils with ASD.1er Colloque international du GNCRA 2022Lyon, FranceMay 2022
  HAL
• 73 inproceedingsI.Isabeau Saint-Supery, K.Kattalin Etchegoyhen, A.Annouck Amestoy, M.Manuel Bouvard, C.Charles Consel, H.Hélène Sauzéon and C.Cécile Mazon. User-centered design of the ToGather web application: a tool to support family-professional and interprofessional collaboration for the education of middle school students with ASD.1er Colloque international du GNCRA 2022Lyon, FranceMay 2022
  HAL
• 74 miscI.Isabeau Saint-Supery. ToGather, an interactive website for the stakeholders of school inclusion of children with ASD: an iterative design including user testing.Bordeaux, FranceNovember 2022
  HAL
• 75 miscM.Masataka Sawayama, Y.Yoann Lemesle and P.-Y.Pierre-Yves Oudeyer. Watching artificial intelligence through the lens of cognitive science methodologies.July 2022
  HALback to text
• 76 inproceedingsY.Yadurshana Sivashankar, H.Hélène Sauzéon and M. A.Myra A. Fernandes. Active and Visually-Guided Navigation Benefit Route Memory.Psychonomic Society 2022 - 63rd Annual MeetingBoston, United StatesNovember 2022
  HAL

Scientific popularization

• 77 misc R.Rania Abdelghani, P.-Y.Pierre-Yves Oudeyer, C.Catherine de Vulpillières and H.Hélène Sauzéon. Curiosité, apprentissage et numérique : Que dit la recherche ? September 2022
  HAL
• 78 miscG.Gautier Hamon, M.Mayalen Etcheverry, B.-C. W.Bert Wang-Chak Chan, C.Clément Moulin-Frier and P.-Y.Pierre-Yves Oudeyer. Learning Sensorimotor Agency in Cellular Automata.January 2022
  HALback to text
• 79 articleE.Eric Meyer, I.Isabeau Saint-Supery, H.Hélène Sauzéon and C.Cécile Mazon. L'inclusion scolaire des élèves avec TSA : Prospection et analyse documentaire pour co-concevoir et valider des outils numériques favorables à l'inclusion scolaire en milieu ordinaire des élèves avec TSA.Fondation Internationale de la Recherche Appliquée sur le Handicap392022, 55
  HAL