7.1.1 highway-env

• Name: An environment for autonomous driving decision-making
• Keywords: Generic modeling environment, Simulation, Autonomous Cars, Artificial intelligence
• Functional Description: The environment is composed of several variants, each of which corresponds to driving scenes: highway, roundabout, intersection, merge, parking, etc. The road network is described by a graph, and is then populated with simulated vehicles. Vehicle kinematics follows a simple Bicycle model, and their behavior is determined by models derived from road traffic simulation literature. The ego-vehicle has access to a description of the scene through several types of observations, and its behavior is controlled through an action space, either discrete (change of lanes, of cruising speed) or continuous ( accelerator pedal, steering wheel angle). The objective function to maximize is also described by the environment and may vary depending on the task to be solved. The interface of the library is inherited from the standard defined by OpenAI Gym, consisting of four main methods: gym.make(id), env.step(action), env.reset(), and env.render().
• URL: https://github.com/eleurent/highway-env
• Author: Edouard Leurent
• Contact: Edouard Leurent

7.1.2 rlberry

• Keywords: Reinforcement learning, Simulation, Artificial intelligence
• Functional Description: rlberry is a reinforcement learning (RL) library in Python for research and education. The library provides implementations of several RL agents for you to use as a starting point or as baselines, provides a set of benchmark environments, very useful to debug and challenge your algorithms, handles all random seeds for you, ensuring reproducibility of your results, and is fully compatible with several commonly used RL libraries like OpenAI gym and Stable Baselines.
• URL: https://github.com/rlberry-py/rlberry
• Contact: Omar Darwiche Domingues

7.1.3 justicia

• Name: Justicia: A Stochastic SAT Approach to Formally Verify Fairness
• Keywords: Fairness, Machine learning, Verification, Fairness Verification, Fair and ethical machine learning, Formal methods
• Functional Description: justicia is a fairness verifier written in Python. The library provides a stochastic SAT encoding of multiple fairness definitions and fair ML algorithms. justicia then further verifies the fairness metric achieved by the corresponding ML algorithm. It is now available as an official Python package and can be installed using pip.
• News of the Year: 2020
• URL: https://www.github.com/meelgroup/justicia
• Contact: Debabrota Basu
• Participant: Bishwamittra Ghosh
• Partner: National University of Singapore

Statistical efficiency of Thompson sampling for combinatorial semi-bandits

40

We investigate stochastic combinatorial multi-armed bandit with semi-bandit feedback (CMAB). In CMAB, the question of the existence of an efficient policy with an optimal asymptotic regret (up to a factor poly-logarithmic with the action size) is still open for many families of distributions, including mutually independent outcomes, and more generally the multivariate sub-Gaussian family. We propose to answer the above question for these two families by analyzing variants of the Combinatorial Thompson Sampling policy (CTS). For mutually independent outcomes in $[0,1]$, we propose a tight analysis of CTS using Beta priors. We then look at the more general setting of multivariate sub-Gaussian outcomes and propose a tight analysis of CTS using Gaussian priors. This last result gives us an alternative to the Efficient Sampling for Combinatorial Bandit policy (ESCB), which, although optimal, is not computationally efficient.

Sub-sampling for Efficient Non-Parametric Bandit Exploration

19

In this paper we propose the first multi-armed bandit algorithm based on re-sampling that achieves asymptotically optimal regret simultaneously for different families of arms (namely Bernoulli, Gaussian and Poisson distributions). Unlike Thompson Sampling which requires to specify a different prior to be optimal in each case, our proposal RB-SDA does not need any distribution-dependent tuning. RB-SDA belongs to the family of Sub-sampling Duelling Algorithms (SDA) which combines the sub-sampling idea first used by the BESA 61 and SSMC 63 algorithms with different sub-sampling schemes. In particular, RB-SDA uses Random Block sampling. We perform an experimental study assessing the flexibility and robustness of this promising novel approach for exploration in bandit models.

Monte-Carlo Graph Search: the Value of Merging Similar States

38

We consider the problem of planning in a Markov Decision Process (MDP) with a generative model and limited computational budget. Despite the underlying MDP transitions having a graph structure, the popular Monte-Carlo Tree Search algorithms such as UCT rely on a tree structure to represent their value estimates. That is, they do not identify together two similar states reached via different trajectories and represented in separate branches of the tree. In this work, we propose a graph-based planning algorithm, which takes into account this state similarity. In our analysis, we provide a regret bound that depends on a novel problem-dependent measure of difficulty, which improves on the original tree-based bound in MDPs where the trajectories overlap, and recovers it otherwise. Then, we show that this methodology can be adapted to existing planning algorithms that deal with stochastic systems. Finally, numerical simulations illustrate the benefits of our approach.

Planning in Markov Decision Processes with Gap-Dependent Sample Complexity

33

We propose MDP-GapE, a new trajectory-based Monte-Carlo Tree Search algorithm for planning in a Markov Decision Process in which transitions have a finite support. We prove an upper bound on the number of calls to the generative models needed for MDP-GapE to identify a near-optimal action with high probability. This problem-dependent sample complexity result is expressed in terms of the sub-optimality gaps of the state-action pairs that are visited during exploration. Our experiments reveal that MDP-GapE is also effective in practice, in contrast with other algorithms with sample complexity guarantees in the fixed-confidence setting, that are mostly theoretical.

Spectral bandits

13

Smooth functions on graphs have wide applications in manifold and semi-supervised learning. In this work, we study a bandit problem where the payoffs of arms are smooth on a graph. This framework is suitable for solving online learning problems that involve graphs, such as content-based recommendation. In this problem, each item we can recommend is a node of an undirected graph and its expected rating is similar to the one of its neighbors. The goal is to recommend items that have high expected ratings. We aim for the algorithms where the cumulative regret with respect to the optimal policy would not scale poorly with the number of nodes. In particular, we introduce the notion of an effective dimension, which is small in real-world graphs, and propose three algorithms for solving our problem that scale linearly and sublinearly in this dimension. Our experiments on contentrecommendation problem show that a good estimator of user preferences for thousands of items can be learned from just tens of node evaluations.

Solving Bernoulli Rank-One Bandits with Unimodal Thompson Sampling

46

Stochastic Rank-One Bandits 68, 69 are a simple framework for regret minimization problems over rank-one matrices of arms. The initially proposed algorithms are proved to have logarithmic regret, but do not match the existing lower bound for this problem. We close this gap by first proving that rank-one bandits are a particular instance of unimodal bandits, and then providing a new analysis of Unimodal Thompson Sampling (UTS), initially proposed by Paladino et al. 70. We prove an asymptotically optimal regret bound on the frequentist regret of UTS and we support our claims with simulations showing the significant improvement of our method compared to the state-of-the-art.

A Practical Algorithm for Multiplayer Bandits when Arm Means Vary Among Players

20

We study a multiplayer stochastic multi-armed bandit problem in which players cannot communicate, and if two or more players pull the same arm,a collision occurs and the involved players receive zero reward. We consider the challenging heterogeneous setting, in which different arms may have different means for different players, and propose a new and efficient algorithm that combines the idea of leveraging forced collisions for implicitcommunication and that of performing matching eliminations. We present a finite-time analysis of our algorithm, giving the first sublinear minimaxregret bound for this problem, and prove that if the optimal assignment of players to arms is unique, our algorithm attains the optimal $O(ln(T\left)\right)$ regret, solving an open question raised at NeurIPS 2018 by Bistritz and Leshem 62.

Budgeted online influence maximization

41

We introduce a new budgeted framework for on-line influence maximization, considering the total cost of an advertising campaign instead of the common cardinality constraint on a chosen influ-encer set. Our approach models better the real-world setting where the cost of influencers varies and advertizers want to find the best value for their overall social advertising budget. We propose an algorithm assuming an independent cascade diffusion model and edge-level semi-bandit feedback, and provide both theoretical and experimental results. Our analysis is also valid for the cardinality-constraint setting and improves the state of the art regret bound in this case.

Fixed-confidence guarantees for Bayesian best-arm identification

45

We investigate and provide new insights on the sampling rule called Top-Two Thompson Sampling (TTTS). In particular, we justify its use for fixed-confidence best-arm identification. We further propose a variant of TTTS called Top-Two Transportation Cost (T3C), which disposes of the computational burden of TTTS. As our main contribution, we provide the first sample complexity analysis of TTTS and T3C when coupled with a very natural Bayesian stopping rule, for bandits with Gaussian rewards, solving one of the open questions raised by Russo 71. We also provide new posterior convergence results for TTTS under two models that are commonly used in practice: bandits with Gaussian and Bernoulli rewards and conjugate priors.

The Influence of Shape Constraints on the Thresholding Bandit Problem

21

We investigate the stochastic Thresholding Bandit problem (TBP) under several shape constraints. On top of (i) the vanilla, unstructured TBP, we consider the case where (ii) the sequence of arm's means ${\left({\mu }_k\right)}_k$ is monotonically increasing MTBP, (iii) the case where ${\left({\mu }_k\right)}_k$ is unimodal UTBP and (iv) the case where ${\left({\mu }_k\right)}_k$ is concave CTBP. In the TBP problem the aim is to output, at the end of the sequential game, the set of arms whose means are above a given threshold. The regret is the highest gap between a misclassified arm and the threshold. In the fixed budget setting, we provide problem independent minimax rates for the expected regret in all settings, as well as associated algorithms. We prove that the minimax rates for the regret are (i) $\sqrt{log\left(K\right)K/T}$ for TBP, (ii) $\sqrt{log\left(K\right)/T}$ for MTBP, (iii) $\sqrt{K/T}$ for UTBP and (iv) $\sqrt{loglogK/T}$ for CTBP, where $K$ is the number of arms and $T$ is the budget. These rates demonstrate that the dependence on $K$ of the minimax regret varies significantly depending on the shape constraint. This highlights the fact that the shape constraints modify fundamentally the nature of the TBP problem to the other.

Covariance-adapting algorithm for semi-bandits with application to sparse outcomes

42

We investigate stochastic combinatorial semi-bandits, where the entire joint distribution of outcomes impacts the complexity of the problem instance (unlike in the standard bandits). Typical distributions considered depend on specific parameter values, whose prior knowledge is required in theory but quite difficult to estimate in practice; an example is the commonly assumed sub-Gaussian family. We alleviate this issue by instead considering a new general family of sub-exponential distributions, which contains bounded and Gaussian ones. We prove a new lower bound on the regret on this family, that is parameterized by the unknown covariance matrix, a tighter quantity than the sub-Gaussian matrix. We then construct an algorithm that uses covariance estimates, and provide a tight asymptotic analysis of the regret. Finally, we apply and extend our results to the family of sparse outcomes, which has applications in many recommender systems.

Gamification of pure exploration for linear bandits

24

We investigate an active pure-exploration setting, that includes best-arm identification, in the context of linear stochastic bandits. While asymptotically optimal algorithms exist for standard multi-arm bandits, the existence of such algorithms for the best-arm identification in linear bandits has been elusive despite several attempts to address it. First, we provide a thorough comparison and new insight over different notions of optimality in the linear case, including $G$-optimality, transductive optimality from optimal experimental designand asymptotic optimality. Second, we design the first asymptotically optimal algorithm for fixed-confidence pure exploration in linear bandits. As a consequence, our algorithm naturally bypasses the pitfall caused by a simple but difficult instance, that most prior algorithms had to be engineered to deal with explicitly. Finally, we avoid the need to fully solve an optimal design problem by providing an approach that entails an efficient implementation.

Stochastic bandits with vector losses: Minimizing ${\ell }^{\infty }$-norm of relative losses

60

Multi-armed bandits are widely applied in scenarios like recommender systems, for which the goal is to maximize the click rate. However, more factors should be considered, e.g., user stickiness, user growth rate, user experience assessment, etc. In this paper, we model this situation as a problem of K-armed bandit with multiple losses. We define relative loss vector of an arm where the i-th entry compares the arm and the optimal arm with respect to the i-th loss. We study two goals: (a) finding the arm with the minimum ${\ell }^{\infty }$-norm of relative losses with a given confidence level (which refers to fixed-confidence best-arm identification); (b) minimizing the ${\ell }^{\infty }$-norm of cumulative relative losses (which refers to regret minimization). For goal (a), we derive a problem-dependent sample complexity lower bound and discuss how to achieve matching algorithms. For goal (b), we provide a regret lower bound of $\Omega \left(T^{2/3}\right)$ and provide a matching algorithm.

Efficient Change-Point Detection for Tackling Piecewise-Stationary Bandits

52

We introduce GLR-klUCB, a novel algorithm for the piecewise i.i.d. non-stationary bandit problem with bounded rewards. This algorithm combines an efficient bandit algorithm, kl-UCB, with an efficient, parameter-free, changepoint detector, the Bernoulli Generalized Likelihood Ratio Test, for which we provide new theoretical guarantees of independent interest. Unlike previous non-stationary bandit algorithms using a change-point detector, GLR-klUCB does not need to be calibrated based on prior knowledge on the arms' means. We prove that this algorithm can attain a $O\left(\sqrt{TA{{\rm Y}}_{T}log\left(T\right)}\right)$ regret in $T$ rounds on some “easy” instances, where A is the number of arms and ${{\rm Y}}_T$ the number of change-points, without prior knowledge of ${{\rm Y}}_T$. In contrast with recently proposed algorithms that are agnostic to ${{\rm Y}}_T$, we perform a numerical study showing that GLR-klUCB is also very efficient in practice, beyond easy instances.

Adversarial Attacks on Linear Contextual Bandits

55

Contextual bandit algorithms are applied in a wide range of domains, from advertising to recommender systems, from clinical trials to education. In many of these domains, malicious agents may have incentives to attack the bandit algorithm to induce it to perform a desired behavior. For instance, an unscrupulous ad publisher may try to increase their own revenue at the expense of the advertisers; a seller may want to increase the exposure of their products, or thwart a competitor's advertising campaign. In this paper, we study several attack scenarios and show that a malicious agent can force a linear contextual bandit algorithm to pull any desired arm $T-o\left(T\right)$ times over a horizon of $T$ steps, while applying adversarial modifications to either rewards or contexts that only grow logarithmically as $O(logT)$. We also investigate the case when a malicious agent is interested in affecting the behavior of the bandit algorithm in a single context (e.g., a specific user). We first provide sufficient conditions for the feasibility of the attack and we then propose an efficient algorithm to perform the attack. We validate our theoretical results on experiments performed on both synthetic and real-world datasets.

Forced-exploration free Strategies for Unimodal Bandits

58

We consider a multi-armed bandit problem specified by a set of Gaussian or Bernoulli distributions endowed with a unimodal structure. Although this problem has been addressed in the literature 64, the state-of-the-art algorithms for such structure make appear a forced-exploration mechanism. We introduce IMED-UB, the first forced-exploration free strategy that exploits the unimodal-structure, by adapting to this setting the Indexed Minimum Empirical Divergence (IMED) strategy introduced by Honda and Takemura 66. This strategy is proven optimal. We then derive KLUCB-UB, a KLUCB version of IMED-UB, which is also proven optimal. Owing to our proof technique, we are further able to provide a concise finite-time analysis of both strategies in an unified way. Numerical experiments show that both IMED-UB and KLUCB-UB perform similarly in practice and outperform the state-of-the-art algorithms.

Optimal Strategies for Graph-Structured Bandits

59

We study a structured variant of the multi-armed bandit problem specified by a set of Bernoulli distributions $\nu ={\left({\nu }_{a,b}\right)}_{a\in 𝒜,b\in ℬ}$ with means ${\left({\mu }_{a,b}\right)}_{a\in 𝒜,b\in ℬ}\in {[0,1]}^{𝒜\times ℬ}$ and by a given weight matrix $\omega ={\left({\omega }_{b,b^{\text{'}}}\right)}_{b,b^{\text{'}}\in ℬ}$, where $𝒜$ is a finite set of arms and $ℬ$ is a finite set of users. The weight matrix $\omega$ is such that for any two users $b,b^{\text{'}}\in ℬ,{max}_{a\in 𝒜}|{\mu }_{a,b}-{\mu }_{a,b^{\text{'}}}|\le {\omega }_{b,b^{\text{'}}}$. This formulation is flexible enough to capture various situations, from highly-structured scenarios ($\omega \in {\{0,1\}}^{ℬ\times ℬ}$) to fully unstructured setups ($\omega \equiv 1$).We consider two scenarios depending on whether the learner chooses only the actions to sample rewards from or both users and actions. We first derive problem-dependent lower bounds on the regret for this generic graph-structure that involves a structure dependent linear programming problem. Second, we adapt to this setting the Indexed Minimum Empirical Divergence (IMED) algorithm introduced by Honda and Takemura (2015), and introduce the IMED-GS${}^{☆}$ algorithm. Interestingly, IMED-GS${}^{☆}$ does not require computing the solution of the linear programming problem more than about $log\left(T\right)$ times after $T$ steps, while being provably asymptotically optimal. Also, unlike existing bandit strategies designed for other popular structures, IMED-GS${}^{☆}$ does not resort to an explicit forced exploration scheme and only makes use of local counts of empirical events. We finally provide numerical illustration of our results that confirm the performance of IMED-GS${}^{☆}$.

On Multi-Armed Bandit Designs for Dose-Finding Trials

51

We study the problem of finding the optimal dosage in early stage clinical trials through the multi-armed bandit lens. We advocate the use of the Thompson Sampling principle, a flexible algorithm that can accommodate different types of monotonicity assumptions on the toxicity and efficacy of the doses. For the simplest version of Thompson Sampling, based on a uniform prior distribution for each dose, we provide finite-time upper bounds on the number of sub-optimal dose selections, which is unprecedented for dose-finding algorithms. Through a large simulation study, we then show that variants of Thompson Sampling based on more sophisticated prior distributions outperform state-of-the-art dose identification algorithms in different types of dose-finding studies that occur in phase I or phase I/II trials.

Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL

3

In reinforcement learning, policy gradient algorithms optimize the policy directly and rely on sampling efficiently an environment. Nevertheless, while most sampling procedures are based on direct policy sampling, self-performance measures could be used to improve such sampling prior to each policy update. Following this line of thought, we introduce SAUNA, a method where non-informative transitions are rejected from the gradient update. The level of information is estimated according to the fraction of variance explained by the value function: a measure of the discrepancy between V and the empirical returns. In this work, we use this metric to select samples that are useful to learn from, and we demonstrate that this selection can significantly improve the performance of policy gradient methods. In this paper: (a) We define SAUNA's metric and introduce its method to filter transitions. (b) We conduct experiments on a set of benchmark continuous control problems. SAUNA significantly improves performance. (c) We investigate how SAUNA reliably selects samples with the most positive impact on learning and study its improvement on both performance and sample efficiency.

Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning

34

Bayesian Reinforcement Learning (BRL) offers a decision-theoretic solution to the reinforcement learning problem. While “model-based” BRL algorithms have focused either on maintaining a posterior distribution on models, BRL “model-free” methods try to estimate value function distributions but make strong implicit assumptions or approximations. We describe a novel Bayesian framework, inferential induction, for correctly inferring value function distributions from data, which leads to a new family of BRL algorithms. We design an algorithm, Bayesian Backwards Induction (BBI), with this framework. We experimentally demonstrate that BBI is competitive with the state of the art. However, its advantage relative to existing BRL model-free methods is not as great as we have expected, particularly when the additional computational burden is taken into account.

Tightening Exploration in Upper Confidence Reinforcement Learning

8

The upper confidence reinforcement learning (UCRL2) algorithm introduced in 67 is a popular method to perform regret minimization in unknown discrete Markov Decision Processes under the average-reward criterion. Despite its nice and generic theoretical regret guarantees, this algorithm and its variants have remained until now mostly theoretical as numerical experiments in simple environments exhibit long burn-in phases before the learning takes place. In pursuit of practical efficiency, we present UCRL3, following the lines of UCRL2, but with two key modifications: First, it uses state-of-the-art time-uniform concentration inequalities to compute confidence sets on the reward and (component-wise) transition distributions for each state-action pair. Furthermore, to tighten exploration, it uses an adap-tive computation of the support of each transition distribution, which in turn enables us to revisit the extended value iteration procedure of UCRL2 to optimize over distributions with reduced support by disregarding low probability transitions, while still ensuring near-optimism. We demonstrate, through numerical experiments in standard environments, that reducing exploration this way yields a substantial numerical improvement compared to UCRL2 and its variants. On the theoretical side, these key modifications enable us to derive a regret bound for UCRL3 improving on UCRL2, that for the first time makes appear notions of local diameter and local effective support, thanks to variance-aware concentration bounds.

“I'm sorry Dave, I'm afraid I can't do that” Deep Q-Learning From Forbidden Actions

43

The use of Reinforcement Learning (RL) is still restricted to simulation or to enhance human-operated systems through recommendations. Real-world environments (e.g. industrial robots or power grids) are generally designed with safety constraints in mind implemented in the shape of valid actions masks or contingency controllers. For example, the range of motion and the angles of the motors of a robot can be limited to physical boundaries. Violating constraints thus results in rejected actions or entering in a safe mode driven by an external controller, making RL agents incapable of learning from their mistakes. In this paper, we propose a simple modification of a state-of-the-art deep RL algorithm (DQN), enabling learning from forbidden actions. To do so, the standard Q-learning update is enhanced with an extra safety loss inspired by structured classification. We empirically show that it reduces the number of hit constraints during the learning phase and accelerates convergence to near-optimal policies compared to using standard DQN. Experiments are done on a Visual Grid World Environment and Text-World domain.

A Machine of Few Words Interactive Speaker Recognition with Reinforcement Learning

44

Speaker recognition is a well known and studied task in the speech processing domain. It has many applications, either for security or speaker adaptation of personal devices. In this paper, we present a new paradigm for automatic speaker recognition that we call Interactive Speaker Recognition (ISR). In this paradigm, the recognition system aims to incrementally build a representation of the speakers by requesting personalized utterances to be spoken in contrast to the standard text-dependent or text-independent schemes. To do so, we cast the speaker recognition task into a sequential decision-making problem that we solve with Reinforcement Learning. Using a standard dataset, we show that our method achieves excellent performance while using little speech signal amounts. This method could also be applied as an utterance selection mechanism for building speech synthesis systems.

HIGhER: Improving instruction following with Hindsight Generation for Experience Replay

22

Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios. This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases. In this paper, we propose an orthogonal approach called Hindsight Generation for Experience Replay (HIGhER) that extends the Hindsight Experience Replay approach to the language-conditioned policy setting. Whenever the agent does not fulfill its instruction, HIGhER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, HIGhER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We show the efficiency of our approach in the BabyAI environment, and demonstrate how it complements other instruction following methods.

Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications

57

In this paper, we deepen the analysis of continuous time Fictitious Play learning algorithm to the consideration of various finite state Mean Field Game settings (finite horizon, $\gamma$-discounted), allowing in particular for the introduction of an additional common noise. We first present a theoretical convergence analysis of the continuous time Fictitious Play process and prove that the induced exploitability decreases at a rate $O\left(\frac{1}{t}\right)$. Such analysis emphasizes the use of exploitability as a relevant metric for evaluating the convergence towards a Nash equilibrium in the context of Mean Field Games. These theoretical contributions are supported by numerical experiments provided in either model-based or model-free settings. We provide hereby for the first time converging learning dynamics for Mean Field Games in the presence of common noise.

Regret bounds for kernel-based reinforcement learning

53

We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning problems whose state-action space is endowed with a metric. We introduce Kernel-UCBVI, a model-based optimistic algorithm that leverages the smoothness of the MDP and a non-parametric kernel estimator of the rewards and transitions to efficiently balance exploration and exploitation. Unlike existing approaches with regret guarantees, it does not use any kind of partitioning of the state-action space. For problems with K episodes and horizon H, we provide a regret bound of $O\left(H^3{K}^{max(12,\frac{2d}{2d+1})}\right)$, where d is the covering dimension of the joint state-action space. We empirically validate Kernel-UCBVI on discrete and continuous MDPs.

CopyCAT: Taking Control of Neural Policies with Constant Attacks

32

We propose a new perspective on adversarial attacks against deep reinforcement learning agents. Our main contribution is CopyCAT, a targeted attack able to consistently lure an agent into following an outsider's policy. It is pre-computed, therefore fast inferred, and could thus be usable in a real-time scenario. We show its effectiveness on Atari 2600 games in the novel read-only setting. In this setting, the adversary cannot directly modify the agent's state – its representation of the environment – but can only attack the agent's observation – its perception of the environment. Directly modifying the agent's state would require a write-access to the agent's inner workings and we argue that this assumption is too strong in realistic settings.

Self-Attentional Credit Assignment for Transfer in Reinforcement Learning

25

The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample efficient. Our main contribution is Secret, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.

Robust-Adaptive Control of Linear Systems: beyond Quadratic Costs

36

We consider the problem of robust and adaptive model predictive control (MPC) of a linear system, with unknown parameters that are learned along the way (adaptive), in a critical setting where failures must be prevented (robust). This problem has been studied from different perspectives by different communities. However, the existing theory deals only with the case of quadratic costs (the LQ problem), which limits applications to stabilisation and tracking tasks only. In order to handle more general (non-convex) costs that naturally arise in many practical problems, we carefully select and bring together several tools from different communities, namely non-asymptotic linear regression, recent results in interval prediction, and tree-based planning. Combining and adapting the theoretical guarantees at each layer is non trivial, and we provide the first end-to-end suboptimality analysis for this setting. Interestingly, our analysis naturally adapts to handle many models and combines with a data-driven robust model selection strategy, which enables to relax the modelling assumptions. Last, we strive to preserve tractability at any stage of the method, that we illustrate on two challenging simulated environments.

Robust-Adaptive Interval Predictive Control for Linear Uncertain Systems

37

We consider the problem of stabilization of a linear system, under state and control constraints, and subject to bounded disturbances and unknown parameters in the state matrix. First, using a simple least square solution and available noisy measurements, the set of admissible values for parameters is evaluated. Second, for the estimated set of parameter values and the corresponding linear interval model of the system, two interval predictors are recalled and an unconstrained stabilizing control is designed that uses the predicted intervals. Third, to guarantee the robust constraint satisfaction, a model predictive control algorithm is developed, which is based on solution of an optimization problem posed for the interval predictor. The conditions for recursive feasibility and asymptotic performance are established. Efficiency of the proposed control framework is illustrated by numeric simulations.

The challenge of controlling microgrids in the presence of rare events with Deep Reinforcement Learning

15

The increased penetration of renewable energies and the need to decarbonize the grid come with a lot of challenges. Microgrids, power grids that can operate independently from the main system, are seen as a promising solution. They range from a small building to a neighbourhood or a village. As they co-locate generation, storage and consumption, microgrids are often built with renewable energies. At the same time, because they can be disconnected from the main grid, they can be more resilient and less dependent on central generation. Due to their diversity and distributed nature, advanced metering and control will be necessary to maximize their potential. This paper presents a reinforcement learning algorithm to tackle the energy management of an off-grid microgrid, represented as a Markov Decision Process. The main objective function of the proposed algorithm is to minimize the global operating cost. By nature, rare events occur in physical systems. One of the main contribution of this paper is to demonstrate how to train agents in the presence of rare events. We prove that merging the combined experience replay method with a novel methods called “Memory Counter” unstucks the agent during its learning phase. Compared to baselines, we show that an extended version of Double Deep Q-Network with a priority list of actions into the decision making strategy process lowers significantly the operating cost. Experiments are conducted using two years of real-world data from Ecole Polytechnique in France.

Geometric Deep Reinforcement Learning for Dynamic DAG Scheduling

30

In practice, it is quite common to face combinatorial optimization problems which contain uncertainty along with non-determinism and dynamicity. These three properties call for appropriate algorithms; reinforcement learning (RL) is dealing with them in a very natural way. Today, despite some efforts, most real-life combinatorial optimization problems remain out of the reach of reinforcement learning algorithms. In this paper, we propose a reinforcement learning approach to solve a realistic scheduling problem, and apply it to an algorithm commonly executed in the high performance computing community, the Cholesky factorization. On the contrary to static scheduling, where tasks are assigned to processors in a predetermined ordering before the beginning of the parallel execution, our method is dynamic: task allocations and their execution ordering are decided at runtime, based on the system state and unexpected events, which allows much more flexibility. To do so, our algorithm uses graph neural networks in combination with an actor-critic algorithm (A2C) to build an adaptive representation of the problem on the fly. We show that this approach is competitive with state-of-the-art heuristics used in high-performance computing runtime systems. Moreover, our algorithm does not require an explicit model of the environment, but we demonstrate that extra knowledge can easily be incorporated and improves performance. We also exhibit key properties provided by this RL approach, and study its transfer abilities to other instances.

Interdisciplinary Research in Artificial Intelligence: Challenges and Opportunities

14

The use of artificial intelligence (AI) in a variety of research fields is speeding up multiple digital revolutions, from shifting paradigms in healthcare, precision medicine and wearable sensing, to public services and education offered to the masses around the world, to future cities made optimally efficient by autonomous driving. When a revolution happens, the consequences are not obvious straight away and, to date, there is no uniformly adapted framework to guide AI research to ensure a sustainable societal transition. To answer this need, here we analyze three key challenges to interdisciplinary AI research, and deliver three broad conclusions: 1) future development of AI should not only impact other scientific domains but should also take inspiration and benefit from other fields of science, 2) AI research must be accompanied by decision explainability, dataset bias transparency aswell as development of evaluation methodologies and creation of regulatory agencies to ensure responsibility, and 3) AI education should receive more attention, efforts and innovation from the educational and scientific communities. Our analysis is of interest not only to AI practitioners but also to other researchers and the general public as it offers ways to guide the emerging collaborations and interactions toward the most fruitful outcomes.

International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium

11

We leveraged the largely untapped resource of electronic health record data to address critical clinical and epidemiological questions about Coronavirus Disease 2019 (COVID-19). To do this, we formed an international consortium (4CE) of 96 hospitals across 5 countries (https://www.covidclinical.net/). Contributors utilized the Informatics for Integrating Biology and the Bedside (i2b2) or Observational Medical Outcomes Partnership (OMOP) platforms to map to a common data model. The group focused on comorbidities and temporal changes in key laboratory test values. Harmonized data were analyzed locally and converted to a shared aggregate form for rapid analysis and visualization of regional differences and global commonalities. Data covered 27,584 COVID-19 cases with 187,802 laboratory tests. Case counts and laboratory trajectories were concordant with existing literature. Laboratory tests at the time of diagnosis showed hospital-level differences equivalent to country-level variation across the consortium partners. Despite the limitations of decentralized data generation, we established a framework to capture the trajectory of COVID-19 disease in patients and their response to interventions.

Machine learning applications in drug development

16

Due to the huge amount of biological and medical data available today, along with well-established machine learning algorithms, the design of largely automated drug development pipelines can now be envisioned. These pipelines may guide, or speed up, drug discovery; provide a better understanding of diseases and associated biological phenomena; help planning preclinical wet-lab experiments, and even future clinical trials. This automation of the drug development process might be key to the current issue of low productivity rate that pharmaceutical companies currently face. In this survey, we will particularly focus on two classes of methods: sequential learning and recommender systems, which are active biomedical fields of research.

Restarted Bayesian Online Change-point Detector achieves Optimal Detection Delay

17

In this paper, we consider the problem of sequential change-point detection where both the changepoints and the distributions before and after the change are assumed to be unknown. For this problem of primary importance in statistical and sequential learning theory, we derive a variant of the Bayesian Online Change Point Detector proposed by 65 which is easier to analyze than the original version while keeping its powerful message-passing algorithm. We provide a non-asymptotic analysis of the false-alarm rate and the detection delay that matches the existing lower-bound. We further provide the first explicit high-probability control of the detection delay for such approach. Experiments on synthetic and realworld data show that this proposal outperforms the state-of-art change-point detection strategy, namely the Improved Generalized Likelihood Ratio (Improved GLR) while compares favorably with the original Bayesian Online Change Point Detection strategy.

10.1.1 Inria associate team not involved in an IIL

Associate team “Data Collection for Smart Crop Management” (DC4SCM) has begun in 2020. The partner being in India, the activities have heavily suffered from the covid-19 pandemic.

Scool also participates in the associate team 6PAC with CWI, headed by B. Guedj.

10.1.2 Inria international partners

10.2.1 Visits of international scientists

Prof. Anders Jonsson, University Pompeu Fabra (Spain), spent 1 year in the team on sabbatical, 2019–2020.

10.3.1 Collaborations in European programs, except FP7 and H2020

• Chist-Era Delta, headed by A. Jonsson (University Pompeu Fabra, Spain), local head: É. Kaufmann, 10/2017 – 12/2021.
• EIT Health Covidom Community, with AP-HP, headed by Patrick Jourdain, local head: J.-J. Vie, 4/2020 – 12/2020.

11.1.1 Scientific events: organisation

11.1.2 Scientific events: selection

11.1.3 Journal

11.1.4 Invited talks

Many events were canceled in 2020. Others could not be granted (e.g. O-A. Mailllard was invited for as a keynote speaker at SMMA 2020 but had to decline).

11.1.5 Scientific expertise

• O-A. Maillard:
  • member of the CRCN/ISFP jury at Inria Lille.
  • was asked reviewing expertise for the Astrid ANR committee.
  • is part of Commission Emploi Recherche (CER) in 2020.
• Ph. Preux:
  • member of the national Inria DR jury.
  • member of the CRCN/ISFP jury at Inria Grenoble.
  • is a member of the scientific committee “data science and models” (CSS5) of the IRD.
  • is a member of a group thinking about “responsible AI” in REV3, in Lille.
  • He also declined reviewing many requests (including for the ANR, and IUF).

11.1.6 Research administration

• Philippe Preux is « Délégué Scientifique Adjoint » of Inria-Lille.

  As such, he is a member of the « Commission d'évaluation » of Inria.

  As head of Scool (REP), he is a member of the « Comité des équipes-projets » (CEP)of Inria-Lille.

  He is also a member of the « Bureau scientifique du centre » (BSC) of Inria-Lille.

11.2.1 Teaching

• D. Basu: Learning to Optimise Online with Full and Partial Information, Reading Session, M2 Data Science, UdL.
• D. Baudry taught about 100 h. in 2020, in data science and NLP, in M1 maths and M2 web analyst at the UdL.
• O. Darwiche taught reinforcement learning at École Centrale de Lille and in the data science master of the Université de Lille. He also served as a TA in reinforcement learning at the African Institute for Mathematical Sciences (AIMS) in Ghana in February.
• É. Kaufmann: Data Mining (21h) M1 Maths/Finance, UdL.
• É. Kaufmann: Sequential Decision Making (24h) M2 Data Science, Centrale Lille
• O-A. Maillard: Statistical Reinforcement Learning (42h), MAP/INF641, Master Artificial Intelligence and advanced Visual Computing, École Polytechnique.
• R. Ouhamma taught about 40 hours, algorithmics and programming at UdL, and also proba/stats in M2 at Centrale-Lille.
• S. Perrin taught about 30 hours, Unix and databases at UdL.
• Ph. Preux: « IA et apprentissage automatique », DU IA & Santé, UdL
• H. Saber: as part of his agrégé de mathématiques duty, he taught in the licence and master of mathematics at UdL.
• M. Seurin is ATER, hence teaches 192 hours during the academic year at UdL. He taught machine learning, data science, reinforcement learning and other topics in licence and master MIASHS.
• F. Valentini: « deep learning pour la radiologie », practical session, DU IA & Santé, UdL.
• J.-J. Vie: Introduction to Machine Learning (24h), M2 Mechanical Engineering, Polytech'Lille.
• J.-J. Vie: Deep Learning Do It Yourself (45h), M1, ENS Paris.

Due to the Covid-19 crisis, our participation to some summer schools has been canceled.

11.2.2 Supervision

Apart from Ph.D. students, we supervised the following students in 2020:

• Ph. Preux supervised A. Zeddoun, A. Moulin (master interns), H. Delavenne, A. Tuynmann, A. Vigneron (L3 interns).
• J.-J. Vie supervised Pierre Bourse (L3), Aymeric Floyrac, Salim Nadir (M2).

In Scool, we consider that supervising students is part of the training of a Ph.D. student. Therefore, some of them (like Y. Flet-Berliac and R. Ouhamma) have participated to supervision of master students, under the supervision of a permanent researcher.

11.2.3 Juries

• É. Kaufmann was part of the following juries:
  • PhD: Erwan LeCarpentier, IRIT Toulouse, July
  • PhD: Yann Issartel, Université Paris-Saclay, December
  • PhD: Margaux Brégère, Université Paris-Saclay, December
  • PhD: Andrea Tirinzoni, Polytechnico Milano, Italy, reviewer
• Ph. Preux was part of the following juries:
  • hdr: Nistor Grozavu, Université Paris 13, March, reviewer
  • hdr: Sylvain Lamprier, Sorbonne Université, September, reviewer
  • hdr: Émilie Kaufmann, UdL, November, president
  • PhD: Tanguy Levent, École Polytechnique, December, president
  • PhD: Matthieu Jedor, Université Paris-Saclay, December, reviewer
• O-A. Maillard was part of the following juries:
  • PhD: Robin Vogel, Télécom-Paris, October, examiner
  • PhD: Margaux Brégère, Université Paris-Saclay, December, reviewer.

International journals

• 11 articleGabriel A.G. Brat, Griffin M.G. Weber, NilsN. Gehlenborg, PaulP. Avillach, Nathan P.N. Palmer, LucaL. Chiovato, JamesJ. Cimino, Brett K.B. Beaulieu-Jones, SehiS. L'Yi, Mark S.M. Keller, Douglas S.D. Bell, Robert W.R. Follett, Lav P.L. Patel, Anne SophieA. Jannot, Lemuel R.L. Waitman, GilbertG. Omenn, AlbertoA. Malovini, Jason H.J. Moore, ValentinaV. Tibollo, Shawn NS. Murphy, RiccardoR. Bellazzi, David AD. Hanauer, ArnaudA. Serret-Larmande, AlbaA. Gutierrez-Sacristan, John JJ. Holmes, DouglasD. Bell, Kenneth D.K. Mandl, Jeffrey GJ. Klann, Douglas AD. Murad, LuigiaL. Scudeller, MauroM. Bucalo, KatieK. Kirchoff, JeanJ. Craig, JihadJ. Obeid, VianneyV. Jouhet, RomainR. Griffier, SébastienS. Cossin, BertrandB. Moal, AntonioA. Bellasi, Hans UH. Prokosch, DetlefD. Kraska, PiotrP. Sliz, Amelia L.M.A. Tan, Kee YuanK. Ngiam, AlbertoA. Zambelli, Danielle LD. Mowery, EmilyE. Schiver, BatsalB. Devkota, RobertR. Bradford, MohamadM. Daniar, ChristelC. Daniel, VincentV. Benoit, RomainR. Bey, NicolasN. Paris, PatriciaP. Serre, NinaN. Orlova, JulienJ. Dubiel, MartinM. Hilka, StephaneS. Breant, JudithJ. Leblanc, NicolasN. Griffon, AnitaA. Burgun, MelodieM. Bernaux, ArnaudA. Sandrin, ElisaE. Salamanca, SylvieS. Cormont, ThomasT. Ganslandt, TobiasT. Gradinger, JulienJ. Champ, MartinM. Boeker, PatriciaP. Martel, LoïcL. Estève, AlexandreA. Gramfort, OlivierO. Grisel, DamienD. Leprovost, ThomasT. Moreau, GaelG. Varoquaux, Jill-JênnJ.-J. Vie, DemianD. Wassermann, ArthurA. Mensch, CharlotteC. Caucheteux, ChristianC. Haverkamp, GuillaumeG. Lemaître, SilvanoS. Bosari, AndrewA. South, TianxiT. Cai and IsaacI. Kohane. International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortiumnpj Digital Medicine31December 2020, #109
  HALDOIback to text
• 12 article GuillaumeG. Gautier, RémiR. Bardenet and MichalM. Valko. Fast sampling from beta-ensembles Statistics and Computing 31 7 January 2021
  HAL DOI
• 13 article TomášT. Kocák, RémiR. Munos, BranislavB. Kveton, ShipraS. Agrawal and MichalM. Valko. Spectral bandits Journal of Machine Learning Research 2020
  HAL back to text
• 14 article RemyR. Kusters, DusanD. Misevic, HuguesH. Berry, AntoineA. Cully, YannY. Le Cunff, LoicL. Dandoy, NataliaN. Díaz-Rodríguez, MarionM. Ficher, JonathanJ. Grizou, AliceA. Othmani, ThemisT. Palpanas, MatthieuM. Komorowski, PatrickP. Loiseau, ClémentC. Moulin-Frier, SantinoS. Nanini, DanieleD. Quercia, MicheleM. Sebag, FrançoiseF. Soulié Fogelman, SofianeS. Taleb, LiubovL. Tupikina, VaibhavV. Sahu, Jill-JênnJ.-J. Vie and FatimaF. Wehbi. Interdisciplinary Research in Artificial Intelligence: Challenges and Opportunities Frontiers in Big Data 3 November 2020
  HAL DOI back to text
• 15 article TanguyT. Levent, PhilippeP. Preux, GonzagueG. Henri, RédaR. Alami, PhilippeP. Cordier and YvanY. Bonnassieux. The challenge of controlling microgrids in the presence of rare events with Deep Reinforcement Learning IET Smart Grid 2020
  HAL DOI back to text
• 16 articleClémenceC. Réda, EmilieE. Kaufmann and AndréeA. Delahaye-Duriez. Machine learning applications in drug developmentComputational and Structural Biotechnology Journal182020, 241-252
  HALDOIback to text

International peer-reviewed conferences

• 17 inproceedings RédaR. Alami, Odalric-AmbrymO.-A. Maillard and RaphaelR. Féraud. Restarted Bayesian Online Change-point Detector achieves Optimal Detection Delay International Conference on Machine Learning Wien, Austria July 2020
  HAL back to text
• 18 inproceedings MarcinM. Andrychowicz, AntonA. Raichuk, PiotrP. Stańczyk, ManuM. Orsini, SertanS. Girgin, RaphaëlR. Marinier, LéonardL. Hussenot, MatthieuM. Geist, OlivierO. Pietquin, MarcinM. Michalski, SylvainS. Gelly and OlivierO. Bachem. What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study ICLR 2021 - Ninth International Conference on Learning Representations Vienna / Virtual, Austria May 2021
  HAL
• 19 inproceedings DorianD. Baudry, EmilieE. Kaufmann and Odalric-AmbrymO.-A. Maillard. Sub-sampling for Efficient Non-Parametric Bandit Exploration NeurIPS 2020 Vancouver, Canada December 2020
  HAL back to text
• 20 inproceedings EtienneE. Boursier, EmilieE. Kaufmann, AbbasA. Mehrabian and VianneyV. Perchet. A Practical Algorithm for Multiplayer Bandits when Arm Means Vary Among Players AISTATS 2020 - 23rd International Conference on Artificial Intelligence and Statistics Palermo, Italy August 2020
  HAL back to text
• 21 inproceedingsJamesJ. Cheshire, PierreP. Ménard and AlexandraA. Carpentier. The Influence of Shape Constraints on the Thresholding Bandit ProblemCOLT 2020 - Thirty Third Conference on Learning Theory125Graz / Virtual, Austria2020, 1228-1275
  HALback to text
• 22 inproceedings GeoffreyG. Cideron, MathieuM. Seurin, FlorianF. Strub and OlivierO. Pietquin. HIGhER: Improving instruction following with Hindsight Generation for Experience Replay ADPRL 2020 - IEEE SSCI Conference on Adaptive Dynamic Programming and Reinforcement Learning Camberra / Virtual, Australia December 2020
  HAL back to text
• 23 inproceedings RobertR. Dadashi, LéonardL. Hussenot, MatthieuM. Geist and OlivierO. Pietquin. Primal Wasserstein Imitation Learning ICLR 2021 - Ninth International Conference on Learning Representations Vienna / Virtual, Austria June 2020
  HAL
• 24 inproceedings RémyR. Degenne, PierreP. Ménard, XuedongX. Shang and MichalM. Valko. Gamification of pure exploration for linear bandits International Conference on Machine Learning Vienna / Virtual, Austria 2020
  HAL back to text
• 25 inproceedings JohanJ. Ferret, RaphaëlR. Marinier, MatthieuM. Geist and OlivierO. Pietquin. Self-Attentional Credit Assignment for Transfer in Reinforcement Learning IJCAI 2020 - 29th International Joint Conference on Artificial Intelligence Yokohama / Virtual, Japan July 2020
  HAL back to text
• 26 inproceedings JohanJ. Ferret, OlivierO. Pietquin and MatthieuM. Geist. Self-Imitation Advantage Learning AAMAS 2021 - 20th International Conference on Autonomous Agents and Multiagent Systems Londres / Virtual, United Kingdom May 2021
  HAL
• 27 inproceedings YannisY. Flet-Berliac, JohanJ. Ferret, OlivierO. Pietquin, PhilippeP. Preux and MatthieuM. Geist. Adversarially Guided Actor-Critic ICLR 2021 - International Conference on Learning Representations Vienna / Virtual, Austria May 2021
  HAL
• 28 inproceedings YannisY. Flet-Berliac, RedaR. Ouhamma, Odalric-AmbrymO.-A. Maillard and PhilippeP. Preux. Learning Value Functions in Deep Policy Gradients using Residual Variance ICLR 2021 - International Conference on Learning Representations Vienna / Virtual, Austria May 2021
  HAL
• 29 inproceedings YannisY. Flet-Berliac and PhilippeP. Preux. Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL IJCAI 2020 - International Joint Conference on Artificial Intelligence Yokohama, Japan July 2020
  HAL DOI
• 30 inproceedings NathanN. Grinsztajn, OlivierO. Beaumont, EmmanuelE. Jeannot and PhilippeP. Preux. Geometric Deep Reinforcement Learning for Dynamic DAG Scheduling IEEE SSCI 2020 - Symposium Series on Computational Intelligence SSCI 2020 proceedings Canberra / Virtual, Australia December 2020
  HAL back to text
• 31 inproceedings LéonardL. Hussenot, RobertR. Dadashi, MatthieuM. Geist and OlivierO. Pietquin. Show me the Way: Intrinsic Motivation from Demonstrations AAMAS 2021 - 20th International Conference on Autonomous Agents and Multiagent Systems Virtual, United Kingdom May 2021
  HAL
• 32 inproceedings LéonardL. Hussenot, MatthieuM. Geist and OlivierO. Pietquin. CopyCAT: Taking Control of Neural Policies with Constant Attacks AAMAS 2020 - 19th International Conference on Autonomous Agents and Multi-Agent Systems Virtual, New Zealand May 2020
  HAL back to text
• 33 inproceedings AndersA. Jonsson, EmilieE. Kaufmann, PierreP. Ménard, Omar DO. Domingues, EdouardE. Leurent and MichalM. Valko. Planning in Markov Decision Processes with Gap-Dependent Sample Complexity Neural Information Processing Systems Vancouver, France December 2020
  HAL back to text
• 34 inproceedingsEmilioE. Jorge, HannesH. Eriksson, ChristosC. Dimitrakakis, DebabrotaD. Basu and DivyaD. Grover. Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning"I Can't Believe It's Not Better!" at NeurIPS Workshops"I Can't Believe It's Not Better!" at NeurIPS Workshops137Proceedings of Machine Learning ResearchVancouver, CanadaDecember 2020, 43-52
  HALback to text
• 35 inproceedings EmilieE. Kaufmann, PierreP. Ménard, OmarO. Darwiche Domingues, AndersA. Jonsson, EdouardE. Leurent and MichalM. Valko. Adaptive reward-free exploration Algorithmic Learning Theory Paris, France 2021
  HAL
• 36 inproceedings EdouardE. Leurent, DenisD. Efimov and Odalric-AmbrymO.-A. Maillard. Robust-Adaptive Control of Linear Systems: beyond Quadratic Costs NeurIPS 2020 - 34th Conference on Neural Information Processing Systems Vancouver / Virtual, Canada December 2020
  HAL back to text
• 37 inproceedings EdouardE. Leurent, DenisD. Efimov and Odalric-AmbrymO.-A. Maillard. Robust-Adaptive Interval Predictive Control for Linear Uncertain Systems CDC 2020 - 59th IEEE Conference on Decision and Control Jeju Island / Virtual, South Korea December 2020
  HAL back to text
• 38 inproceedingsEdouardE. Leurent and Odalric-AmbrymO.-A. Maillard. Monte-Carlo Graph Search: the Value of Merging Similar StatesACML 2020 - 12th Asian Conference on Machine Learning129Bangkok / Virtual, Thailand2020, 577 - 602
  HALback to text
• 39 inproceedings Odalric-AmbrymO.-A. Maillard, HippolyteH. Bourel and Mohammad SadeghM. Talebi. Tightening Exploration in Upper Confidence Reinforcement Learning International Conference on Machine Learning Vienna, Austria July 2020
  HAL
• 40 inproceedings PierreP. Perrault, EtienneE. Boursier, VianneyV. Perchet and MichalM. Valko. Statistical efficiency of Thompson sampling for combinatorial semi-bandits Neural Information Processing Systems virtual, France 2020
  HAL back to text
• 41 inproceedings PierreP. Perrault, JenniferJ. Healey, ZhengZ. Wen and MichalM. Valko. Budgeted online influence maximization International Conference on Machine Learning Vienna, Austria 2020
  HAL back to text
• 42 inproceedings PierreP. Perrault, VianneyV. Perchet and MichalM. Valko. Covariance-adapting algorithm for semi-bandits with application to sparse outcomes Conference on Learning Theory Graz, Austria 2020
  HAL back to text
• 43 inproceedings MathieuM. Seurin, PhilippeP. Preux and OlivierO. Pietquin. "I'm sorry Dave, I'm afraid I can't do that" Deep Q-Learning From Forbidden Actions Internationnal Joint Conference on Neural Networks Glasgow, United Kingdom July 2020
  HAL back to text
• 44 inproceedings MathieuM. Seurin, FlorianF. Strub, PhilippeP. Preux and OlivierO. Pietquin. A Machine of Few Words Interactive Speaker Recognition with Reinforcement Learning Interspeech 2020 proceedings Conference of the International Speech Communication Association (INTERSPEECH) Shanghai, China October 2020
  HAL DOI back to text
• 45 inproceedings XuedongX. Shang, RianneR. De Heide, EmilieE. Kaufmann, PierreP. Ménard and MichalM. Valko. Fixed-confidence guarantees for Bayesian best-arm identification International Conference on Artificial Intelligence and Statistics Palermo, Italy 2020
  HAL back to text
• 46 inproceedingsCindyC. Trinh, EmilieE. Kaufmann, ClaireC. Vernade and RichardR. Combes. Solving Bernoulli Rank-One Bandits with Unimodal Thompson SamplingALT 2020 - 31st International Conference on Algorithmic Learning Theory117San Diego, United StatesFebruary 2020, 1 - 28
  HALback to text

National peer-reviewed Conferences

Conferences without proceedings

• 47 inproceedings BenoîtB. Choffin, FabriceF. Popineau, YolaineY. Bourda and Jill-JênnJ.-J. Vie. Evaluating DAS3H on the EdNet Dataset AAAI 2021 - The 35th Conference on Artificial Intelligence / Imagining Post-COVID Education with AI Virtual, United States January 2021
  HAL

Doctoral dissertations and habilitation theses

• 48 thesis EdouardE. Leurent. Safe and Efficient Reinforcement Learning for Behavioural Planning in Autonomous Driving Université de Lille October 2020
  HAL
• 49 thesis PierreP. Perrault. Efficient Learning in Stochastic Combinatorial Semi-Bandits Univeristé Paris-Saclay November 2020
  HAL
• 50 thesis FlorianF. Strub. Multimodal and Interactive Models for Visually Grounded Language Learning Université de Lille; École doctorale, ED SPI 074 : Sciences pour l'Ingénieur January 2020
  HAL

Reports & preprints

• 51 misc MaryamM. Aziz, EmilieE. Kaufmann and Marie-KarelleM.-K. Riviere. On Multi-Armed Bandit Designs for Dose-Finding Trials April 2020
  HAL back to text
• 52 misc LilianL. Besson, EmilieE. Kaufmann, Odalric-AmbrymO.-A. Maillard and JulienJ. Seznec. Efficient Change-Point Detection for Tackling Piecewise-Stationary Bandits December 2020
  HAL back to text
• 53 misc Omar DO. Domingues, PierreP. Ménard, MatteoM. Pirotta, EmilieE. Kaufmann and MichalM. Valko. Regret bounds for kernel-based reinforcement learning Vienna, Austria April 2020
  HAL back to text
• 54 misc HannesH. Eriksson, DebabrotaD. Basu, MinaM. Alibeigi and ChristosC. Dimitrakakis. SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning February 2021
  HAL
• 55 misc EvrardE. Garcelon, BaptisteB. Roziere, LaurentL. Meunier, JeanJ. Tarbouriech, OlivierO. Teytaud, AlessandroA. Lazaric and MatteoM. Pirotta. Adversarial Attacks on Linear Contextual Bandits October 2020
  HAL back to text
• 56 report PierreP. Ménard, Omar DarwicheO. Domingues, AndersA. Jonsson, EmilieE. Kaufmann, EdouardE. Leurent and MichalM. Valko. Fast active learning for pure exploration in reinforcement learning DeepMind July 2020
  HAL
• 57 misc SarahS. Perrin, JulienJ. Pérolat, MathieuM. Laurière, MatthieuM. Geist, RomualdR. Elie and OlivierO. Pietquin. Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications September 2020
  HAL back to text
• 58 misc HassanH. Saber, PierreP. Ménard and Odalric-AmbrymO.-A. Maillard. Forced-exploration free Strategies for Unimodal Bandits June 2020
  HAL back to text
• 59 misc HassanH. Saber, PierreP. Ménard and Odalric-AmbrymO.-A. Maillard. Optimal Strategies for Graph-Structured Bandits July 2020
  HAL back to text
• 60 misc XuedongX. Shang, HanH. Shao and JianJ. Qian. Stochastic bandits with vector losses: Minimizing ${}^{}$-norm of relative losses' October 2020
  HAL back to text