Machine Learning for Multimedia Material.

  Many approaches are used to extract relevant information from multimedia material, ranging from very low-level to higher-level descriptions (classes, captions, ...). That diversity of information is produced by algorithms that have varying degrees of supervision. Lately, fully supervised approaches based on deep learning proved to outperform most older techniques. This is particularly true for the latest developments of Recurrent Neural Networkds (RNN, such as LSTMs) or convolutional neural network (CNNs) for images that reach excellent performance  48. Linkmedia contributes to advancing the state of the art in computing representations for multimedia material by investigating the topics listed below. Some of them go beyond the very processing of multimedia material as they also question the fundamentals of machine learning procedures when applied to multimedia.

• Learning from few samples/weak supervisions. CNNs and RNNs need large collections of carefully annotated data. They are not fitted for analyzing datasets where few examples per category are available or only cheap image-level labels are provided. Linkmedia investigates low-shot, semi-supervised and weakly supervised learning processes: Augmenting scarce training data by automatically propagating labels  51, or transferring what was learned on few very well annotated samples to allow the precise processing of poorly annotated data  60. Note that this context also applies to the processing of heritage collections (paintings, illuminated manuscripts, ...) that strongly differ from contemporary natural images. Not only annotations are scarce, but the learning processes must cope with material departing from what standard CNNs deal with, as classes such as "planes", "cars", etc, are irrelevant in this case.
• Ubiquitous Training. NN (CNNs, LSTMs) are mainstream for producing representations suited for high-quality classification. Their training phase is ubiquitous because the same representations can be used for tasks that go beyond classification, such as retrieval, few-shot, meta- and incremental learning, all boiling down to some form of metric learning. We demonstrated that this ubiquitous training is relatively simpler  51 yet as powerful as ad-hoc strategies fitting specific tasks  65. We study the properties and the limitations of this ubiquitous training by casting metric learning as a classification problem.
• Beyond static learning. Multimedia collections are by nature continuously growing, and ML processes must adapt. It is not conceivable to re-train a full new model at every change, but rather to support continuous training and/or allowing categories to evolve as the time goes by. New classes may be defined from only very few samples, which links this need for dynamicity to the low-shot learning problem discussed here. Furthermore, active learning strategies determining which is the next sample to use to best improve classification must be considered to alleviate the annotation cost and the re-training process  55. Eventually, the learning process may need to manage an extremely large number of classes, up to millions. In this case, there is a unique opportunity of blending the expertise of Linkmedia on large scale indexing and retrieval with deep learning. Base classes can either be "summarized" e.g. as a multi-modal distribution, or their entire training set can be made accessible as an external associative memory  71.
• Learning and lightweight architectures. Multimedia is everywhere, it can be captured and processed on the mobile devices of users. It is necessary to study the design of lightweight ML architectures for mobile and embedded vision applications. Inspired by  75, we study the savings from quantizing hyper-parameters, pruning connections or other approximations, observing the trade-off between the footprint of the learning and the quality of the inference. Once strategy of choice is progressive learning which early aborts when confident enough  56.
• Multimodal embeddings. We pursue pioneering work of Linkmedia on multimodal embedding, i.e., representing multiple modalities or information sources in a single embedded space  69, 68, 70. Two main directions are explored: exploiting adversarial architectures (GANs) for embedding via translation from one modality to another, extending initial work in  70 to highly heterogeneous content; combining and constraining word and RDF graph embeddings to facilitate entity linking and explanation of lexical co-occurrences  45.
• Accountability of ML processes. ML processes achieve excellent results but it is mandatory to verify that accuracy results from having determined an adequate problem representation, and not from being abused by artifacts in the data. Linkmedia designs procedures for at least explaining and possibly interpreting and understanding what the models have learned. We consider heat-maps materializing which input (pixels, words) have the most importance in the decisions  64, Taylor decompositions to observe the individual contributions of each relevance scores or estimating LID 32 as a surrogate for accounting for the smoothness of the space.
• Extracting information. ML is good at extracting features from multimedia material, facilitating subsequent classification, indexing, or mining procedures. Linkmedia designs extraction processes for identifying parts in the images  61, 62, relationships between the various objects that are represented in images  38, learning to localizing objects in images with only weak, image-level supervision  64 or fine-grained semantic information in texts  43. One technique of choice is to rely on generative adversarial networks (GAN) for learning low-level representations. These representations can e.g. be based on the analysis of density  74, shading, albedo, depth, etc.
• Learning representations for time evolving multimedia material. Video and audio are time evolving material, and processing them requests to take their time line into account. In  57, 41 we demonstrated how shapelets can be used to transform time series into time-free high-dimensional vectors, preserving however similarities between time series. Representing time series in a metric space improves clustering, retrieval, indexing, metric learning, semi-supervised learning and many other machine learning related tasks. Research directions include adding localization information to the shapelets, fine-tuning them to best fit the task in which they are used as well as designing hierarchical representations.

Adversarial Machine Learning.

  Systems based on ML take more and more decisions on our behalf, and maliciously influencing these decisions by crafting adversarial multimedia material is a potential source of dangers: a small amount of carefully crafted noise imperceptibly added to images corrupts classification and/or recognition. This can naturally impact the insight users get on the multimedia collection they work with, leading to taking erroneous decisions e.g.

This adversarial phenomenon is not particular to deep learning, and can be observed even when using other ML approaches  37. Furthermore, it has been demonstrated that adversarial samples generalize very well across classifiers, architectures, training sets. The reasons explaining why such tiny content modifications succeed in producing severe errors are still not well understood.

We are left with little choice: we must gain a better understanding of the weaknesses of ML processes, and in particular of deep learning. We must understand why attacks are possible as well as discover mechanisms protecting ML against adversarial attacks (with a special emphasis on convolutional neural networks). Some initial contributions have started exploring such research directions, mainly focusing on images and computer vision problems. Very little has been done for understanding adversarial ML from a multimedia perspective  42.

Linkmedia is in a unique position to throw at this problem new perspectives, by experimenting with other modalities, used in isolation one another, as well as experimenting with true multimodal inputs. This is very challenging, and far more complicated and interesting than just observing adversarial ML from a computer vision perspective. No one clearly knows what is at stake with adversarial audio samples, adversarial video sequences, adversarial ASR, adversarial NLP, adversarial OCR, all this being often part of a sophisticated multimedia processing pipeline.

Our ambition is to lead the way for initiating investigations where the full diversity of modalities we are used to work with in multimedia are considered from a perspective of adversarial attacks and defenses, both at learning and test time. In addition to what is described above, and in order to trust the multimedia material we analyze and/or the algorithms that are at play, Linkmedia investigates the following topics:

• Beyond classification. Most contributions in relation with adversarial ML focus on classification tasks. We started investigating the impact of adversarial techniques on more diverse tasks such as retrieval  31. This problem is related to the very nature of euclidean spaces where distances and neighborhoods can all be altered. Designing defensive mechanisms is a natural companion work.
• Detecting false information. We carry-on with earlier pioneering work of Linkmedia on false information detection in social media. Unlike traditional approaches in image forensics  46, we build on our expertise in content-based information retrieval to take advantage of the contextual information available in databases or on the web to identify out-of-context use of text or images which contributed to creating a false information  58.
• Deep fakes. Progress in deep ML and GANs allow systems to generate realistic images and are able to craft audio and video of existing people saying or doing things they never said or did 54. Gaining in sophistication, these machine learning-based "deep fakes" will eventually be almost indistinguishable from real documents, making their detection/rebutting very hard. Linkmedia develops deep learning based counter-measures to identify such modern forgeries. We also carry on with making use of external data in a provenance filtering perspective  63 in order to debunk such deep fakes.
• Distributions, frontiers, smoothness, outliers. Many factors that can possibly explain the adversarial nature of some samples are in relation with their distribution in space which strongly differs from the distribution of natural, genuine, non adversarial samples. We are investigating the use of various information theoretical tools that facilitate observing distributions, how they differ, how far adversarial samples are from benign manifolds, how smooth is the feature space, etc. In addition, we are designing original adversarial attacks and develop detection and curating mechanisms  32.

Multimedia Knowledge Extraction.

  Information obtained from collections via computer ran processes is not the only thing that needs to be represented. Humans are in the loop, and they gradually improve their level of understanding of the content and nature of the multimedia collection. Discovering knowledge and getting insight is involving multiple people across a long period of time, and what each understands, concludes and discovers must be recorded and made available to others. Collaboratively inspecting collections is crucial. Ontologies are an often preferred mechanism for modeling what is inside a collection, but this is probably limitative and narrow.

Linkmedia is concerned with making use of existing strategies in relation with ontologies and knowledge bases. In addition, Linkmedia uses mechanisms allowing to materialize the knowledge gradually acquired by humans and that might be subsequently used either by other humans or by computers in order to better and more precisely analyze collections. This line of work is instantiated at the core of the iCODA project Linkmedia coordinates.

We are therefore concerned with:

• Multimedia analysis and ontologies. We develop approaches for linking multimedia content to entities in ontologies for text and images, building on results in multimodal embedding to cast entity linking into a nearest neighbor search problem in a high-dimensional joint embedding of content and entities  68. We also investigate the use of ontological knowledge to facilitate information extraction from content  45.
• Explainability and accountability in information extraction. In relation with ontologies and entity linking, we develop innovative approaches to explain statistical relations found in data, in particular lexical or entity co-occurrences in textual data, for example using embeddings constrained with translation properties of RDF knowledge or path-based explanation within RDF graphs. We also work on confidence measures in entity linking and information extraction, studying how the notions of confidence and information source can be accounted for in knowledge basis and used in human-centric collaborative exploration of collections.
• Dynamic evolution of models for information extraction. In interactive exploration and information extraction, e.g., on cultural or educational material, knowledge progressively evolves as the process goes on, requiring on-the-fly design of new models for content-based information extractors from very few examples, as well as continuous adaptation of the models. Combining in a seamless way low-shot, active and incremental learning techniques is a key issue that we investigate to enable this dynamic mechanisms on selected applications.

Searching.

  Search engines must run similarity searches very efficiently. High-dimensional indexing techniques therefore play a central role. Yet, recent contributions in ML suggest to revisit indexing in order to adapt to the specific properties of modern features describing contents.

• Advanced scalable indexing. High-dimensional indexing is one of the foundations of Linkmedia. Modern features extracted from the multimedia material with the most recent ML techniques shall be indexed as well. This, however, poses a series of difficulties due to the dimensionality of these features, their possible sparsity, the complex metrics in use, the task in which they are involved (instance search, $k$-nn, class prototype identification, manifold search  51, time series retrieval, ...). Furthermore, truly large datasets require involving sketching  35, secondary storage and/or distribution  34, 33, alleviating the explosion of the number of features to consider due to their local nature or other innovative methods  50, all introducing complexities. Last, indexing multimodal embedded spaces poses a new series of challenges.
• Improving quality. Scalable indexing techniques are approximate, and what they return typically includes a fair amount of false positives. Linkmedia works on improving the quality of the results returned by indexing techniques. Approaches taking into account neighborhoods  44, manifold structures instead of pure distance based similarities  51 must be extended to cope with advanced indexing in order to enhance quality. This includes feature selection based on intrinsic dimensionality estimation  32.
• Dynamic indexing. Feature collections grow, and it is not an option to fully reindex from scratch an updated collection. This trivially applies to the features directly extracted from the media items, but also to the base class prototypes that can evolve due to the non-static nature of learning processes. Linkmedia will continue investigating what is at stake when designing dynamic indexing strategies.

Navigating.

  Navigating a multimedia collection is very central to its understanding. It differs from searching as navigation is not driven by any specific query. Rather, it is mostly driven by the relationships that various documents have one another. Relationships are supported by the links between documents and/or parts of documents. Links rely on semantic similarity, depicting the fact that two documents share information on the same topic. But other aspects than semantics are also at stake, e.g., time with the dates of creation of the documents or geography with mentions or appearance in documents of some geographical landmarks or with geo-tagged data.

In multimedia collections, links can be either implicit or explicit, the latter being much easier to use for navigation. An example of an implicit link can be the name of someone existing in several different news articles; we, as humans, create a mental link between them. In some cases, the computer misses such configurations, leaving such links implicit. Implicit links are subject to human interpretation, hence they are sometimes hard to identify for any automatic analysis process. Implicit links not being materialized, they can therefore hardly be used for navigation or faceted search. Explicit links can typically be seen as hyperlinks, established either by content providers or, more aligned with Linkmedia, automatically determined from content analysis. Entity linking (linking content to an entity referenced in a knowledge base) is a good example of the creation of explicit links. Semantic similarity links, as investigated in the LIMAH project and as considered in the search and hyperlinking task at MediaEval and TRECVid, are also prototypical links that can be made explicit for navigation. Pursuing work, we investigate two main issues:

• Improving multimodal content-based linking. We exploit achievements in entity linking to go beyond lexical or lexico-visual similarity and to provide semantic links that are easy to interpret for humans; carrying on, we work on link characterization, in search of mechanisms addressing link explainability (i.e., what is the nature of the link), for instance using attention models so as to focus on the common parts of two documents or using natural language generation; a final topic that we address is that of linking textual content to external data sources in the field of journalism, e.g., leveraging topic models and cue phrases along with a short description of the external sources.
• Dynamicity and user-adaptation. One difficulty for explicit link creation is that links are often suited for one particular usage but not for another, thus requiring creating new links for each intended use; whereas link creation cannot be done online because of its computational cost, the alternative is to generate (almost) all possible links and provide users with selection mechanisms enabling personalization and user-adaptation in the exploration process; we design such strategies and investigate their impact on exploration tasks in search of a good trade-off between performance (few high-quality links) and genericity.

Summarizing.

  Multimedia collections contain far too much information to allow any easy comprehension. It is mandatory to have facilities to aggregate and summarize a large body on information into a compact, concise and meaningful representation facilitating getting insight. Current technology suggests that multimedia content aggregation and story-telling are two complementary ways to provide users with such higher-level views. Yet, very few studies already investigated these issues. Recently, video or image captioning  72, 67 have been seen as a way to summarize visual content, opening the door to state-of-the-art multi-document text summarization  47 with text as a pivot modality. Automatic story-telling has been addressed for highly specific types of content, namely TV series  39 and news  59, 66, but still need a leap forward to be mostly automated, e.g., using constraint-based approaches for summarization  36, 66.

Furthermore, not only the original multimedia material has to be summarized, but the knowledge acquired from its analysis is also to summarize. It is important to be able to produce high-level views of the relationships between documents, emphasizing some structural distinguishing qualities. Graphs establishing such relationships need to be constructed at various level of granularity, providing some support for summarizing structural traits.

Summarizing multimedia information poses several scientific challenges that are:

• Choosing the most relevant multimedia aggregation type: Taking a multimedia collection into account, a same piece of information can be present in several modalities. The issue of selecting the most suitable one to express a given concept has thus to be considered together with the way to mix the various modalities into an acceptable production. Standard summarization algorithms have to be revisited so that they can handle continuous representation spaces, allowing them to benefit from the various modalities  40.
• Expressing user’s preferences: Different users may appreciate quite different forms of multimedia summaries, and convenient ways to express their preferences have to be proposed. We for example focus on the opportunities offered by the constraint-based framework.
• Evaluating multimedia summaries: Finding criteria to characterize what a good summary is remains challenging, e.g., how to measure the global relevance of a multimodal summary and how to compare information between and across two modalities. We tackle this issue particularly via a collaboration with A. Smeaton at DCU, comparing the automatic measures we will develop to human judgments obtained by crowd-sourcing.
• Taking into account structuring and dynamicity: Typed links between multimedia fragments, and hierarchical topical structures of documents obtained via work previously developed within the team are two types of knowledge which have seldom been considered as long as summarization is concerned. Knowing that the event present in a document is causally related to another event described in another document can however modify the ways summarization algorithms have to consider information. Moreover the question of producing coarse-to-fine grain summaries exploiting the topical structure of documents is still an open issue. Summarizing dynamic collections is also challenging and it is one of the questions we consider.

Social biases in text generation.

Recent advances in the domain of text generation allow realistic text-based interaction with a computer. These systems rely on complex neural architectures that leverage very large amount of training texts collected the Web. The problem is that these texts contains unwanted biases (sexism, racism, harmful language...) that are sometimes even amplified by the training procedure. Curating the training texts once for all is not feasible due to the complexity of defining a priori what is relevant or not at the training time. Our work on controlled generation 9 takes another point of view and tries to impose constraints at the inference time. This work aims at making the text generation respect application-specific conditions with the help of a simple classifier.

7.1.1 RoSe: A RObust and SEcure Black-Box DNN Watermarking

Participants: Kassem Kallas, Teddy Furon.

7.1.2 Randomized Smoothing under Attack: How Good is it in Pratice?

Participants: Thibault Maho, Teddy Furon, Erwan Le Merrer [WIDE].

7.1.3 Traitor tracing

Participant: Teddy Furon.

7.1.4 FBI: Fingerprinting models with Benign Inputs

Participants: Thibault Maho, Teddy Furon, Erwan Le Merrer [WIDE].

7.1.5 Watermarking Images in Self-Supervised Latent Spaces

Participants: Pierre Fernandez [Meta IA], Alexandre Sablayrolles [Meta IA], Teddy Furon, Hervé Jégou [Meta IA], Matthijs Douze [Meta IA].

7.1.6 Adversarial images with downscaling

Participants: Teddy Furon, Benoît Bonnet, Patrick Bas [CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189].

7.1.7 Temporal Relation Extraction in Clinical Texts

Participants: Yohan Bonescki Gumiel [PUCPR - Pontifícia Universidade Católica do Paraná, Brazil], Lucas Emanuel Silva Oliveira [PUCPR - Pontifícia Universidade Católica do Paraná, Brazil], Vincent Claveau, Natalia Grabar [STL - Savoirs, Textes, Langage (STL) - UMR 8163], Emerson Cabrera Paraiso [PUCPR - Pontifícia Universidade Católica do Paraná, Brazil], Claudia Moro [PUCPR - Pontifícia Universidade Católica do Paraná, Brazil], Deborah Ribeiro Carvalho [PUCPR - Pontifícia Universidade Católica do Paraná, Brazil].

7.1.8 Deep Neural Network Attacks and Defense: The Case of Image Classification

Participants: Hanwei Zhang [LIS Marseille], Teddy Furon, Laurent Amsaleg, Yannis Avrithis [IARAI].

7.1.9 AlignMixup: Improving Representations By Interpolating Aligned Features

Participants: Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, Yannis Avrithis [IARAI].

7.1.10 It Takes Two to Tango: Mixup for Deep Metric Learning

Participants: Shashanka Venkataramanan, Bill Psomas [National Technical University of Athens], Ewa Kijak, Laurent Amsaleg, Konstantinos Karantzalos [National Technical University of Athens], Yannis Avrithis [IARAI].

7.1.11 Confronting Active Learning for Relation Extraction to a Real-life Scenario on French Newspaper Data

Participants: Cyrielle Mallart, Michel Le Nouy [Sipa Ouest-France, Rennes], Guillaume Gravier, Pascale Sébillot.

7.1.12 Statistical study of word embeddings in French transformer models

Participants: Loïc Fosse, Duc-Hau Nguyen, Pascale Sébillot, Guillaume Gravier.

7.1.13 Improving the plausibility of attention weights in natural language inference

Participants: Duc-Hau Nguyen, Guillaume Gravier, Pascale Sébillot.

7.1.14 Graph-based image gradients aggregated with random forests

Participants: Raquel Almeida, Ewa Kijak, Simon Malinowski, Silvio J.R. Guimarães [PUC Minas, Brésil] .

7.1.15 AIP: Adversarial Interaction Priors for Multi-Agent Physics-based Character Control

Participants: Mohamed Younes [MIMETIC], Ewa Kijak, Richard Kulpa [MiMETIC, M2S], Simon Malinowski, Franck Multon [MIMETIC] .

7.1.16 Impact of Data Cleansing for Urban Bus Commercial Speed Prediction

Participants: Gauthier Lyan, David Gross-Amblard [Druid], Jean-Marc Jézéquel [Diverse], Simon Malinowski.

7.1.17 Reasoning over Time into Models with DataTime

Participants: Gauthier Lyan, Jean-Marc Jézéquel [Diverse], David Gross-Amblard [Druid], Romain Lefeuvre [Diverse], Benoit Combemale [Diverse].

7.1.18 Prédiction du niveau de nappes phréatiques : comparaison d'approches locale, globale et hybride

Participants: Lola Beuchée, Thomas Guyet [LIRIS, LBBE], Simon Malinowski.

7.1.19 Temporal Disaggregation of the Cumulative Grass Growth

Participants: Thomas Guyet [LIRIS, LBBE], Laurent Spillemaecker [ENSAI], Simon Malinowski, Anne-Isabelle Graux [PEGASE - Physiologie, Environnement et Génétique pour l'Animal et les Systèmes d'Elevage].

7.2.1 PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding

Participants: Antoine Chaffin [IMATAG], Vincent Claveau, Ewa Kijak.

7.2.2 Which Discriminator for Cooperative Text Generation?

Participants: Antoine Chaffin [IMATAG], Thomas Scialom [reciTAL], Sylvain Lamprier [MLIA-ISIR], Jacopo Staiano [reciTAL], Benjamin Piwowarski [MLIA-ISIR], Ewa Kijak, Vincent Claveau.

7.2.3 Generative Cooperative Networks for Natural Language Generation

Participants: Sylvain Lamprier [MLIA-ISIR], Thomas Scialom [reciTAL], Antoine Chaffin [IMATAG], Vincent Claveau, Ewa Kijak, Jacopo Staiano [reciTAL], Benjamin Piwowarski [MLIA-ISIR].

CIFRE PhD: Incremental dynamic construction of knowledge bases from text mining

Participants: Guillaume Gravier, Cyrielle Mallart, Pascale Sébillot.

Duration: 3 years, started in Dec. 2018 Partner: Ouest France

CIFRE PhD: Few shot learning for object recognition in aerial images

Participants: Yannis Avrithis, Yann Lifchitz.

Duration: 3 years, started in March 2018 Partner: Safran Tech

CIFRE PhD: Deep Learning and Homomorphic encryption

Participants: Teddy Furon, Samuel Tap.

Duration: 3 years, started in December 2020 Partner: ZAMA.ia

CIFRE PhD: Robustness of machine learning against uncertainties

Participants: Teddy Furon, Mathias Rousset, Karim Tit.

Duration: 3 years, started in December 2020 Partner: THALES La Ruche

CIFRE PhD: Semantic multimodal question answering (MQA) in domestic environments

Participants: Yannis Avrithis, Teddy Furon, Deniz Engin.

Duration: 3 years, started in September 2020 Partner: InterDigital

CIFRE PhD: Multimodal detection of fake news

Participants: Vincent Claveau, Ewa Kijak, Antoine Chaffin.

Duration: 3 years, started in November 2020 Partner: IMATAG

CIFRE PhD: Certification of Deep Neural Networks

Participants: Teddy Furon, Kassem Kallas, Quentin Le Roux.

Duration: 3 years, started in November 2022 Partner:THALES

CIFRE PhD: Watermarking and deep learning

Participants: Teddy Furon, Pierre Fernandez.

Duration: 3 years, started in May 2022 Partner: META AI

CIFRE PhD: Domain generalization exploiting synthetic data

Participants: Ewa Kijak, Louis Hemadou.

Duration: 3 years, started in Nov. 2022 Partner: SAFRAN

Telegramme-CNRS bilateral contract: NLP for computational journalism

Participants: Vincent Claveau, Nicolas Fouqué.

Duration: 2 years, started in Jan 2022

9.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

9.1.2 Participation in other International Programs

Chaire Security of AI for Defense Applications (SAIDA)

Participants: Teddy Furon, Laurent Amsaleg, Erwan Le Merrer [WIDE], Mathias Rousset [SIMSMART], Benoit Bonnet, Thibault Maho, Patrick Bas [CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189], Samuel Tap, Karim Tit.

Duration: 4 years, started Sept 2020 ANR-20-CHIA-0011-01

SAIDA targets the AID "Fiabilité de l’intelligence artificielle, vulnérabilités et contre-mesures" chair. It aims at establishing the fundamental principles for designing reliable and secure AI systems: a reliable AI maintains its good performance even under uncertainties; a secure AI resists attacks in hostile environments. Reliability and security are challenged at training and at test time. SAIDA therefore studies core issues in relation with poisoning training data, stealing the parameters of the model or inferring sensitive training from information leaks. Additionally, SAIDA targets uncovering the fundamentals of attacks and defenses engaging AI at test time. Three converging research directions make SAIDA: 1) theoretical investigations grounded in statistics and applied mathematics to discover the underpinnings of reliability and security, 2) connects adversarial sampling and Information Forensics and Security, 3) protecting the training data and the AI system. SAIDA thus combines theoretical investigations with more applied and heuristic studies to guarantee the applicability of the findings as well as the ability to cope with real world settings.

ANR Archival: Multimodal machine comprehension of language for new intelligent interfaces of scientific and cultural mediation

Participants: Laurent Amsaleg, Guillaume Gravier, Guillaume Le Noé-Bienvenu, Duc Hau Nguyen, Pascale Sébillot.

Duration: 3.5 year, started in Dec. 2019

The multidisciplinary and multi-actor ARCHIVAL project aims at yielding collaborations between researchers from the fields of Information and Communication Sciences as well as Computer Sciences around archive value enhancing and knowledge sharing for arts, culture and heritage. The project is structured around the following questionings: What part can machine comprehension methods play towards the reinterpretation of thematic archive collections? How can content mediation interfaces exploit results generated by current AI approaches?

ARCHIVAL teams will explore heterogeneous document collection structuration in order to explicitly reveal implicit links, to explain the nature of these links and to promote them in an intelligible way towards ergonomic mediation interfaces that will guarantee a successful appropriation of contents. A corpus has been delimited from the FMSH “self-management” collection, recently awarded as Collex, which will be completed from the large Canal-U academic audiovisual portal. The analysis and enhancement of this collection is of particular interest for Humanities and Social Sciences in a context where it becomes a necessity to structurally reconsider new models of socioeconomic development (democratic autonomy, social and solidarity-based economy, alternative development,…).

ANR MEERQAT: MultimEdia Entity Representation and Question Answering Tasks

Participants: Laurent Amsaleg, Yannis Avrithis, Ewa Kijak, Shashanka Venkataramanan.

Duration: 3.5 year, started in April 2020 Partners: Inria project-teams Linkmedia, CEA LIST, LIMSI, IRIT.

The overall goal of the project is to tackle the problem of ambiguities of visual and textual content by learning then combining their representations. As a final use case, we propose to solve a Multimedia Question Answering task, that requires to rely on three different sources of information to answer a (textual) question with regard to visual data as well as an external knowledge base containing millions of unique entities, each being represetd by textual and visual content as well as some links to other entities. An important work will deal with the representation of entities into a common tri-modal space, in which one should determine the content to associate to an entity to adequately represent it. The challenge consists in defining a representation that is compact (for performance) while still expressive enough to reflect the potential links between the entity and a variety of others.

MinArm: EVE3

Participants: Teddy Furon.

Duration: 3 year, started in April 2019 Partners: MinArm, CRIStAL Lille, LIRMM, Univ. Troyes, Univ. Paris Saclay

AID-CNRS: FakeNews

Participants: Vincent Claveau, Ewa Kijak, Gauthier Lyan.

Duration: 2 years, started mid-2021

ASTRID: HybrInfox

Participants: Vincent Claveau, Guillaume Gravier.

Duration: 20 months, started Jan. 2022

10.1.1 Scientific events: organisation

10.1.2 Scientific events: selection

10.1.3 Journal

10.1.4 Invited talks

• Teddy Furon was an invited speaker at Conference on Artificial Intelligence for Defense, during the European Cyberweek, and Journées de la Statistiques Rennaises (JSTAR).

10.1.5 Leadership within the scientific community

• Laurent Amsaleg is a member of the Steering Committee of ACM Multimedia for the 2020-2023 term
• Guillaume Gravier is a member of the scientific board of the GDR Traitement automatique des langues
• Guillaume Gravier is a referent on AI within Allistene representing the CPU
• Pascale Sébillot is a member of the board of the GDR Traitement automatique des langues

10.1.6 Scientific expertise

• Teddy Furon was a scientific expert for the call for projects ANR ASTRID.
• Ewa Kijak was a scientific expert for the call for projects ANR AAPG.
• Ewa Kijak was member of the selection committees in charge of recruiting an associate professor for INSA, ISTIC and Centrale-Supelec Paris
• Vincent Claveau was a reviewer for the Prize Paul Caseau (EDF – Académie des Technologies).
• Vincent Claveau was member of the selection committee in charge of recruiting an associate professor for Centrale-Supelec Paris
• Vincent Claveau was a reviewer for: the Emergence call for project of the MSH Paris-Saclay; the call for project of the LabEx CIMI ; the CIFRE PhD selection, ANRT.

10.1.7 Research administration

• Guillaume Gravier is director of IRISA (UMR 6074)
• Pascale Sébillot is deputy director of IRISA (UMR 6074)
• Guillaume Gravier is the scientific leader of the AI for Semantic Data Analytics (PNRIA) doctoral program for the Rennes site
• Guillaume Gravier is a member of the executive board of CREACH Labs, a general partnership agreement between the Brittany region, MINARM and higher education and research (ESR) institutions
• Guillaume Gravier is a member of the board of directors of the Images & Networks competitiveness cluster
• Teddy Furon est membre de la commission du personnel et président de la commission des délégations Inria.
• Pascale Sébillot is deputy director of the Scientific Advisory Committee of IRISA (UMR 6074)
• Pascale Sébillot was a member of the board of the MathSTIC doctoral school until August 2022

10.2.1 Teaching

• Licence: Pascale Sébillot, Natural Language Processing, 6h, L3, INSA Rennes, France
• Master: Laurent Amsaleg, Bases de données avancées, 25h, M2, INSA Rennes, France
• Master: Teddy Furon, Rare Event Simulations, 40h, INSA Rennes, France
• Master: Pascale Sébillot, Natural Language Processing, 6h, M1, INSA Rennes, France
• Engineering school: Vincent Claveau, Machine Learning, 18h, 3rd year, INSA Rennes, France
• Engineering school: Vincent Claveau, Natural Language Processing, 12h, ESIR, Univ. Rennes, France
• Licence: Guillaume Gravier, Base de données, 26h, L2, INSA Rennes
• Licence: Guillaume Gravier, Natural language processing, 12h, L3, INSA Rennes
• Licence: Guillaume Gravier, Markov models, 6h, L3, INSA Rennes
• Master: Guillaume Gravier, Natural Language Processing, 6h, M1, INSA Rennes
• Master: Guillaume Gravier, Natural Language Processing, 33h, M2, ENSAI
• Ewa Kijak is head of the Image engineering track (M1-M2) of ESIR, Univ. Rennes
• Master: Ewa Kijak, Supervised machine learning, 15h, M2R, Univ. Rennes
• Master: Ewa Kijak, Image retrieval, 12h, M2, ESIR
• Master: Ewa Kijak, Image classification, 27h, M1, ESIR
• Master: Ewa Kijak, Image processing, 45h, M1, ESIR, Univ. Rennes
• Master: Simon Malinowski, Basics of Data Analytics for Data Science, 24h, EIT Data Science Master 1, Rennes
• Master: Simon Malinowski, Prediction Methods, 30h, M1 MIAGE and Data Science EIT Master 1, Rennes
• Master: Simon Malinowski, Statisical Data Mining, 24h, M2 MIAGE, ISTIC, Rennes
• Master: Simon Malinowski, Symbolic Data Mining, 12h, M2 MIAGE, ISTIC, Rennes
• Simon Malinowski is responsible for the Master 2 MIAGE parcours Classique
• Simon Malinowski is responsible for the M2 studies within the DataScience track of the EIT-digital master school

10.2.2 Supervision

• PhD in progress: Shashanka Venkataramanan, Metric learning for instance- and category-level visual representations. Started in Dec. 2020. Yannis Avrithis, Ewa Kijak & Laurent Amsaleg
• PhD in progress: Deniz Engin, Video query answering in domestic environments. Started in Sept. 2020. Teddy Furon, Yannis Avrithis, Laurent Amsaleg
• PhD in progress: Raquel Almeida, Learning hierarchichal models for multimedia data. Started Jan. 2019, Ewa Kijak & Simon Malinowski & Laurent Amsaleg
• PhD in progress: Louis Hemadou, Domain generalization exploiting synthetic data. Université de Rennes, Started Nov. 2022. Ewa Kijak (with Frederic Jurie, GREYC, Caen)
• PhD in progress: Antoine Chaffin, Multimodal indexing and generation. Université de Rennes, Started Oct. 2020. Vincent Claveau, Ewa Kijak
• PhD in progress: Mohamed Younes, Learning and simulating strategies in sports for VR training. Started Dec. 2020. Ewa Kijak, Simon Malinowski (with Franck Multon and Richard Kulpa, Mimetic team)
• PhD in progress: Duc Hau Nguyen, Generation and justification of semantic links with attention mechanisms. Université de Rennes, Started Sept. 2020. Pascale Sébillot, Guillaume Gravier
• PhD in progress: Hugo Thomas, Zero-shot and few shot relation extraction in press archives. Université de Rennes. Started Oct. 2022. Pascale Sébillot, Guillaume Gravier
• PhD in progress: Paul Estano, Dynamic-Precision Training of Deep Neural Network Models on the Edge. Started Feb. 2022. Guillaume Gravier (with TARAN and DANTE project-teams, Silviu-Joan Filip and Elisa Riccietti)
• PhD in progress: Benoit Bonnet, Understanding, taming, and defending from adversarial examples. Started Nov. 2019. Teddy Furon (with Patrick Bas, CNRS CRIsTAL, Lille)
• PhD in progress: Thibault Maho, Black-box attacks against deep learning algorithms. Started Feb. 2022. Teddy Furon (with Erwan Le Merrer, team project WIDE)
• PhD in progress: Karim Tit, Robustness assessment of deep neural networks. Started Feb. 2021. Teddy Furon (with Mathias Rousset, team-project SIMSMART)
• PhD in progress: Samuel Tap, Learning in the encrypted domain. Started Mar. 2021. Teddy Furon.
• PhD in progress: Victor Kötzler, Membership Inference Attack. Started Feb. 2022. Teddy Furon (with Yufei Han, team-project CIDRE).
• PhD in progress: Quentin Le Roux, Security Assessment of pre-trained neural networks. Started Nov. 2022. Teddy Furon.
• PhD: Cyrielle Mallart, Dynamic and incremental construction of knowlege graphs with text mining. INSA Rennes. Defended Nov. 2022.

10.2.3 Juries

• Laurent Amsaleg was reviewer for the PhD of Zhengyu Zhao, Rethinking Realism: Advancing the Transferability and Imperceptibility of Adversarial Images, Radboud University, February.
• Teddy Furon was reviewer for the PhD of Quentin Giboulot (UTT Troyes), and of Amine Hmani (Telecom ParisTech).
• Teddy Furon was reviewer for the HDR of Adrien Chan Hon Tong (Onera).
• Vincent Claveau was a reviewer for the PhD of Miriam Benballa (LITIS, Normandie Univ.)
• Vincent Claveau was an examiner for the PhD of Eliott Maitre (IRIT, Univ. Toulouse 3)
• Pascale Sébillot was reviewer for the PhD of Quentin Portes Université Toulouse III, June 2022
• Pascale Sébillot was examiner for the PhD of Étienne Simon Sorbonne université, July 2022
• Pascale Sébillot was president for the PhD of Betty Fabre Université de Rennes 1, September
• Pascale Sébillot was examiner for the PhD of Luis-Gil Moreno-Jiménez Avignon université, November

10.3.1 Education

• Teddy Furon presented 7 classes Chiche! in Lycées Chateaubriand (Rennes) and Jean Bodin (Les Ponts de Cé).

10.3.2 Interventions

• Vincent Claveau was interviewed by BiKiNi, TV Rennes, Cercle de la Presse, France Bleu Armorique
• Vincent Claveau participated in roundtable discussions: i-expo, Paris and ERRIE, Univ. Rennes

International journals

• 4 articleR.Raquel Almeida, E.Ewa Kijak, S.Simon Malinowski, Z. K.Zenilton K.G. Patrocínio Jr, A.Arnaldo Araújo and S. J.Silvio J.F. Guimarães. Graph-based image gradients aggregated with random forests.Pattern Recognition Letters2022
  HALDOIback to text
• 5 articleY. B.Yohan Bonescki Gumiel, L. E.Lucas Emanuel Silva E Oliveira, V.Vincent Claveau, N.Natalia Grabar, E. C.Emerson Cabrera Paraiso, C.Claudia Moro and D. R.Deborah Ribeiro Carvalho. Temporal Relation Extraction in Clinical Texts.ACM Computing Surveys547September 2022, 1-36
  HALDOIback to text
• 6 articleG.Gauthier Lyan, D.David Gross-Amblard, J.-M.Jean-Marc Jézéquel and S.Simon Malinowski. Impact of Data Cleansing for Urban Bus Commercial Speed Prediction.SN Computer Science3822022, 1-11
  HALDOIback to text
• 7 articleG.Gauthier Lyan, J.-M.Jean-Marc Jézéquel, D.David Gross-Amblard, R.Romain Lefeuvre and B.Benoit Combemale. Reasoning over Time into Models with DataTime.Software and Systems ModelingDecember 2022, 1-25
  HALback to text

International peer-reviewed conferences

• 8 inproceedingsL.Lola Beuchée, T.Thomas Guyet and S.Simon Malinowski. Prédiction du niveau de nappes phréatiques : comparaison d'approches locale, globale et hybride.EGC 2022 - Conférence francophone sur l'Extraction et la Gestion des ConnaissancesBlois, FranceJanuary 2022
  HALback to text
• 9 inproceedingsA.Antoine Chaffin, V.Vincent Claveau and E.Ewa Kijak. PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding.NAACL 2022 - Conference of the North American Chapter of the Association for Computational Linguistics: Human Language TechnologiesHybrid: Seattle, Washington + Online, United StatesJuly 2022, 1-15
  HALback to textback to text
• 10 inproceedingsA.Antoine Chaffin, T.Thomas Scialom, S.Sylvain Lamprier, J.Jacopo Staiano, B.Benjamin Piwowarski, E.Ewa Kijak and V.Vincent Claveau. Which Discriminator for Cooperative Text Generation?SIGIR 2022 - 45th International ACM SIGIR Conference on Research and Development in Information RetrievalMadrid, SpainACMJuly 2022, 2360-2365
  HALDOIback to text
• 11 inproceedingsP.Pierre Fernandez, A.Alexandre Sablayrolles, T.Teddy Furon, H.Hervé Jégou and M.Matthijs Douze. Tatouage Numérique d'Images dans l'Espace Latent de Réseaux Auto-Supervisés.GRETSI 2022 - Colloque Francophone de Traitement du Signal et des ImagesNancy, FranceSeptember 2022, 1-4
  HALback to text
• 12 inproceedingsP.Pierre Fernandez, A.Alexandre Sablayrolles, T.Teddy Furon, H.Hervé Jégou and M.Matthijs Douze. Watermarking Images in Self-Supervised Latent Spaces.ICASSP 2022 - IEEE International Conference on Acoustics, Speech and Signal ProcessingSingapore, SingaporeIEEEMay 2022, 1-5
  HALback to text
• 13 inproceedingsT.Teddy Furon, B.Benoit Bonnet and P.Patrick Bas. Adversarial images with downscaling.ICIP 2022 - 29th IEEE International Conference on Image ProcessingBordeaux, FranceIEEEOctober 2022, 1-5
  HALback to text
• 14 inproceedingsT.Thomas Guyet, L.Laurent Spillemaecker, S.Simon Malinowski and A.-I.Anne-Isabelle Graux. Temporal Disaggregation of the Cumulative Grass Growth.Lecture Notes in Computer Science. LNCS 13364. Part IIICPRAI 2022 - 3rd International Conference on Pattern Recognition and Artificial Intelligence13364Pattern Recognition and Artificial IntelligenceParis, FranceSpringer International PublishingMay 2022, 383-394
  HALDOIback to text
• 15 inproceedingsK.Kassem Kallas and T.Teddy Furon. RoSe: A RObust and SEcure Black-Box DNN Watermarking.WIFS 2022 - IEEE 14th International Workshop on Information Forensics and SecurityShanghai, ChinaIEEEDecember 2022, 1-5
  HALback to text
• 16 inproceedingsS.Sylvain Lamprier, T.Thomas Scialom, A.Antoine Chaffin, V.Vincent Claveau, E.Ewa Kijak, J.Jacopo Staiano and B.Benjamin Piwowarski. Generative Cooperative Networks for Natural Language Generation.Proceedings of Machine Learning ResearchICML 2022 - 39th International Conference on Machine Learning162International Conference on Machine Learning, 17-23 July 2022, Baltimore, Maryland, USABaltimore, MD, United StatesPMLR2022, 11891--11905
  HALback to text
• 17 inproceedingsT.Thibault Maho, T.Teddy Furon and E.Erwan Le Merrer. FBI: Fingerprinting models with Benign Inputs.CAID 2022 - Conference on Artificial Intelligence for DefenseActes de la 4ème Conference on Artificial Intelligence for Defense (CAID 2022)Rennes, FranceNovember 2022
  HALback to text
• 18 inproceedingsT.Thibault Maho, T.Teddy Furon and E.Erwan Le Merrer. Randomized Smoothing under Attack: How Good is it in Pratice?ICASSP 2022 - IEEE International Conference on Acoustics, Speech and Signal ProcessingSingapore, SingaporeIEEEMay 2022, 1-5
  HALback to text
• 19 inproceedingsC.Cyrielle Mallart, M.Michel Le Nouy, G.Guillaume Gravier and P.Pascale Sébillot. Confronting Active Learning for Relation Extraction to a Real-life Scenario on French Newspaper Data.InterNLP 2022 - 2nd Workshop on Interactive Learning for Natural Language ProcessingProceedings of 2nd Workshop on Interactive Learning for Natural Language Processing, InterNLP@NeurIPS 2022New Orleans, United StatesDecember 2022, 1-7
  HALback to text
• 20 inproceedingsL.Laurent Spillemaecker, T.Thomas Guyet, S.Simon Malinowski and A.-I.Anne-Isabelle Graux. Désagrégation temporelle du cumul annuel de croissance de l'herbe.EGC 2022 - Conférence francophone sur l'Extraction et gestion des connaissancesRNTI-E-38EGC 2022Blois, France2022, 27-38
  HALback to text
• 21 inproceedingsS.Shashanka Venkataramanan, E.Ewa Kijak, L.Laurent Amsaleg and Y.Yannis Avrithis. AlignMixup: Improving Representations By Interpolating Aligned Features.CVPR 2022 - IEEE/CVF Conference on Computer Vision and Pattern RecognitionNew Orleans, United StatesIEEEJune 2022, 1-13
  HALback to text
• 22 inproceedingsS.Shashanka Venkataramanan, B.Bill Psomas, E.Ewa Kijak, L.Laurent Amsaleg, K.Konstantinos Karantzalos and Y.Yannis Avrithis. It Takes Two to Tango: Mixup for Deep Metric Learning.ICLR 2022 - 10th International Conference on Learning RepresentationsVirtual, FranceApril 2022, 1-21
  HALback to text

National peer-reviewed Conferences

• 23 inproceedingsA.Antoine Chaffin, V.Vincent Claveau and E.Ewa Kijak. Discriminator-guided decoding with Monte Carlo Tree Search for constrained text generation.Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principaleTALN 2022 - 29e conférence sur le Traitement Automatique des Langues NaturellesAvignon, FranceATALA2022, 27-41
  HALback to text
• 24 inproceedingsA.Antoine Chaffin, T.Thomas Scialom, S.Sylvain Lamprier, J.Jacopo Staiano, B.Benjamin Piwowarski, E.Ewa Kijak and V.Vincent Claveau. Choosing The Right Teammate For Cooperative Text Generation.Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principaleTALN 2022 - 29e conférence sur le Traitement Automatique des Langues NaturellesAvignon, FranceATALA2022, 12-26
  HALback to text
• 25 inproceedingsL.Loïc Fosse, D.-H.Duc-Hau Nguyen, P.Pascale Sébillot and G.Guillaume Gravier. Une étude statistique des plongements dans les modèles transformers pour le français.Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principaleTraitement Automatique des Langues Naturelles (TALN 2022)Avignon, FranceATALA2022, 247-256
  HALback to text
• 26 inproceedingsD.Duc Hau Nguyen, G.Guillaume Gravier and P.Pascale Sébillot. Filtrage et régularisation pour améliorer la plausibilité des poids d'attention dans la tâche d'inférence en langue naturelle.Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principaleTALN 2022 - Traitement Automatique des Langues NaturellesAvignon, FranceATALA2022, 95-103
  HALback to text

Scientific book chapters

• 27 inbookT.Teddy Furon. Traitor tracing.Multimedia securityMarch 2022
  HALback to text
• 28 inbookH.Hanwei Zhang, T.Teddy Furon, L.Laurent Amsaleg and Y.Yannis Avrithis. Deep Neural Network Attacks and Defense: The Case of Image Classification.Multimedia Security 1Multimedia Security 1: Authentication and Data Hiding1WileyMarch 2022
  HALDOIback to text

Reports & preprints

• 29 miscM.Mathieu Chambe, E.Ewa Kijak, Z.Zoltan Miklos, O.Olivier Le Meur, R.Rémi Cozot and K.Kadi Bouatouch. HDR-LFNet: Inverse Tone Mapping using Fusion Network.March 2022
  HAL

Other scientific publications

• 30 inproceedingsM.Mohamed Younes, E.Ewa Kijak, R.Richard Kulpa, S.Simon Malinowski and F.Franck Multon. AIP: Adversarial Interaction Priors for Multi-Agent Physics-based Character Control.SIGGRAPH Asia 2022 - 15th ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in AsiaSIGGRAPH Asia 2022 PostersDaegu, South KoreaDecember 2022, 2
  HALDOIback to text