Section: New Software and Platforms

Joint object-action learning

Joint learning of object and action detectors

Keywords: Detection - Video sequences - Zero-shot

Scientific Description: we propose to jointly detect object-action instances in uncontrolled videos, e.g. cat eating, dog running or car rolling. We build an end-to-end two stream network architecture for joint learning of objects and actions. We cast this joint learning problem by leveraging a multitask objective. We compare our proposed end-to-end multitask architecture with alternative ones: (i) treating every possible combination of actions and objects as a separate class (Cartesian) and (ii) considering a hierarchy of objects-actions: the first level corresponds to objects and the second one to the valid actions for each object (hierarchical). We show that our method performs as well as these two alternatives while (a) requiring fewer parameters and (b) enabling zero-shot learning of the actions performed by a specific object, i.e., when training for an object class alone without its actions, our multitask network is able to predict actions for that object class by leveraging actions performed by other objects. our multitask objective not only allows to effectively detect object-action pairs but also leads to performance improvements on each individual task (i.e., detection of either objects or actions). We compare to the state of the art for object-action detection on the Actor-Action (A2D) dataset and we outperform it.

Functional Description: While most existing approaches for detection in videos focus on objects or human actions separately, we aim at jointly detecting objects performing actions, such as cat eating or dog jumping. We introduce an end-to-end multitask objective that jointly learns object-action relationships. We compare it with different training objectives, validate its effectiveness for detecting objects-actions in videos, and show that both tasks of object and action detection benefit from this joint learning. Moreover, the proposed architecture can be used for zero-shot learning of actions: our multitask objective leverages the commonalities of an action performed by different objects, e.g. dog and cat jumping , enabling to detect actions of an object without training with these object-actions pairs. In experiments on the A2D dataset, we obtain state-of-the-art results on segmentation of object-action pairs. We finally apply our multitask architecture to detect visual relationships between objects in images of the VRD dataset.

• Participants: Vasiliki Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari and Cordelia Schmid

• Contact: Vasiliki Kalogeiton

• Publication: Joint learning of object and action detectors

• URL: https://github.com/vkalogeiton/joint-object-action-learning