Section: New Results

Video segmentation with shape constraint

Participant : Guillaume Charpiat.

keywords: video segmentation, graph-cut, shape growth, shape statistics, shape prior, dynamic time warping

Video segmentation with growth constraint

This is joint work with Yuliya Tarabalka (Ayin Inria team) and Björn Menze (ETH Zurich, also MIT and collaborator of Asclepios Inria team).

Context : One of the important challenges in computer vision is the automatic segmentation of objects in videos. This task becomes more difficult when image sequences are subject to low signal-to-noise ratio or low contrast between intensities of neighboring structures in the image scene. Such challenging data is acquired routinely, for example in medical imaging or in satellite remote sensing. While individual frames could be analyzed independently, temporal coherence in image sequences provides crucial information to make the problem easier. In this work, we focus on segmenting shapes in image sequences which only grow or shrink in time, and on making use of this knowledge as a constraint to help the segmentation process.

Approach and applications : We had proposed last year an approach based on graph-cut (see Figure 15 ), able to obtain efficiently (linear time in the number of pixels in practice), for any given video, its globally-optimal segmentation satisfying the growth constraint. This year we applied this method to three different applications :

• forest fires in satellite images,

• organ development in medical imaging (brain tumor, in multimodal MRI 3D volumes),

• sea ice melting in satellite observation, with a shrinking constraint instead of growth (see Figure 16 ).

The results on the first application were published in IGARSS (International Geoscience and Remote Sensing Symposium) [48] , while the last two applications and the theory were published in BMCV [47] . A journal paper is also currently under review. A science popularization article was also published [53] . Not related but also with the Ayin Inria team was published the last of a series of articles about optimizers for point process models [40] , introducing graph-cuts in the multiple birth and death approach in order to detect numerous objects that should not overlap.

Video segmentation with statistical shape prior

This is joint work with Maximiliano Suster (leader of the Neural Circuits and Behaviour Group at Bergen University, Norway).

Context : The zebrafish larva is a model organism widely used in biology to study genetics. Therefore, analyzing its behavior in video sequences is particularly important for this research field. For this, there is a need to segment the animal in the video, in order to estimate its speed, and also more precisely to extract its shape, in order to express for instance how much it is bent, how fast it bends, etc. However, as the animal is stimulated by the experimenter with a probe, the full zebrafish larva is not always visible because of occlusion.

Approach : We build a shape prior based on a training set of examples of non-occluded shapes, and use it to segment new images where the animal is occluded. This is however not straightforward.

• Building a training set of shape deformations : Given a set of training images containing non-occluded animals, we extract their contours via multiple robust thresholdings and morphomathematical operations. For each contour, we then estimate automatically the location of the tip of the tail. We then compute point-to-point correspondences between all contours, using a modified version of Dynamic Time Warping, as well as the approximate tip location information. This is done in a translation- and rotation-invariant way.

• Building the shape prior : Based on these matchings, the mean shape is computed, as well as modes of deformation with PCA.

• Segmenting occluded images : Images with occluded shapes are pre-processed in a similar way to non-occluded ones; however, the resulted segmentation does not contain only the parts of the larva but also the probe, which has potentially similar colors and location, and is moving. To identify the probe, whose shape depends on the video sequence, we make use of its rigidity and of temporal coherency. Then a segmentation criterion is designed to push an active contour towards the zones of interest (in a way that is robust to initialization), while keeping a shape which is feasible according to the shape prior.

Examples of data and results for a preliminary algorithm are shown in Figure 17 , with the associated shape prior shown in Figure 18 .