Section: Research Program

3D object and scene modeling, analysis, and retrieval

This part of our research focuses on geometric models of specific 3D objects at the local (differential) and global levels, physical and statistical models of materials and illumination patterns, and modeling and retrieval of objects and scenes in large image collections. Our past work in these areas includes research aimed at recognizing rigid 3D objects in cluttered photographs taken from arbitrary viewpoints (Rothganger et al., 2006), segmenting video sequences into parts corresponding to rigid scene components before recognizing these in new video clips (Rothganger et al., 2007), retrieval of particular objects and buildings from images and videos (Sivic and Zisserman, 2003) and (Philbin et al., 2007), and a theoretical study of a general formalism for modeling central and non-central cameras using the formalism and terminology of classical projective geometry (Ponce, 2009 and Batog et al., 2010).

We have also developed multi-view stereopsis algorithms that have proven remarkably effective at recovering intricate details and thin features of compact objects and capturing the overall structure of large-scale, cluttered scenes. We have obtained a US patent 8,331,615 (The patent: "Match, Expand, and Filter Technique for Multi-View Stereopsis" was issued December 11, 2012 and assigned patent number 8,331,615.) for the corresponding software (PMVS, https://github.com/pmoulon/CMVS-PMVS ) which is available under a GPL license and used for film production by ILM and Weta as well as by Google in Google Maps. It is also the basic technology used by Iconem, a start-up founded by Y. Ubelmann, a Willow collaborator. We have also applied our multi-view-stereo approach to model archaeological sites together with developing representations and efficient retrieval techniques to enable matching historical paintings to 3D models of archaeological sites (Russel et al., 2011).

Our current efforts in this area, outlined in detail in Section  7.1 , are focused on: (i) continuing our theoretical study of multi-view camera geometry [17] , [25] . (ii) modelling new representations of large-scale visual place recognition in structured image collections of urban environments [16] , and (iii) developing new weakly supervised and deep learning approaches to large scale place recognition and retrieval [21] .