Abstract:
In semantic 3D modelling the goal is to find a dense geometric model from images and at the same time also infer the semantic classes of the individual parts of the reconstructed model. Having a semantically annotated dense 3D model gives a much richer representation of the scene than just the geometry. For example questions such as what is the volume of a building can directly be answered. This is difficult with just a geometric model where the knowledge about which parts of the geometry belong to the building is not present. Also by solving the problem of dense 3D reconstruction and class segmentation jointly, prior knowledge such as the ground is usually a surface which is close to horizontal can be included.

In this tutorial, we use a volumetric representation of the scene. Traditionally, this is done as a two label problem where each voxel gets label into either being in the free space or in the occupied space. For our semantic formulation we extend the representation to a multi-label problem. A voxel either belongs to the free space or to one out of multiple semantic classes describing the occupied space. While such a formulation is quite memory intensive it allows for a very rich description of the scene. For example also not directly observed surfaces between semantic labels can be represented, for example the ground can extend underneath a building. Priors on the surface orientation, that are learnt from training data, are used to faithfully fill in such surfaces between different semantic labels.

In a first part we will introduce the multi-label formulation we are solving to facilitate semantic 3D modelling. The multi-label problem is formulated as a convex optimization problem. The domain gets treated as a continuous space which allows for fully isotropic penalization of surface areas. This reduces the artefacts of the discrete nature of the image pixel grid. In the original formulation only isotropic smoothness terms which need to form a metric over the label space were possible. We will present an extension of these formulations to allow for non-metric and anisotropic surface penalization at the same time. The multi-label formulation is then used for our joint formulation of the dense 3D reconstruction and class segmentation. Furthermore, we will show how 3D object shape priors can be modelled.

Description:
This tutorial aims at giving a panorama of algorithms used to create, manipulate and optimize 3D meshes, with a focus on some representative ones. Starting from raw 3D pointsets, I will describe a set of algorithms that can be used to reconstruct a mesh and then optimize it for different usages, such as visualization, evaluation of some shape indicators and numerical computations.

The tutorial will be illustrated with examples, all available as code sources in my Geogram/Graphite software. I will describe the implementation details with the “tips and tricks” that can make these algorithms efficient, even on multi-million points/polygons datasets. I will also describe several techniques to deal with robustness issues, develop computer arithmetics in arbitrary precision, and discuss whether it is necessary to use it or not.

Introduction
Meshes: what/why/how

I) From points to meshes: surface reconstruction
– Several categories of surface reconstruction methods
– Focus on methods that use a Delaunay triangulation
– Crust and Co-Cone methods, and their variants

II) Improving the mesh: re-meshing methods
– Several categories of surface remeshing algorithms
– Using derivatives: focus on mesh parameterization
– Using integrals: focus on centroidal Voronoi tesselation

III) Implementation – the gritty details
– Make it as simple as possible (but not simpler)
– Halfedges and edgeuses, useful or harmful ?
– Robustness and computer arithmetics

Speaker: Bruno Levy is a researcher in geometry processing and applied mathematics. He leads the ALICE project team in INRIA Nancy Grand-Est and in the LORIA lab. His research topic is numerical geometry, with applications in numerical simulation, real-time rendering and scientific visualization. Some of his results (e.g., LSCM) are used in several commercial and open-source softwares. He defended his Ph.D. thesis in 1999 and received the Gilles Kahn /Académie des Sciences SPECIF national award in 2000. Then he did a post-doc in Stanford. He was program co-chair of ACM SPM in 2007 and 2008, ACM/EG SGP in 2010, Pacific Graphic in 2012 and Eurographics in 2014. He was associate editor of TVCG (IEEE) and Graphical Models (Elsevier). He was awarded a grant from the European Research Council in 2008, received the Lorraine regional young researcher award in 2010 and the INRIA national young researcher award in 2011. Since 2013 he focuses on problems from applied mathematics, and develops new numerical methods to solve non-linear partial derivative equations. He developed the first efficient algorithm to solve the Monge-Ampere equation in 3D (published in ESAIM Mathematical Modeling and Analysis).

Abstract: Computer vision research for movies and theme park productions at Disney poses unique challenges in terms of reliability and performance of the algorithms and quality of results.For instance, many existing strategies for image-based 3D reconstruction or video processing have been carefully designed towards sparsely sampled, heterogeneous image data. Such techniques (involving, e.g., complex global optimization) usually become fundamentally impractical to apply to content captured with a dense array of modern cinema cameras at 120 fps at 4k resolution and beyond. At the same time, these methods ignore novel challenges and opportunities arising from extremely dense input. In this talk, I will present our ongoing work in the Imaging and Video Group at Disney Research on 3D scene modeling and processing from densely sampled videos and light fields, which breaks with a number of established practices in computer vision in order to achieve result quality and speed unparalleled by existing methods.

Bio: Alexander Sorkine-Hornung is Senior Research Scientist at Disney Research Zurich, heading the Imaging and Video group. Before joining Disney, Alexander was a postdoctoral researcher at the Computer Graphics Laboratory at ETH Zurich. He obtained his PhD in Computer Science at RWTH Aachen in 2008. Alexander’s research interests lie in all areas related to digital image and video processing. A representative list of topics includes light field and video processing, big data imaging, image-based 3D reconstruction and rendering, color processing, stereoscopy, and visual saliency analysis. Furthermore, he is interested in innovative applications of image-based technologies in the fields of 2D animation and interactive environments. In 2012 Alexander received the Eurographics Young Researcher Award. The research and technologies developed by his group already had significant impact on various Disney park attractions and movie productions, with film credits on movies such as Maleficent, Cinderella, and Big Hero 6.

http://zurich.disneyresearch.com/~ahornung/

Abstract: Through examples such as 123d® Catch®, Autodesk®Recap® or Memento®, we will show how 3D vision has surfaced on different markets and in different products at Autodesk, and has become a reality for many users. Through the lens of concrete applications and use cases, we will take a look at the current status and upcoming challenges for 3D vision.

Bio: Luc Robert obtained his PhD in 1993 from Inria/Ecole Polytechnique. After a 1-year post-doc at Carnegie Mellon University, he joined Inria in 1995 as a Research Scientist. In 1998, he started the REALVIZ company, acquired by Autodesk in 2008. He is now Senior Software Architect for the Reality Solutions department at Autodesk, that develops technologies and products related to photogrammetry and 3D scanning.

http://www.autodesk.com

Abstract: Over the past decade advances in computer vision have enabled the 3D reconstruction of dynamic scenes from multiple view video. This has allowed video-based free-viewpoint rendering with the photo-realism of video whilst allowing interactive viewpoint control. This technology initially pioneered for highly controlled indoor scenes has been extended to free-viewpoint rendering of large-scale outdoor scenes such as sports for TV production. Free-viewpoint video content is limited to the replay of the captured performance. This talk will present results of recent research in 4D vision research for actor performance capture which allows both video-realistic free-viewpoint rendering and interactive control of movement. Recent research has introduced methods for spatio-temporal alignment and parametric representation of dynamic shape and appearance from capture performance to allow interactive control whilst maintaining the realism of the captured video. This opens-up the potential for reuse of 4D performance capture to create video-realistic characters for immersive entertainment. This talk will review recent advances and identify future research challenges for 4D vision in entertainment and human motion analysis.

Bio: Adrian Hilton, BSc(hons),DPhil,CEng, is Professor of Computer Vision and Graphics and Director of the Centre for Vision, Speech and Signal Processing at the University of Surrey, UK. He leads research investigating the use of computer vision for applications in entertainment content production, visual interaction and clinical analysis.
His interest is in robust computer vision to model and understand real world scenes and his work in bridging-the-gap between real and computer generated imagery combines the fields of computer vision, graphics and animation to investigate new methods for reconstruction, modelling and understanding of the real world from images and video. Applications include: sports analysis (soccer, rugby and athletics), 3D TV and film production, visual effects, character animation for games, digital doubles for film and facial animation for visual communication. Contributions include technologies for the first hand-held 3D scanner, modeling of people from images and 3D video for games, broadcast and film production. Current research is focused on video-based measurement in sports, multiple camera systems in film and TV production, and 3D video for highly realistic animation of people and faces. Research is conducted in collaboration with UK companies and international institutions in the creative industries
Adrian is currently the Principal Investigator of the multi-million EPSRC Progamme Grant S3A: ‘Future Spatial Audio for Immersive Listener Experience at Home’ (2013-2018), he also leads several EU and UK/ industry projects. Adrian currently holds a 5-year Royal Society Wolfson Research Merit Award (2013-2018).

Adrian Hilton’s Website