SDM-NET: Deep Generative Network for Structured Deformable Mesh

Lin Gao¹          Jie Yang¹          Tong Wu¹          Yu-Jie Yuan¹

Hongbo Fu²       Yu-Kun Lai³       Hao(Richard) Zhang⁴

¹Institute of Computing Technology, Chinese Academy of Sciences

²City University of Hong Kong       ³Cardiff University       ⁴Simon Fraser University

Accepted by Siggraph Asia 2019

Figure: Our deep generative neural network, SDM-NET, produces structured meshes composed of deformable parts. Part structures and geometries are jointly encoded into a latent space by an autoencoder, enabling quality 3D shape generation. We show shape interpolation results exhibiting flexible structure and fine geometric details. This is achieved by linearly interpolating airplane and chair latent codes and then reconstruction from the in-between codes.

Abstract

We introduce SDM-NET, a deep generative neural network which produces structured deformable meshes. Specifically, the network is trained to generate a spatial arrangement of closed, deformable mesh parts, which respect the global part structure of a shape collection, e.g., chairs, airplanes, etc. Our key observation is that while the overall structure of a 3D shape can be complex, the shape can usually be decomposed into a set of parts, each homeomorphic to a box, and the finer-scale geometry of the part can be recovered by deforming the box. The architecture of SDM-NET is that of a two-level variational autoencoder (VAE). At the part level, a PartVAE learns a deformable model of part geometries. At the structural level, we train a Structured Parts VAE (SP-VAE), which jointly learns the part structure of a shape collection and the part geometries, ensuring a coherence between global shape structure and surface details. Through extensive experiments and comparisons with the state-of-the-art deep generative models of shapes, we demonstrate the superiority of SDM-NET in generating meshes with visual quality, flexible topology, and meaningful structures, which benefit shape interpolation and other subsequently modeling tasks.

Paper

SDM-NET: Deep Generative Network for Structured Deformable Mesh

[ArXiv Preprint]

Code

[Link]

Data

[Link]

Methodology

Encoding of Shape Part

Figure: Encoding of each shape part, including its structure and geometry information. The former includes both the support and symmetry information, and the latter describes the deformation of the bounding box, compactly represented as the latent vector of PartVAE.

PartVAE for Encoding Part Geometry

Figure: Architecture of PartVAE for encoding the geometry details of a part represented as the deformation of the associated template box. V is the number of vertices in the template box. N(0, I) is the Gaussian distribution with 0 mean and identity covariance.

Supporting Structure Analysis

Figure: Illustration of support and symmetry relations between parts of an airplane model. The support relations, detected by the approach in [Huang et al. 2015b], turn an undirected adjacency graph (Middle) to a directed support graph (Right).

SP-VAE for Structured Deformable Mesh Encoding

Figure: Architecture of Structured Parts VAE (SP-VAE). The detailed geometry is encoded by PartVAE. The support- and symmetry-induced structure and the associated latent code of PartVAE are encoded by the Structured Parts VAE.

Shape Interpolation

Figure: Shape interpolation of cups and chairs with different topologies using our method. Interpolating cars with different geometry using our method. The first and last columns are the input shapes for interpolation.

Shape Generation

Figure: Random generation of monitor and chair shapes using our method, where 990 the training data is from ModelNet [Wu et al. 2015].

Shape Editing

Figure: Our generated Structure Deformable Mesh is directly editable. We show an example editing sequence. (a) is the decoded shape of our method. After applying the deletion operation (b), we obtain a chair without armrests (c). Dragging a leg of the chair makes all four legs longer due to the equal length constraint (d). Finally, we deform the back of the chair to obtain (e).

Additional Materials

Video

BibTex