Texturify: Generating Textures on 3D Shape Surfaces

1Technical University of Munich, 2Max Planck Institute for Intelligent Systems, 3Apple

Texturify learns to generate geometry-aware textures for untextured collections of 3D objects. Our method trains from only a collection of images and a collection of untextured shapes, which are both often available, without requiring any explicit 3D color supervision or shape-image correspondence. Textures are created directly on the surface of a given 3D shape, enabling generation of high-quality, compelling textured 3D shapes.

Press G to toggle between geometry and Texturified meshes. Press R to reset view.

Abstract

Texture cues on 3D objects are key to compelling visual representations, with the possibility to create high visual fidelity with inherent spatial consistency across different views. Since the availability of textured 3D shapes remains very limited, learning a 3D-supervised data-driven method that predicts a texture based on the 3D input is very challenging.

We thus propose Texurify, a GAN-based method that leverages a 3D shape dataset of an object class and learns to reproduce the distribution of appearances observed in real images by generating high-quality textures.

In particular, our method does not require any 3D color supervision or correspondence between shape geometry and images to learn the texturing of 3D objects. Texurify operates directly on the surface of the 3D objects by introducing face convolutional operators on a hierarchical 4-RoSy parameterization to generate plausible object-specific textures. Employing differentiable rendering and adversarial losses that critique individual views and consistency across views, we effectively learn the high-quality surface texturing distribution from real-world images.

Experiments on car and chair shape collections show that our approach outperforms state of the art by an average of 22% in FID score.

Video

Latent Interpolation

The texture latent space learned by our method produces smoothly-varying valid textures when traversing across the latent space for a fixed shape.

Interpolate start reference image.

Start Frame

Loading...
Interpolation end reference image.

End Frame

Interpolate start reference image.

Start Frame

Loading...
Interpolation end reference image.

End Frame

Style Consistency

The learned latent space is consistent in style across different shapes, i.e. the same code represents a similar style across shapes, and can be used, for example, in style transfer applications.

[ Press G to toggle between geometry and Texturified meshes. Press R to reset view. ]

Method Overview

Surface features from an input 3D mesh are encoded through a face convolution-based encoder and decoded through a StyleGAN2-inspired decoder to generate textures directly on the surface of the mesh.

To ensure that generated textures are realistic, the textured mesh is differentiably rendered from different view points and is critiqued by two discriminators.

An image discriminator DI operates on full image views from the real or rendered views, while a patch-consistency discriminator DP encourages consistency between views by operating on patches coming from a single real view or patches from different views of rendered images.

Related Links

For more work on similar tasks, please check out

Learning Texture Generators for 3D Shape Collections from Internet Photo Sets learns a texture generator for 3D objects from 2D image collections using a common UV parameterization.

Texture Fields: Learning Texture Representations in Function Space learns a texture generator parameterized as an implicit field.

SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans tackles scene completion both in terms of geometry and texture using a 3D grid parameterization.

Text2Mesh: Text-Driven Neural Stylization for Meshes optimizes color and geometric details over a variety of source meshes, driven by a target text prompt.

TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes, our motivation for using 4-RoSy parameterization, introduces a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes, and show its application on point cloud segmentation.

BibTeX

@inproceedings{siddiqui2022texturify,
  author    = {Yawar Siddiqui and
               Justus Thies and
               Fangchang Ma and
               Qi Shan and
               Matthias Nie{\ss}ner and
               Angela Dai},
  editor    = {Shai Avidan and
               Gabriel J. Brostow and
               Moustapha Ciss{\'{e}} and
               Giovanni Maria Farinella and
               Tal Hassner},
  title     = {Texturify: Generating Textures on 3D Shape Surfaces},
  booktitle = {Computer Vision - {ECCV} 2022 - 17th European Conference, Tel Aviv,
               Israel, October 23-27, 2022, Proceedings, Part {III}},
  series    = {Lecture Notes in Computer Science},
  volume    = {13663},
  pages     = {72--88},
  publisher = {Springer},
  year      = {2022},
  url       = {https://doi.org/10.1007/978-3-031-20062-5\_5},
  doi       = {10.1007/978-3-031-20062-5\_5},
  timestamp = {Tue, 15 Nov 2022 15:21:36 +0100},
  biburl    = {https://dblp.org/rec/conf/eccv/SiddiquiTMSND22.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}