Gaussian Shell Maps for Efficient 3D Human Generation

Rameen Abdal*1, Wang Yifan*1, Zifan Shi*2, Yinghao Xu1, Ryan Po1, Zhengfei Kuang1, Qifeng Chen2 Dit-Yan Yeung2, Gordon Wetzstein1
CVPR 2024
1Stanford University, 2HKUST
* Equal Contribution
Paper arXiv Code
Gaussian Shell Maps: Gaussian Shell Maps is an efficient framework for 3D human generation connecting 3D Gaussians with CNN-based generators. 3D Gaussians are anchored to “shells” derived from the SMPL template (only two shells are visualized for clarity), and the appearance is modeled in texture space. Trained only on 2D images, we show that our method can generate diverse articulable humans in real-time with state-of-the-art quality directly in high resolution without the need for upsampling and hence avoiding aliasing artifacts.
Method Overview : We propose an expressive yet highly efficient representation, Gaussian Shell Map (GSM), for 3D human generation. Combining the idea of 3D Gaussians and Shell Maps, we sample 3D Gaussians on “shells”, which are mesh layers offsetted from the SMPL template, forming a shell volume to model complex and diverse geometry and appearance; the Gaussian parameters are learned in the texture space, allowing us to leverage existing CNN-based generative architecture. Articulation is straightforward by interpolating the deformation of the shell. The generation is supervised by single-view 2D images using several discriminator critics, including part-specific face, hands, and feet discriminators.

Overview





Poses



Semantic Editing




Interpolations





Animations




Walking sequence


Jumping sequence


Swing sequence



Novel Views




Ours



AG3D



BibTeX

@misc{abdal2023gaussian,
      title={Gaussian Shell Maps for Efficient 3D Human Generation}, 
      author={Rameen Abdal and Wang Yifan and Zifan Shi and Yinghao Xu and Ryan Po and Zhengfei Kuang and Qifeng Chen and Dit-Yan Yeung and Gordon Wetzstein},
      year={2023},
      eprint={2311.17857},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}