Summary
Real-time 3D avatar editing like a video game character creator.
- Disentangled Latent Editing: one separate latent space for hair, face, torso, and glasses
- Color Control: We can precisely change the color of the hair, the skin, and the torso
- Shape and Light: We can control the shape and light without using any prior annotation
Motivation
Recent 3D Gaussian Splatting (3DGS) GANs for human heads synthesize and render photorealistic 3D models in real-time and offer a vast variety in identity and appearance. However, controlling specific semantic attributes such as hair color or glasses remains challenging, as edits in the entangled latent space often induce unintended changes in identity or appearance. Although there are several methods that aim to disentangle the latent space post-training by estimating directions that only modify certain features, these methods cannot guarantee complete disentanglement and often require pre-trained classifiers.
Method
We propose a new generator architecture that synthesizes components, such as hair, skin, glasses, and torso, completely independently. This allows for changing the latent vector for one region while keeping the remaining parts fixed. Further, we achieve this separation using only sparse information such as the hair or skin color, eliminating the requirement of segmentation masks or geometric priors, often seen in prior work. To ensure matching shape and lighting conditions during editing, we allow minimal shared information via context tokens between the independent generators. These tokens even allow us to control the shape and light, without any prior annotation.
Results
| Method | FID |
|---|---|
| GSGAN | 7.68 |
| GGHead | 7.78 |
| CGS-GAN | 4.53 |
| Ours | 4.42 |
Light & Shape Context
The generator implicitly learns light and shape attributes without any annotation.
Light
Shape
Recolor
We are able to directly modulate the output colors of each component. This even allows us to apply out-of-distribution colors, which usually break the synthesis using latent space editing methods.
Editor
Create your own 3D avatar in ganviz.
Latent Space
We can apply GAN Space on each of the sub-latent spaces to achieve precise editing control.
This work has been a collaboration between the Fraunhofer Heinrich-Hertz Institute, Humboldt University Berlin, and NVIDIA.
