Conference proceeding
Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis
COMPUTER VISION - ECCV 2022, PT XVII, v 13677, pp 236-253
01 Jan 2022
Featured in Collection : UN Sustainable Development Goals @ Drexel
Abstract
Over the years, 2D GANs have achieved great successes in photorealistic portrait generation. However, they lack 3D understanding in the generation process, thus they suffer from multi-view inconsistency problem. To alleviate the issue, many 3D-aware GANs have been proposed and shown notable results, but 3D GANs struggle with editing semantic attributes. The controllability and interpretability of 3D GANs have not been much explored. In this work, we propose two solutions to overcome these weaknesses of 2D GANs and 3D-aware GANs. We first introduce a novel 3D-aware GAN, SURF-GAN, which is capable of discovering semantic attributes during training and controlling them in an unsupervised manner. After that, we inject the prior of SURF-GAN into StyleGAN to obtain a high-fidelity 3D-controllable generator. Unlike existing latent-based methods allowing implicit pose control, the proposed 3D-controllable StyleGAN enables explicit pose control over portrait generation. This distillation allows direct compatibility between 3D control and many StyleGAN-based techniques (e.g., inversion and stylization), and also brings an advantage in terms of computational resources. Our codes are available at https://github.com/jgkwak95/SURF-GAN.
Metrics
Details
- Title
- Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis
- Creators
- Jeong-gi Kwak - Korea UniversityYuanming Li - Korea UniversityDongsik Yoon - Korea UniversityDonghyeon Kim - Korea UniversityDavid Han - Drexel UniversityHanseok Ko - Korea University
- Contributors
- S Avidan (Editor)G Brostow (Editor)M Cisse (Editor)G M Farinella (Editor)T Hassner (Editor)
- Publication Details
- COMPUTER VISION - ECCV 2022, PT XVII, v 13677, pp 236-253
- Series
- Lecture Notes in Computer Science
- Publisher
- Springer Nature
- Number of pages
- 18
- Grant note
- DMLab
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Electrical and Computer Engineering
- Web of Science ID
- WOS:000904106100015
- Scopus ID
- 2-s2.0-85142690681
- Other Identifier
- 991021930834904721
UN Sustainable Development Goals (SDGs)
This publication has contributed to the advancement of the following goals:
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- International collaboration
- Web of Science research areas
- Computer Science, Artificial Intelligence
- Imaging Science & Photographic Technology