Understanding Pure CLIP Guidance for Voxel Grid NeRF Models
Han-Hung Lee, Angel Xuan Chang
Simon Fraser University
 |
 |
 |
 |
Abstract
In this paper, we explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access to any datasets, a setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no systematic empirical study of mechanics for preventing adversarial generations within CLIP. We use neural radiance fields with explicit density and color grids to exacerbate this problem as the parameter space is larger than coordinate-based MLPs. Thus, we illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural networks provide an additional layer of regularization, resulting in better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.
Model
Results with Different Guidance Models
 |
 |
 |
 |
|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |