Understanding Pure CLIP Guidance for Voxel Grid NeRF Models

Github Paper

Han-Hung Lee, Angel Xuan Chang

Simon Fraser University

medieval people celebrating a festival with many stalls; trending on artstation.
an armoured knight with wings; trending on artstation.
night city with vaporwave aesthetic.
fantasy style garden; trending on artstation.


In this paper, we explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access to any datasets, a setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no systematic empirical study of mechanics for preventing adversarial generations within CLIP. We use neural radiance fields with explicit density and color grids to exacerbate this problem as the parameter space is larger than coordinate-based MLPs. Thus, we illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural networks provide an additional layer of regularization, resulting in better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.


Results with Different Guidance Models

Implicit + OpenCLIP ViT-B/32
Implicit + OpenCLIP ViT-B/16
Implicit + CLIP ViT-B/16
steampunk city; trending on artstation.
medieval people celebrating a festival with many stalls; trending on artstation.
a scary creature with many eyes and tentacles; trending on artstation.
a fanstasy castle; trending on artstation.
a souls like dark fantasy cityscape; trending on artstation.
a cyberpunk space station in space; trending on artstation.
traditional japanese buildings; trending on artstation.
Tokyo nights in lofi vibes.
Tokyo city; trending on artstation.