arXiv research

A neural scene representation framework enforcing 3D transformations.

ROOTS learns to represent and render 3D scenes with object-centric models.

NeRF-VAE generates 3D scenes with geometric structure from few images.

3D adversarial logos can fool object detectors in real-world settings.

Pix2Shape learns 3D scene representations from single images without supervision.

Proposes a model to generate 3D-aware images from 2D images.

Highly expressive models such as deep neural networks (DNNs) have been widely applied to various applications. However, recent studies show that DNNs are vulnerable to adversarial examples, which are carefully crafted inputs aiming to mislead the predictions. Currently, the majority of these studies have focused on per…

StyleNeRF generates high-resolution images with 3D consistency and style control.

Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of o…

Worldsheet wraps a 3D mesh sheet onto a single image to synthesize novel views.

Designing of touchless user interface is gaining popularity in various contexts. Using such interfaces, users can interact with electronic devices even when the hands are dirty or non-conductive. Also, user with partial physical disability can interact with electronic devices using such systems. Research in this direct…

AR-GANs learn depth and DoF from unlabeled images using aperture rendering and focus cues.

Understanding the 3-dimensional structure of the world is a core challenge in computer vision and robotics. Neural rendering approaches learn an implicit 3D model by predicting what a camera would see from an arbitrary viewpoint. We extend existing neural rendering to more complex, higher dimensional scenes than previo…

We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algor…

DreamFusion uses text-to-image diffusion models to create 3D images efficiently.

New method reconstructs 3D shapes from 2D images using Kendall's shape space.

Synthetic images rendered by graphics engines are a promising source for training deep networks. However, it is challenging to ensure that they can help train a network to perform well on real images, because a graphics-based generation pipeline requires numerous design decisions such as the selection of 3D shapes and …

In this work we propose a model that can manipulate individual visual attributes of objects in a real scene using examples of how respective attribute manipulations affect the output of a simulation. As an example, we train our model to manipulate the expression of a human face using nonphotorealistic 3D renders of a f…

Recently, multiple formulations of vision problems as probabilistic inversions of generative models based on computer graphics have been proposed. However, applications to 3D perception from natural images have focused on low-dimensional latent scenes, due to challenges in both modeling and inference. Accounting for th…

Neural Processes combine the strengths of neural networks and Gaussian processes to achieve both flexible learning and fast prediction in stochastic processes. However, a large class of problems comprises underlying temporal dependency structures in a sequence of stochastic processes that Neural Processes (NP) do not e…

Topology-GS improves 3D GS for better structural and feature integrity.

Using generative models for Inverse Graphics is an active area of research. However, most works focus on developing models for supervised and semi-supervised methods. In this paper, we study the problem of unsupervised learning of 3D geometry from single images. Our approach is to use a generative model that produces 2…

CARV reduces compute cost for downstream pipelines using diffusion models.

We present a technique for efficiently synthesizing images of atmospheric clouds using a combination of Monte Carlo integration and neural networks. The intricacies of Lorenz-Mie scattering and the high albedo of cloud-forming aerosols make rendering of clouds---e.g. the characteristic silverlining and the "whiteness" …

Advances in deep generative networks have led to impressive results in recent years. Nevertheless, such models can often waste their capacity on the minutiae of datasets, presumably due to weak inductive biases in their decoders. This is where graphics engines may come in handy since they abstract away low-level detail…

The idea of computer vision as the Bayesian inverse problem to computer graphics has a long history and an appealing elegance, but it has proved difficult to directly implement. Instead, most vision tasks are approached via complex bottom-up processing pipelines. Here we show that it is possible to write short, simple …

We explore visual representations of tilings corresponding to Schläfli symbols. In three dimensions, we call these tilings "honeycombs". Schläfli symbols encode, in a very efficient way, regular tilings of spherical, euclidean and hyperbolic spaces in all dimensions. In three dimensions, there are only a finite number …

Deep networks have achieved excellent results in perceptual tasks, yet their ability to generalize to variations not seen during training has come under increasing scrutiny. In this work we focus on their ability to have invariance towards the presence or absence of details. For example, humans are able to watch cartoo…

This paper describes InfoGAN, an information-theoretic extension to the Generative Adversarial Network that is able to learn disentangled representations in a completely unsupervised manner. InfoGAN is a generative adversarial network that also maximizes the mutual information between a small subset of the latent varia…

Automates hair color digitization using imaging and deep learning.

AR app visualizes Quranic Surah al-Fil for Islamic education.

We present the OpenAI Remote Rendering Backend (ORRB), a system that allows fast and customizable rendering of robotics environments. It is based on the Unity3d game engine and interfaces with the MuJoCo physics simulation library. ORRB was designed with visual domain randomization in mind. It is optimized for cloud de…

We focus on explicitly learning disentangled representation for natural image generation, where the underlying spatial structure and the rendering on the structure can be independently controlled respectively, yet using no tuple supervision. The setting is significant since tuple supervision is costly and sometimes eve…

We create real-time geodesic rendering for non-isotropic geometries.

We study knots in 3d Chern-Simons theory with complex gauge group $SL(N,\mathbb{C})$, in the context of its relation with 3d $\mathcal{N}=2$ theory (the so-called 3d-3d correspondence). The defect has either co-dimension 2 or co-dimension 4 inside the 6d $(2,0)$ theory, which is compactified on a 3-manifold $\hat{M}$. …

Study of 3d-3d correspondence involving $q$-Weyl algebra and 3d-index.

This paper improves anomaly detection in lane rendering images for safer navigation.

3D dual field theories for Virasoro minimal models constructed using Seifert fiber spaces.

3D flying wings created for any angle asymptotic cones.

Proposes a new effective central charge for 3d N=2 theories.

Autonomous driving requires 3D perception of vehicles and other objects in the in environment. Much of the current methods support 2D vehicle detection. This paper proposes a flexible pipeline to adopt any 2D detection network and fuse it with a 3D point cloud to generate 3D information with minimum changes of the 2D d…

DVAO predicts volumetric ambient occlusion for real-time volume rendering.

Smooth 3D flows from non-smooth starting points.

3D Axial-Attention improves lung nodule classification accuracy.

3D steady gradient Ricci solitons are all O(2)-symmetric.

3D Convolutional Neural Networks (3D-CNN) have been used for object recognition based on the voxelized shape of an object. However, interpreting the decision making process of these 3D-CNNs is still an infeasible task. In this paper, we present a unique 3D-CNN based Gradient-weighted Class Activation Mapping method (3D…

Deep generative architectures provide a way to model not only images but also complex, 3-dimensional objects, such as point clouds. In this work, we present a novel method to obtain meaningful representations of 3D shapes that can be used for challenging tasks including 3D points generation, reconstruction, compression…

Graph Neural Networks improve 3D object detection in LiDAR point clouds.