We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algor…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “photorealistic rendering”
AutoSimulate efficiently optimizes synthetic data generation.
problem Optimizing synthetic data generation for machine learning.
method Differentiable approximation of the objective function for efficient optimization.
result Significantly faster (up to 50x) and more efficient (up to 30x) synthetic data generation.
Highly expressive models such as deep neural networks (DNNs) have been widely applied to various applications. However, recent studies show that DNNs are vulnerable to adversarial examples, which are carefully crafted inputs aiming to mislead the predictions. Currently, the majority of these studies have focused on per…
Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of o…
Worldsheet wraps a 3D mesh sheet onto a single image to synthesize novel views.
problem Synthesizing novel views from a single image with large viewpoint changes.
method Shrink-wrapping a planar mesh sheet onto the input image, consistent with learned depth.
result Worldsheet consistently outperforms prior methods on single-image view synthesis.
3DB framework tests and debugs computer vision models using photorealistic simulation.
problem Discovering vulnerabilities and understanding model decision-making in computer vision systems.
method Unified framework using photorealistic simulation to test and debug vision models.
result Insights generated by 3DB transfer to the physical world, enabling robustness analysis.
Generative Adversarial Networks are a new family of generative models, frequently used for generating photorealistic images. The theory promises for the GAN to eventually reach an equilibrium where generator produces pictures indistinguishable for the training set. In practice, however, a range of problems frequently p…
Efficiently distills pretrained text-to-image models without real data, improving FID and CLIP scores.
problem Slow iterative refinement process of diffusion-based text-to-image models.
method Guided Score identity Distillation with Long and Short Classifier-Free Guidance.
result Achieves state-of-the-art FID performance with competitive CLIP score.
Despite the remarkable progress in face recognition related technologies, reliably recognizing faces across ages still remains a big challenge. The appearance of a human face changes substantially over time, resulting in significant intra-class variations. As opposed to current techniques for age-invariant face recogni…
Automates hair color digitization using imaging and deep learning.
problem Challenges in capturing and rendering realistic hair colors.
method Combines imaging, path-tracing, and self-supervised machine learning.
result Accurately captures and renders hair color with synthetic images.
In this paper, we show how novel transfer reinforcement learning techniques can be applied to the complex task of target driven navigation using the photorealistic AI2THOR simulator. Specifically, we build on the concept of Universal Successor Features with an A3C agent. We introduce the novel architectural contributio…
A neural scene representation framework enforcing 3D transformations.
problem Learning 3D scene representations from images without 3D supervision.
method Introducing a loss enforcing equivariance of the scene representation with 3D transformations.
result Real-time neural rendering with comparable results to models requiring minutes for inference.
The increasingly photorealistic sample quality of generative image models suggests their feasibility in applications beyond image generation. We present the Neural Photo Editor, an interface that leverages the power of generative neural networks to make large, semantically coherent changes to existing images. To tackle…
We present the OpenAI Remote Rendering Backend (ORRB), a system that allows fast and customizable rendering of robotics environments. It is based on the Unity3d game engine and interfaces with the MuJoCo physics simulation library. ORRB was designed with visual domain randomization in mind. It is optimized for cloud de…
We focus on explicitly learning disentangled representation for natural image generation, where the underlying spatial structure and the rendering on the structure can be independently controlled respectively, yet using no tuple supervision. The setting is significant since tuple supervision is costly and sometimes eve…
We create real-time geodesic rendering for non-isotropic geometries.
problem Challenging visualization of non-isotropic geometries.
method Novel methods for real-time native geodesic rendering.
result Methods can be applied to visualization, machine learning, and video games.
Synthesizing high resolution photorealistic images has been a long-standing challenge in machine learning. In this paper we introduce new methods for the improved training of generative adversarial networks (GANs) for image synthesis. We construct a variant of GANs employing label conditioning that results in 128x128 r…
ROOTS learns to represent and render 3D scenes with object-centric models.
problem Learning to represent and render 3D scenes with object-centric compositionality.
method Probabilistic generative model for learning object representations and scene rendering from partial observations.
result The model can infer 3D object representations and render scenes from arbitrary viewpoints.
This paper improves anomaly detection in lane rendering images for safer navigation.
problem Anomalies in lane rendering images can mislead drivers, posing safety risks.
method Proposes a four-phase pipeline using Transformer models, self-supervised pre-training, and fine-tuning.
result The pipeline enhances detection accuracy and reduces training time.
OBSER framework infers sub-environments from objects, outperforming scene-based methods.
problem Zero-shot recognition of environments from object distributions.
method Bayesian framework using metric and self-supervised learning models to estimate object distributions in latent space.
result OBSER framework reliably performs inference in open-world and photorealistic environments, outperforming scene-based methods.
AR-GANs learn depth and DoF from unlabeled images using aperture rendering and focus cues.
problem Learning depth and DoF from unlabeled natural images with diverse viewpoints and shapes.
method Aperture rendering and focus cues to learn depth and DoF from unlabeled images.
result AR-GANs effectively learn depth and DoF from various datasets, including flower, bird, and face images.
DVAO predicts volumetric ambient occlusion for real-time volume rendering.
problem Predicting per-voxel ambient occlusion in volumetric data sets.
method Deep learning neural network that considers global information through transfer function.
result DVAO supports real-time volume interaction and generalizes to various modalities.
Understanding the 3-dimensional structure of the world is a core challenge in computer vision and robotics. Neural rendering approaches learn an implicit 3D model by predicting what a camera would see from an arbitrary viewpoint. We extend existing neural rendering to more complex, higher dimensional scenes than previo…
NeRF-VAE generates 3D scenes with geometric structure from few images.
problem Generating 3D scenes from few images with geometric consistency.
method Combines NeRF and VAE, incorporating shared geometric structure.
result NeRF-VAE can infer and render geometrically-consistent scenes from unseen environments.
We present a generative autoencoder that provides fast encoding, faithful reconstructions (eg. retaining the identity of a face), sharp generated/reconstructed samples in high resolutions, and a well-structured latent space that supports semantic manipulation of the inputs. There are no current autoencoder or GAN model…
Many machine learning image classifiers are vulnerable to adversarial attacks, inputs with perturbations designed to intentionally trigger misclassification. Current adversarial methods directly alter pixel colors and evaluate against pixel norm-balls: pixel perturbations smaller than a specified magnitude, according t…
Importance sampling is one of the most widely used variance reduction strategies in Monte Carlo rendering. In this paper, we propose a novel importance sampling technique that uses a neural network to learn how to sample from a desired density represented by a set of samples. Our approach considers an existing Monte Ca…
Improves spline quality and accuracy in computational microscopy.
problem Detecting slender, overlapping structures in microscopy images.
method Differentiable rendering approach for spline refinement.
result Achieves high reliability and sub-pixel accuracy.
Survey on GANs for generating visual arts, music, and literature.
problem Tackles the challenge of generating art using GANs.
method Uses generative adversarial networks (GANs) to generate visual arts, music, and literary text.
result Performance comparison and description of various GAN architectures presented.
As synthetic imagery is used more frequently in training deep models, it is important to understand how different synthesis techniques impact the performance of such models. In this work, we perform a thorough evaluation of the effectiveness of several different synthesis techniques and their impact on the complexity o…
Curiosity enhanced by audio-visual associations improves learning efficiency.
problem Challenges in reinforcement learning, especially predicting the future.
method Exploits multiple modalities (audio and vision) to predict novel associations.
result Improves exploration and learning efficiency in various environments.
We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to …
We propose a new algorithm for training generative adversarial networks that jointly learns latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). By fixing the identity portion of the latent codes, we can generate diverse images of the same subject, and by fixing the ob…
Semi-supervised learning algorithms reduce the high cost of acquiring labeled training data by using both labeled and unlabeled data during learning. Deep Convolutional Networks (DCNs) have achieved great success in supervised tasks and as such have been widely employed in the semi-supervised learning. In this paper we…
We present a technique for efficiently synthesizing images of atmospheric clouds using a combination of Monte Carlo integration and neural networks. The intricacies of Lorenz-Mie scattering and the high albedo of cloud-forming aerosols make rendering of clouds---e.g. the characteristic silverlining and the "whiteness" …
3D adversarial logos can fool object detectors in real-world settings.
problem Creating robust adversarial attacks in 3D rendering views.
method Constructing 3D adversarial logos via texture mapping and differentiable rendering.
result 3D adversarial logos are more versatile and robust than traditional adversarial patches.
We explore a new domain of learning to infer user interface attributes that helps developers automate the process of user interface implementation. Concretely, given an input image created by a designer, we learn to infer its implementation which when rendered, looks visually the same as the input image. To achieve thi…
In this paper we present Percival, a browser-embedded, lightweight, deep learning-powered ad blocker. Percival embeds itself within the browser's image rendering pipeline, which makes it possible to intercept every image obtained during page execution and to perform blocking based on applying machine learning for image…
Even as deep neural networks (DNNs) have achieved remarkable success on vision-related tasks, their performance is brittle to transformations in the input. Of particular interest are semantic transformations that model changes that have a basis in the physical world, such as rotations, translations, changes in lighting…
Algorithm finds optimal affine transformation to minimize overall distortion.
problem Minimizing distortion in affine transformations.
method Riemannian geometry approach to define and minimize distortion.
result Mean distorting transformation found for minimizing overall distortion.
Reverse-BSDE Monte Carlostat.ML
New approach uses FBSDE to sample complex distributions.
problem Sampling multidimensional distributions with known normalization constants.
method Reformulated FBSDE to avoid gradient estimation; numerical solution using Deep Learning.
result Unique solution to FBSDE proved; numerical method for sampling.
Action-bisimulation learns long-horizon controllability for reinforcement learning.
problem Learning relevant state features in high-dimensional observations for robust reinforcement learning.
method Action-bisimulation encoding, inspired by bisimulation invariance, extends single-step controllability to multi-step.
result Action-bisimulation pretraining improves sample efficiency in various environments.
A machine learning environment for detecting autonomous vehicle corner cases.
problem Testing autonomous driving software in the real world is difficult.
method Connecting CARLA simulation software to TensorFlow and custom AI client software.
result The system can identify situations where AI software fails to understand the scenario.
Advances in deep generative networks have led to impressive results in recent years. Nevertheless, such models can often waste their capacity on the minutiae of datasets, presumably due to weak inductive biases in their decoders. This is where graphics engines may come in handy since they abstract away low-level detail…
The idea of computer vision as the Bayesian inverse problem to computer graphics has a long history and an appealing elegance, but it has proved difficult to directly implement. Instead, most vision tasks are approached via complex bottom-up processing pipelines. Here we show that it is possible to write short, simple …
We describe a method of rendering real-time scenes in Nil geometry, and use this to give an expository account of some interesting geometric phenomena. You can play around with the simulation at www.3-dimensional.space/nil.html.
VectorNet predicts car behavior using vectorized HD maps and agent dynamics.
problem Predicting behavior in multi-agent systems with self-driving cars.
method VectorNet uses hierarchical graph neural networks on vectorized representations of HD maps and agent trajectories.
result VectorNet achieves comparable or better performance than state-of-the-art methods while using fewer parameters and less computational power.
Mozilla Research is developing Servo, a parallel web browser engine, to exploit the benefits of parallelism and concurrency in the web rendering pipeline. Parallelization results in improved performance for pinterest.com but not for google.com. This is because the workload of a browser is dependent on the web page it i…