CDDN tackles visual relationship detection with context-dependent diffusion networks.
problem Combustion of combinatorial explosion in relation triplets detection.
method CDDN framework using semantic and visual scene graphs for adaptive information aggregation.
result CDDN achieves state-of-the-art performance on visual relationship detection datasets.
A novel method for visual question answering using scene graphs and reinforcement learning.
problem Answering free-form questions about images with deep linguistic and visual understanding.
method Context-driven, sequential reasoning based on scene graphs and reinforcement learning.
result Our method almost reaches human performance on the GQA dataset.
Generative model improves scene graph prediction for rare compositions.
problem Challenges in inferring unseen or rare scene graph compositions.
method Synthesizing rare scene graphs by perturbing real ones, using conditional GANs.
result Marginal but consistent improvements in zero- and few-shot metrics.
Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple inter-related objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structu…
GraphQ system uses GNNs to search for subgraph patterns in graphs.
problem Efficiently identifying and matching subgraph patterns in graph data.
method Graph neural networks (GNNs) for encoding graph data and NeuroAlign for node alignment.
result NeuroAlign improves node-alignment accuracy by 19-29% compared to baseline GNNs.
Deep neural networks decode natural visual scenes from neural spikes.
problem Decoding visual scenes from neural spikes for brain-machine interfaces.
method Developed a novel spike-image decoder (SID) using deep neural networks.
result SID reconstructs natural visual scenes from neural spikes with high accuracy.
Constellation learns group-level visual relationships for abstract reasoning.
problem Learning configurational properties of entire groups of objects.
method Introduces Constellation, a network that learns relational abstractions over static visual scenes.
result Offers a basis for abstract relational reasoning and sensory imagination.
This work proposes a method to compose visual relations more faithfully.
problem Composing relations between objects in images is challenging due to their entanglement.
method Represent each relation as an unnormalized density (energy-based model) to compose relations factorizedly.
result The proposed method generates and edits scenes with multiple sets of relations more faithfully.
Graph embedding techniques are useful to characterize spectral signature relations for hyperspectral images. However, such images consists of disjoint classes due to spatial details that are often ignored by existing graph computing tools. Robust parameter estimation is a challenge for kernel functions that compute suc…
Improves AI agents' 3D navigation by learning from failures and 3D spatial relationships.
problem Challenges in data efficiency, obstacle avoidance, and generalization in 3D visual navigation.
method Incorporates attention on 3D spatial relationships and a target skill extension module into DRL framework.
result Significantly improves navigation performance and generalization across targets and scenes.
New task AQA tackles acoustic reasoning from sound scenes.
problem Promote research in acoustic reasoning.
method Generate acoustic scenes from elementary sounds and formulate questions.
result Preliminary results with models FiLM and MAC show promise.
New dataset tests mental rotation from single images, improving model understanding of 3D scenes.
problem Understanding how a scene looks from a different viewpoint using a single image.
method Created CLEVR-MRT dataset, explored neural architectures for volumetric scene representations.
result Demonstrated the effectiveness of volumetric representations in answering mental rotation questions.
GENESIS generates and samples 3D scenes by capturing object interactions.
problem Lack of models that explicitly capture object interactions in scene generation.
method Object-centric latent variables, spatial GMM, amortized inference, autoregressive prior.
result First object-centric generative model of 3D visual scenes.
Recent progress on automatic generation of image captions has shown that it is possible to describe the most salient information conveyed by images with accurate and meaningful sentences. In this paper, we propose an image caption system that exploits the parallel structures between images and sentences. In our model, …
Scene text magnifier enhances readability for visually impaired.
problem Helps visually impaired read natural scene text.
method Four CNN-based networks: character erasing, extraction, magnify, synthesis.
result Effective text magnification without background alteration.
Scene understanding remains a significant challenge in the computer vision community. The visual psychophysics literature has demonstrated the importance of interdependence among parts of the scene. Yet, the majority of methods in computer vision remain local. Pictorial structures have arisen as a fundamental parts-bas…
Model learns disentangled object location and appearance representations.
problem Learning disentangled representations of object location and appearance.
method Probabilistic generative model with amortized variational inference.
result Fully disentangled object location and appearance representations.
We consider the problem of naming objects in complex, natural scenes containing widely varying object appearance and subtly different names. Informed by cognitive research, we propose an approach based on sharing context based object hypotheses between visual and lexical spaces. To this end, we present the Visual Seman…
Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images. We propose to exploit proprioceptive motor signals to provide unsuperv…
New model separates object attributes for better perceptual grouping.
problem Perceptual grouping of complex visual scenes.
method Spatial mixture models with learnable priors.
result Outperforms state-of-the-art methods in perceptual grouping.
We consider learning based methods for visual localization that do not require the construction of explicit maps in the form of point clouds or voxels. The goal is to learn an implicit representation of the environment at a higher, more abstract level. We propose to use a generative approach based on Generative Query N…
Neural networks trained on natural scenes exhibit Gestalt closure.
problem Understanding how visual perception groups elements in natural scenes.
method Used deep-learning methods to investigate natural scene statistics and their relation to Gestalt laws.
result A neural network trained on natural images exhibits closure on synthetic edge fragment displays.
A framework isolates VQA reasoning from perception for better model evaluation.
problem Improper separation of visual perception and reasoning in VQA models.
method Introducing a framework and a top-down calibration technique to decouple reasoning from perception.
result Improved evaluation of VQA models by separating reasoning from perception.
Method infers depth from sparse points and camera motion.
problem Depth inference from limited sparse data.
method Constructs a planar scaffolding and uses predictive cross-modal criterion.
result State-of-the-art performance on depth completion benchmark.
New model separates objects in scenes, enabling novel arrangements and depth.
problem Lack of modular, compositional scene modeling in generative models.
method Ensemble of generative models (experts) compete for explaining different parts of a scene.
result Model generates scenes with novel object arrangement and depth ordering.
Ray-marching method visualizes 8 Thurston geometries in real-time.
problem Accurately rendering and visualizing Thurston geometries in real-time.
method Ray-marching algorithms with theoretical framework for non-Euclidean geometries.
result Accurate interactive real-time views of Thurston geometries achieved.
Generates coherent 3D scenes from monocular videos without supervision.
problem Lack of 3D scene modeling in video generation models.
method Trains a model to generate 3D scenes with moving objects and a background from monocular videos.
result Trained model generates coherent 3D scenes with multiple moving objects and a background.
Proposes Deep Scenes for interaction-aware scene understanding in reinforcement learning for autonomous driving.
problem Leveraging deep reinforcement learning for high-level decision making in autonomous driving requires handling variable-length sequences of different object types and interactions.
method Introduces Deep Scenes architecture, an extension of Deep Sets or Graph Convolutional Networks, to learn complex interaction-aware scene representations.
result Graph-Q and DeepScene-Q algorithms outperform state-of-the-art methods in evaluations with SUMO.
Improves reinforcement learning agent's scene-specific value function.
problem High variance in samples for policy gradient computations in multi-scene environments.
method Proposes dynamic value estimation (DVE) for multiple MDPs, clustering value functions across scenes.
result Lower sample variance and more accurate scene-specific value function estimates.
Proposes a deep structured generative model for complex scenes.
problem Difficulty in generating images with complicated structures.
method Boosts GANs with structure information encoded by a stochastic and-or graph (sAOG).
result Successfully captures and generates images with complex structures.
The paper proposes a model to forecast traffic motion from sensor data.
problem Accurately predicting traffic motion for safe vehicle maneuvers.
method Implicit latent variable model using interaction graphs and graph neural networks.
result Achieves state-of-the-art motion forecasting and interaction understanding.
In this work, we consider an extension of graphical models to random graphs, trees, and other objects. To do this, many fundamental concepts for multivariate random variables (e.g., marginal variables, Gibbs distribution, Markov properties) must be extended to other mathematical objects; it turns out that this extensio…
A new memory model enhances deep learning's visual understanding.
problem Lack of short-term memory in deep learning models.
method Introduces a biologically inspired visual working memory architecture.
result Model achieves competitive classification performance and reconstructs images.
Deep network predicts action sequences for complex tasks from a scene image.
problem Scalable task and motion planning from initial scene images.
method Deep convolutional recurrent neural network that predicts action sequences.
result Predicts promising action sequences, reducing motion planning problems.
Graph neural network predicts vehicle interactions and trajectories for autonomous driving.
problem Predicting future motion of vehicles in traffic scenes.
method Graph neural network that jointly predicts interaction modes and 5-second future trajectories.
result Jointly predicting trajectories and interaction modes leads to lower trajectory error.
BPI models 2D patterns on multiple planes and 3D scene from a single image.
problem Understanding and editing images with multiple 2D planes and 3D scene from a single image.
method Box Program Induction (BPI) with neural networks and search-based algorithm.
result Holistic, structured scene representation enables 3D-aware image editing.
Study shows bottlenecks improve image segmentation quality.
problem Robust object discovery in real-world images remains challenging.
method Empirical investigation of reconstruction bottlenecks in GENESIS model.
result Reconstruction bottlenecks determine reconstruction and segmentation quality.
Deep RL agent reconfigures dynamic visual sensor networks efficiently.
problem Efficiently reconfigure dynamic visual sensor networks.
method Modified asynchronous advantage actor-critic framework with Relational Network module, trained in an abstract simulation environment.
result System validated using real-world inputs and preexisting algorithms.
Proposes local coordinate frames for improving model performance in complex dynamical systems.
problem Improving model performance in complex, non-linear, and time-dependent dynamical systems.
method Introduces roto-translation invariant local coordinate frames for geometric graphs.
result The approach outperforms state-of-the-art models in various complex scenarios.
RICH models scenes as hierarchical tree to learn and generate complex compositions.
problem Learning compositional structures between parts and objects in natural scenes.
method RICH uses a latent scene graph to organize entities into a tree structure and employs a top-down inference approach.
result RICH learns and generates complex scene hierarchies from unlabeled data.
New architecture tracks objects in cluttered scenes without supervision.
problem Learning to track objects in large scenes with many objects.
method Spatially invariant computations and representations.
result Outperforms competing methods in tracking objects in cluttered scenes.
Topic modeling based on latent Dirichlet allocation (LDA) has been a framework of choice to perform scene recognition and annotation. Recently, a new type of topic model called the Document Neural Autoregressive Distribution Estimator (DocNADE) was proposed and demonstrated state-of-the-art performance for document mod…
Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent's action…
SMT improves robotic long-horizon tasks by embedding and utilizing past observations.
problem Long-horizon tasks in partially observable environments require effective long-term memory.
method Scene Memory Transformer (SMT) embeds and uses attention to exploit spatio-temporal dependencies.
result SMT outperforms existing policies in visual navigation tasks.
Data clustering has received a lot of attention and numerous methods, algorithms and software packages are available. Among these techniques, parametric finite-mixture models play a central role due to their interesting mathematical properties and to the existence of maximum-likelihood estimators based on expectation-m…
KeyIn discovers keyframes to simplify video analysis.
problem Extracting essential dynamics from videos interleaved with predictable details.
method A hierarchical Keyframe-Inpainter (KeyIn) model learns keyframes and fills in the rest.
result KeyIn finds informative keyframes across various video dynamics.
Hierarchical Foresight improves robot vision tasks by planning long-term goals.
problem Compounding uncertainty and scalability issues in long horizon video prediction.
method Subgoal generation and planning using hierarchical visual foresight (HVF).
result Achieves nearly 200% performance improvement in vision-based manipulation tasks.
Recent studies in the field of human vision science suggest that the human responses to the stimuli on a visual display are non-deterministic. People may attend to different locations on the same visual input at the same time. Based on this knowledge, we propose a new stochastic model of visual attention by introducing…