End-to-end autonomous driving models are vulnerable to simple physical manipulations of images.
problem Vulnerability of end-to-end autonomous driving models to subtle adversarial manipulations of images.
method Developed novel end-to-end attacks using simple physical manipulations (painting black lines on the road) and used Bayesian Optimization to efficiently search for successful attacks.
result Simple physical manipulations can cause autonomous driving models to follow unintended paths, highlighting the vulnerability of these models.
In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execu…
Motivated by vision-based control of autonomous vehicles, we consider the problem of controlling a known linear dynamical system for which partial state information, such as vehicle position, is extracted from complex and nonlinear data, such as a camera image. Our approach is to use a learned perception map that predi…
We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to …
Deep reinforcement learning provides a promising approach for vision-based control of real-world robots. However, the generalization of such models depends critically on the quantity and variety of data available for training. This data can be difficult to obtain for some types of robotic systems, such as fragile, smal…
Proposes Gaussian Processes for more accurate time-correlated measurement noise in robotics.
problem Time-correlated measurement noise in robotics applications.
method Gaussian Processes as a non-parametric model for correlated measurement noise.
result Improved performance of Kalman filtering with Gaussian Processes.
Framework simplifies vision-based control and goal discovery.
problem Learning proportional control from visual data.
method Introduces NewtonianVAE for proportional control and goal discovery.
result Dramatic simplification and acceleration of vision-based controllers.
Improves sample efficiency and generalization in vision-based RL by enhancing exploration.
problem Low sample efficiency in vision-based RL using images as observations.
method Integrates state representation learning to enhance exploration and sample diversity.
result Significant improvement in sample efficiency across various environments.
Paper presents a method for efficient robot adaptation using fine-tuning.
problem Continuous adaptation of robot learning systems in real-world scenarios.
method Fine-tuning previously learned policies using off-policy reinforcement learning.
result Fine-tuning leads to substantial performance gains and adaptation to new conditions.
TACTO simulates high-resolution touch sensing for robotics.
problem Accurate simulation of touch sensing in robotics.
method Fast, flexible, open-source simulator for vision-based tactile sensors.
result Demonstrated TACTO's effectiveness in grasping stability prediction and marble manipulation control.
This work tackles real-world robotic reinforcement learning challenges.
problem Limited success of reinforcement learning in real-world robotics.
method Proposes a system for autonomous real-world learning without instrumentation.
result Demonstrates a complete system that learns without human intervention.
Q2-Opt improves robot grasping success and efficiency.
problem Improving robot grasping success and efficiency in vision-based tasks.
method Quantile QT-Opt, a distributional variant of Q-learning for continuous domains.
result Q2-Opt achieves superior grasping success and is more sample efficient.
Modern vision-based reinforcement learning techniques often use convolutional neural networks (CNN) as universal function approximators to choose which action to take for a given visual input. Until recently, CNNs have been treated like black-box functions, but this mindset is especially dangerous when used for control…
FedVision uses federated learning to improve object detection without transmitting data.
problem Challenges in building object detection models on large training datasets due to privacy and cost issues.
method Federated learning (FL) platform for easy integration by non-experts.
result Significant efficiency improvement and cost reduction in smart city applications.
V-SysId identifies keypoints and 3D system from unlabeled videos.
problem Identifying keypoints and 3D system from unlabeled videos.
method Alternates between parameter estimation and extrinsic camera calibration, using motion equations as weak supervision.
result Utility of the approach demonstrated across various settings.
Existing model-based reinforcement learning methods often study perception modeling and decision making separately. We introduce joint Perception and Control as Inference (PCI), a general framework to combine perception and control for partially observable environments through Bayesian inference. Based on the fact that…
A framework disentangles controllable objects from visual signals for improved RL.
problem Improving sample efficiency and game performance in vision-based RL.
method Action-conditioned video prediction to disentangle controllable objects.
result Improved sample efficiency and game performance in Atari games.
Study the tradeoff between signal distortion and human perception over finite channels.
problem Characterize the distortion-perception tradeoff for finite channels with arbitrary metrics.
method Solve linear programming problems to compute the distortion-perception function and optimal reconstructions.
result DP function is piecewise linear in the perception index.
A framework isolates VQA reasoning from perception for better model evaluation.
problem Improper separation of visual perception and reasoning in VQA models.
method Introducing a framework and a top-down calibration technique to decouple reasoning from perception.
result Improved evaluation of VQA models by separating reasoning from perception.
Weakly-supervised RL identifies meaningful tasks, improving performance in complex environments.
problem Learning to efficiently explore and distinguish between meaningful and irrelevant tasks.
method Weak supervision to automatically disentangle meaningful tasks from a large space of nonsensical tasks.
result The learned subspace of meaningful tasks leads to substantial performance gains, especially in complex environments.
End-to-end autonomous driving perception learns latent features for better performance.
problem Current autonomous driving systems are complex and require human engineering.
method Sequential latent representation learning for end-to-end perception.
result End-to-end perception model solves detection, tracking, localization, and mapping problems.
Geometric model explains music perception combining neuroscience and acoustics.
problem Rationalize and predict psycho-acoustic phenomena in music perception.
method Combining neuroscientific theories with acoustic observations, a geometric model of the space of all chords is created.
result The geometric model allows for rigorous studies of psychoacoustic quantities like roughness and harmonicity.
RETR improves indoor radar perception with a novel transformer model.
problem Indoor radar perception lacks models tailored for multi-view radar settings.
method RETR extends DETR architecture with depth-prioritized feature similarity, tri-plane loss, and radar-to-camera transformation.
result RETR outperforms state-of-the-art methods by 15.38+ AP for object detection and 11.91+ IoU for instance segmentation.
Researchers study how teachers' advising relationships influence their perceptions of satisfaction and students, not policy influence.
problem Understanding the relationship between teachers' advising relationships and their perceptions of satisfaction and students.
method Proposed a novel joint model of network and item responses (JNIRM) with correlated latent variables.
result Teachers' advising relationships contribute more to satisfaction and students than to influence over educational policies.
Study how actions affect perception in embodied agents using group theory.
problem Understanding how actions influence perception in autonomous agents.
method Mathematical formalism of group theory applied to sensory commutativity of action sequences.
result Introduced Sensory Commutativity Probability (SCP) to measure action effects on perception.
Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging e…
Reconstruction-based learning produces uninformative features for perception tasks.
problem Misalignment between reconstruction-based learning and perception tasks.
method Investigated the impact of input space reconstruction on feature learning for perception tasks.
result Reconstruction-based learning allocates model capacity to a subspace with uninformative features for perception tasks.
Tensor models decode human perception and memory using SPO triples.
problem Understanding implicit and explicit perception and memory in the brain.
method Tensor models with SPO triples, dual representations, and four layers.
result Semantic memory is crucial for explicit perception and declarative memories.
PeL separates sensory interface optimization from decision learning.
problem Optimizing sensory interfaces without task-specific information.
method Formal separation of perception and decision learning, using metrics for stability, informativeness, and geometry.
result Updates preserving invariants are orthogonal to decision gradients.
Model learns tool affordances from vision, enabling tool selection.
problem Learning tool affordances from visual input.
method Vision-based generative model with task predictor.
result Agents can select appropriate tools based on task success criteria.
Unsupervised fire and smoke segmentation from IR videos.
problem Early detection of fire from infrared videos.
method Spatial, temporal, and motion information fusion using optical flow, divergence, and intensity values.
result Markov Random Field outperforms other clustering algorithms in fire and smoke segmentation.
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor s…
PERCEPT detects changes in high-dimensional data streams using topological data analysis.
problem Detecting changes in high-dimensional data streams, especially when embedded in a low-dimensional space.
method Leverages topological data analysis to learn embedded topology as a point cloud via persistence diagrams, then applies non-parametric monitoring for detecting changes.
result Demonstrates efficient detection of online changes from high-dimensional data streams.
Sensorimotor contingency theory offers a promising account of the nature of perception, a topic rarely addressed in the robotics community. We propose a developmental framework to address the problem of the autonomous acquisition of sensorimotor contingencies by a naive robot. While exploring the world, the robot inter…
While perception tasks such as visual object recognition and text understanding play an important role in human intelligence, the subsequent tasks that involve inference, reasoning and planning require an even higher level of intelligence. The past few years have seen major advances in many perception tasks using deep …
Default-ERM shortcut learning persists even without additional information.
problem Default-ERM shortcut learning in perception tasks despite stable feature sufficiency.
method Studied linear perception task; developed margin control (MARG-CTRL) loss functions.
result Margin control mitigates shortcut learning on various tasks.
DGP learns speech recognition by modeling complex relationships between utterances.
problem Modeling complex relationships in speech recognition without relational data.
method Bayesian nonparametric deep learning method (DGP) that generates infinite probabilistic graphs.
result DGP successfully infers relationships among utterances without relational data during training.
The study improves deep learning models for safer autonomous vehicles.
problem Robustness of deep neural network models in autonomous driving.
method Analyzes and proposes solutions for deep learning model robustness.
result Enhanced deep learning models for safer autonomous vehicles.
In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best …
We present a method for fast training of vision based control policies on real robots. The key idea behind our method is to perform multi-task Reinforcement Learning with auxiliary tasks that differ not only in the reward to be optimized but also in the state-space in which they operate. In particular, we allow auxilia…
A robot learns to classify images with limited perception using a layered reinforcement learning approach.
problem Image classification for robots with partial perception.
method Three-layer architecture using deep reinforcement learning, including meta-layer, action-layer, and classification-layer.
result The method achieves high accuracy on the MNIST dataset and provides explainability of the agent's decision-making process.
New RBM method for missing data inference, comparing performance to existing methods.
problem Missing data inference and perception-distortion trade-off.
method Linearization of RBM effective energy function for missing data.
result Proposed method outperforms existing reconstruction procedures in missing data inference.
Cooperative perception improves 3D object detection in autonomous vehicles.
problem Limited field-of-view and occlusion in single sensor data.
method Early fusion of point clouds from multiple sensors, late fusion of independently detected bounding boxes, and hybrid combination.
result Early fusion approach outperforms late fusion by significantly higher recall (95%) compared to single-point sensing (30%).
An algorithm detects anomalies based on human perception principles.
problem Anomaly detection in data.
method Inspired by Gestalt psychology and Helmholtz principle, the algorithm models anomalies as unexpected elements in random distributions.
result The algorithm efficiently detects anomalies with minimal user intervention and promising results on multivariate data.
New coding theorem shows achievable rate matches theoretical limit.
problem Unknown existence of encoders and decoders for RDPF.
method Used stochastic, variable-length codes to prove RDPF achievable.
result Achievable rate matches theoretical rate-distortion-perception function.
Method transfers feature representation from large to small models using perception coherence.
problem Transfer feature representation from large to small models.
method Defines perception coherence, proposes loss function to minimize.
result Method outperforms or achieves on-par performance compared to strong baseline methods.
Bayesian models explain human time perception biases.
problem Understanding human time perception using Bayesian inference.
method Agent-based machine learning models and empirical data analysis.
result Bayesian models can replicate human time estimation biases.
In Autonomous Vehicles (AVs), one fundamental pillar is perception, which leverages sensors like cameras and LiDARs (Light Detection and Ranging) to understand the driving environment. Due to its direct impact on road safety, multiple prior efforts have been made to study its the security of perception systems. In cont…