Method learns from video demonstrations with human feedback.
problem Teaching autonomous agents using video demonstrations and human feedback.
method Constructs a mapping between standard and visual representations using a neural network.
result Effective in teaching a hopper agent to perform a backflip with minimal human feedback.
New method learns agent actions from raw video demonstrations.
problem Imitation learning for natural-looking movements.
method Generative Adversarial Networks (GANs) for joint state and reward learning.
result Adversarial imitation learning on raw videos produces similar performance to state-of-the-art methods.
Generative model learns compact codes for video recovery.
problem Efficiently represent and reconstruct videos from missing data.
method Generative network trained to map compact latent codes to images, with low-rank and similarity constraints.
result Can recover true video sequences even if not in pretrained network's range.
Researchers create a flickering attack to fool video recognition networks.
problem Adversarial manipulation of video classification networks.
method Introducing a flickering temporal perturbation to fool video classifiers.
result Achieved high fooling ratio and temporal-invariant perturbation.
New approach learns traffic behavior from unlabelled video.
problem Learning from demonstration in settings with unlabelled raw video data.
method Calibrates camera, detects objects, tracks them, and uses trajectories for LfD.
result Can learn traffic behavior from raw video without additional expert knowledge.
Paper improves video categorization using temporal coherence.
problem Video categorization in multiple modalities.
method Temporal coherence-based regularization for multimodal models.
result Models with temporal coherence outperform state-of-the-art.
CNN-AIRL improves video game performance using expert demonstrations.
problem Designing reward functions for complex video games is difficult.
method Adversarial IRL with CNNs, autoencoder state representation, reward normalization.
result Achieves high-level performance on Catcher game, outperforming baseline.
Ground-A-Video edits videos without training, preserving intended changes.
problem Complex multi-attribute video editing with omitted or wrong changes.
method Grounding-guided video-to-video translation with Cross-Frame Gated Attention.
result Zero-shot multi-attribute video editing with improved accuracy and frame consistency.
New method learns robot actions from videos without explicit labels.
problem Training robots to perform tasks from few demonstrations.
method Uses images and text for task-agnostic and general representation, synthesizes hallucinated actions, and applies dense correspondences.
result Trains robot policies solely from RGB videos, achieving diverse tasks across different robots and environments.
Lower-dimensional video discriminators improve GAN performance.
problem High curvature in unconstrained video discriminator loss surfaces.
method Proposed Lower-Dimensional Video Discriminators (LDVD GANs).
result LDVD GANs double Temporal-GAN performance and achieve state-of-the-art performance.
UDVD uses deep learning to denoise videos without supervision.
problem Lack of clean video data for training deep learning models.
method UDVD is a CNN trained solely on noisy video data, adapting to local motion.
result UDVD performs as well as supervised methods, even with limited training data.
Current deep learning results on video generation are limited while there are only a few first results on video prediction and no relevant significant results on video completion. This is due to the severe ill-posedness inherent in these three problems. In this paper, we focus on human action videos, and propose a gene…
Due to burdensome data requirements, learning from demonstration often falls short of its promise to allow users to quickly and naturally program robots. Demonstrations are inherently ambiguous and incomplete, making correct generalization to unseen situations difficult without a large number of demonstrations in varyi…
Paper introduces adversarial lossy compression for video artifacts reduction.
problem Unpleasant reconstruction artifacts in standard video coding schemes at low bit-rates.
method Adversarial lossy video compression model minimizing an adversarial distortion objective.
result Reduction of perceptual artifacts and detail reconstruction under extreme compression.
DIVE learns video representations even with missing data.
problem Missing data in video sequences.
method Disentangled Imputed Video autoEncoder (DIVE) with missingness latent variable.
result DIVE outperforms state-of-the-art baselines in imputing and predicting missing video frames.
Recent studies have demonstrated the power of recurrent neural networks for machine translation, image captioning and speech recognition. For the task of capturing temporal structure in video, however, there still remain numerous open research questions. Current research suggests using a simple temporal feature pooling…
Paper defends against adversarial videos by detecting and reducing imperceptible perturbations.
problem Adversarial videos can fool well-trained video classification models.
method Temporal consistency between frames and spatial denoising to detect and reduce perturbations.
result The proposed method significantly improves robustness against adversarial attacks.
Agent learns third-person manipulation tasks from a single video.
problem Learning from third-person videos to perform novel tasks.
method Decoupling high-level task generation from low-level action prediction.
result Agent successfully learns and performs tasks in unseen scenarios.
Paper proposes SMFN for high-res spherical video super-resolution.
problem Super-resolution of 360-degree panoramic videos is expensive and challenging.
method Deformable convolutions, mixed attention mechanism, dual learning strategy, weighted mean square error loss function.
result The proposed SMFN method improves super-resolution of equatorial regions in 360-degree videos.
We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can …
Jointly trains images and videos using residual vectors.
problem Generating high-quality videos from images and vice versa.
method Simultaneously learns latent variables for images and videos using residual vectors.
result Improves sample quality and diversity in video generation and image generation.
A method learns to imitate from a single video demonstration using contrastive training and Siamese networks.
problem Learning to imitate from video demonstrations without direct access to state or action information.
method Contrastive training with Siamese recurrent neural networks to learn rewards and an RL policy to minimize distance.
result The method significantly improves policy learning and outperforms current techniques in various environments.
Method learns multi-stage tasks from single video, overcoming challenges of raw pixel learning and insufficient demonstrations.
problem Learning multi-stage vision-based tasks from a single video of a human performing the task.
method Learn primitive behaviors from video demonstrations and dynamically compose them to perform multi-stage tasks.
result Demonstrated learning of various tasks on real robots using raw pixel inputs and minimal demonstrations.
Extracts controllable models from videos of real-world activities.
problem Creating realistic and controllable character models from video data.
method Two networks: one for pose and control signal to next pose, and another for pose, new pose, and background to output frame.
result High-quality, controllable character models can be generated from arbitrary videos.
Introduces Motion Programs for better video analysis of human motion.
problem Current video analysis focuses on raw pixels or keypoints, missing higher-level motion primitives.
method Introduces Motion Programs as a neuro-symbolic representation of motions as a composition of high-level primitives.
result Motion Programs accurately describe diverse human motions and improve downstream tasks.
Making predictions of future frames is a critical challenge in autonomous driving research. Most of the existing methods for video prediction attempt to generate future frames in simple and fixed scenes. In this paper, we propose a novel and effective optical flow conditioned method for the task of video prediction wit…
Improved speech recognition using EEG and video.
problem Enhancing continuous speech recognition systems.
method Implemented a CTC-based ASR model using EEG features.
result EEG features improve continuous visual speech recognition.
Detect objects from motion without annotations.
problem Weakly supervised object detection.
method Train model on videos of moving objects and negative scenes.
result Detects objects in single images without annotations.
Paper presents a new video compression method using autoencoders.
problem Efficient video compression with reduced quality loss.
method 3D autoencoder with discrete latent space and autoregressive prior trained jointly.
result Method outperforms state-of-the-art learned video compression networks.
Our goal is to predict future video frames given a sequence of input frames. Despite large amounts of video data, this remains a challenging task because of the high-dimensionality of video frames. We address this challenge by proposing the Decompositional Disentangled Predictive Auto-Encoder (DDPAE), a framework that …
Proposes a self-supervised method for generating spatial audio from monaural audio and video.
problem Generating spatial audio from monaural audio and video recordings is challenging and expensive.
method Uses a self-supervised network with an auxiliary classifier to classify video channels and generate spatial audio.
result The proposed method effectively generates spatial audio from monaural audio and video.
New method tackles video inverse problems using image diffusion models.
problem Spatio-temporal degradation in video inverse problems.
method Leverages image diffusion models to treat time dimension as batch dimension, introduces batch-consistent diffusion sampling.
result Achieves state-of-the-art reconstructions for various spatio-temporal degradations.
Improved video and movie description using multitask learning.
problem Lack of training data and poor generalization in video captioning.
method Multitask learning encoder-decoder framework for video sequences.
result Improved performance on multi-caption and single-caption datasets.
STOVE predicts convincing physical behavior in videos.
problem Learning models from videos with objects and interactions.
method Compositional state-space model combining image and dynamics models.
result Predicts videos with convincing physical behavior over hundreds of timesteps.
Proposes interpretable attention for video action recognition.
problem Efficient video action recognition with attention mechanisms.
method Spatial-temporal attention mechanism with saliency masks and convolutional LSTM.
result Improves video action recognition accuracy and spatial-temporal localization.
There are many forms of feature information present in video data. Principle among them are object identity information which is largely static across multiple video frames, and object pose and style information which continuously transforms from frame to frame. Most existing models confound these two types of represen…
Rolling Diffusion improves video prediction by progressively corrupting frames based on their temporal position.
problem Improving video prediction accuracy by accounting for temporal dynamics.
method A sliding window denoising process that assigns more noise to frames that appear later in a sequence.
result Rolling Diffusion outperforms standard diffusion models in tasks with complex temporal dynamics.
New algorithm learns tasks from video demonstrations using proprioceptive information.
problem Lack of idealized conditions in imitation learning from video demonstrations.
method Proposes an algorithm that leverages proprioceptive state representations for policy learning.
result Outperforms other IfO algorithms by a large margin in MuJoCo domains.
State-only imitation learning improves dexterous manipulation learning from videos.
problem High sample complexity in complex domains like dexterous manipulation.
method Train an inverse dynamics model to predict actions from states and train the policy jointly.
result Performs on par with state-action approaches and outperforms RL alone.
V-BAD is a first black-box video attack framework that successfully fools deep video recognition models.
problem Vulnerability of video recognition models to black-box adversarial attacks.
method Tentative perturbations transferred from image models and partition-based rectifications for patches of tentative perturbations.
result V-BAD can craft both untargeted and targeted attacks with high success rates using a manageable number of queries.
This paper presents two unsupervised learning layers (UL layers) for label-free video analysis: one for fully connected layers, and the other for convolutional ones. The proposed UL layers can play two roles: they can be the cost function layer for providing global training signal; meanwhile they can be added to any re…
Programmatic Motion Concepts learn human actions from paired videos.
problem Learning motion concepts from paired video and action sequences.
method Semi-supervised learning architecture for hierarchical motion representation.
result Outperforms established baselines, especially in small data settings.
Paper introduces graph-based transforms for video compression.
problem Efficiently represent video signals for compression.
method Develops two techniques for designing graph-based transforms (GL-GBTs and EA-GBTs).
result Graph-based transforms outperform traditional KLT in video compression.
New method identifies physical constants from video data alone.
problem Identifying physical constants from video data.
method Proves level-set slope-coverage condition ensures local affine mapping to true physical state, enabling exact parameter recovery.
result Underdamped systems identifiable from a single video clip, other regimes require three diverse trajectories.
Online anomaly detection in surveillance videos with false alarm rate bounds.
problem Lack of theoretical performance analysis and online decision making in anomaly detection.
method Proposes an online anomaly detection method with asymptotic bounds on false alarm rate.
result Demonstrates effectiveness on publicly available data sets, outperforming state-of-the-art algorithms.
A framework disentangles controllable objects from visual signals for improved RL.
problem Improving sample efficiency and game performance in vision-based RL.
method Action-conditioned video prediction to disentangle controllable objects.
result Improved sample efficiency and game performance in Atari games.
Paper introduces DACAL for high-resolution photo and video enhancement.
problem Photo and video enhancement with weak supervision.
method Divide-and-conquer adversarial learning approach with hierarchical decomposition.
result State-of-the-art performance in high-resolution photo and video enhancement.
Polynomial fusion layer improves speech-driven facial animation.
problem Recent facial synthesis relies on low-dimensional representations and concatenation, ignoring higher-order interactions.
method Proposes a polynomial fusion layer to model higher-order interactions of facial encodings.
result Demonstrates improved video quality, audiovisual synchronisation, and blink generation.