Model learns disentangled frames from video, enabling future frame prediction.
problem Learning disentangled representations from video sequences.
method Temporal coherence and adversarial loss for disentanglement.
result Model can predict future frames coherently.
Method learns from video demonstrations with human feedback.
problem Teaching autonomous agents using video demonstrations and human feedback.
method Constructs a mapping between standard and visual representations using a neural network.
result Effective in teaching a hopper agent to perform a backflip with minimal human feedback.
DIVE learns video representations even with missing data.
problem Missing data in video sequences.
method Disentangled Imputed Video autoEncoder (DIVE) with missingness latent variable.
result DIVE outperforms state-of-the-art baselines in imputing and predicting missing video frames.
Improved representations from multiple video views.
problem Learning reliable representations across unaligned video modalities.
method Correlation-based representation learning on a 4-way parallel multimodal dataset.
result Best models achieve retrieval rates up to 96.9% on open-domain instructional videos.
Paper improves video feature learning for better downstream tasks.
problem Improving video feature learning for better performance on downstream tasks.
method Self-supervised learning approach using contrastive bidirectional transformer, extending BERT for real-valued feature vectors.
result Significantly improved performance on video classification, captioning, and segmentation tasks.
SummaryNet automates video summarisation using deep learning.
problem Creating informative video summaries from videos.
method Two-stream convolutional network for spatial and temporal features, encoder-decoder model for salient features, sigmoid regression with LSTM for frame probability.
result SummaryNet achieves comparable or better results than state-of-the-art methods on benchmark datasets.
Programmatic Motion Concepts learn human actions from paired videos.
problem Learning motion concepts from paired video and action sequences.
method Semi-supervised learning architecture for hierarchical motion representation.
result Outperforms established baselines, especially in small data settings.
We identify action representations from video data, proving their statistical benefits.
problem Identifying latent action policies from video data.
method Entropy-regularized LAPO objective, formalizing desiderata for action representations.
result Entropy-regularized LAPO identifies action representations satisfying desiderata under suitable conditions.
Generative model learns compact codes for video recovery.
problem Efficiently represent and reconstruct videos from missing data.
method Generative network trained to map compact latent codes to images, with low-rank and similarity constraints.
result Can recover true video sequences even if not in pretrained network's range.
New method compresses facial videos using GANs and latent space optimization.
problem Efficiently compressing facial videos at low bit rates.
method Leverages StyleGAN for latent space representation and compression, learns optimal compression through entropy model and perceptual loss.
result Significantly reduces perceptual distortion at low bit rates compared to state-of-the-art codecs.
Improved video and movie description using multitask learning.
problem Lack of training data and poor generalization in video captioning.
method Multitask learning encoder-decoder framework for video sequences.
result Improved performance on multi-caption and single-caption datasets.
This paper separates static and dynamic features in video data.
problem Combining static and dynamic features in video data.
method Hierarchical Variational Auto-encoders with factored prior distributions.
result The model successfully separates static and dynamic features.
New method learns robot actions from videos without explicit labels.
problem Training robots to perform tasks from few demonstrations.
method Uses images and text for task-agnostic and general representation, synthesizes hallucinated actions, and applies dense correspondences.
result Trains robot policies solely from RGB videos, achieving diverse tasks across different robots and environments.
Spectral Inference Networks learn eigenfunctions from data using optimization.
problem Learning eigenfunctions of linear operators from data.
method Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators and use stochastic optimization.
result Spectral Inference Networks accurately recover eigenfunctions and discover interpretable representations from video data.
End-to-end deep generative model for video compression.
problem Efficiently compressing video data with deep learning.
method Variational autoencoder (VAE) for sequential data, combined with neural image compression techniques.
result Our model achieves competitive rate-distortion results on diverse video content.
New model learns human actions from video data without labels.
problem Learning human actions from unlabeled video data.
method Clustering-aware structure-constrained low-rank representation (CS-LRR) model integrating spectral clustering and hierarchical subspace clustering.
result Efficiently learns human action attributes from video data without requiring labeled data.
CB-GLNs learn video data's complex dependencies via graph representation.
problem Capturing complex dependency structures in sequential data like videos.
method Represent video data as a graph, find compositional dependencies via graph-cut and message passing.
result CB-GLNs efficiently learn video data's semantic compositional structure.
Paper presents unsupervised learning for visual representations using patches from unlabelled videos.
problem Learning visual representations without labeled data.
method Trains a model for foreground and background classification using patches extracted from unlabelled videos.
result Model achieves 45.3 mAP, close to best unsupervised learning techniques.
Unified model trained on images and videos using masked autoencoding.
problem Training a single model for multiple visual modalities.
method Masked autoencoding on a Vision Transformer.
result Unified model achieves comparable or better performance than single-modality models.
This work predicts and interpolates long-range videos using unsupervised landmarks.
problem Predicting and interpolating long-range video data with occlusions and appearance changes.
method Unsupervised latent structure inference followed by temporal prediction in a latent space.
result High-quality long-range video interpolation and extrapolation achieved through landmark representation.
TCT learns multimodal sequence representations by translating from related sequences.
problem Challenges in learning semantic representations from multimodalities.
method Transformer based Cross-modal Translator (TCT) combined with Multimodal Transformer Network (MTN).
result Proposed method achieves new state-of-the-art performance on video-grounded dialogue.
Recent progress in using recurrent neural networks (RNNs) for image description has motivated the exploration of their application for video description. However, while images are static, working with videos requires modeling their dynamic temporal structure and then properly integrating that information into a natural…
New metric for video generation models, addressing quality, coherence, and diversity.
problem Lack of metrics for evaluating generative models of video.
method Proposed Fréchet Video Distance (FVD) and StarCraft 2 Videos (SCV) benchmark.
result FVD correlates well with human judgment of generated videos.
PredNet fails to fully adhere to predictive coding principles.
problem Evaluation of PredNet's adherence to predictive coding theory.
method Critical analysis of PredNet's architecture and performance on a video action classification dataset.
result PredNet does not fully follow predictive coding principles but shows potential with top-down conditioning.
A new model disentangles object recognition and dynamics from video data.
problem Temporal reasoning in dynamically changing video data.
method Kalman variational auto-encoder framework for unsupervised learning.
result Model disentangles object representation and dynamics, outperforming other methods.
Introduces Motion Programs for better video analysis of human motion.
problem Current video analysis focuses on raw pixels or keypoints, missing higher-level motion primitives.
method Introduces Motion Programs as a neuro-symbolic representation of motions as a composition of high-level primitives.
result Motion Programs accurately describe diverse human motions and improve downstream tasks.
Polynomial fusion layer improves speech-driven facial animation.
problem Recent facial synthesis relies on low-dimensional representations and concatenation, ignoring higher-order interactions.
method Proposes a polynomial fusion layer to model higher-order interactions of facial encodings.
result Demonstrates improved video quality, audiovisual synchronisation, and blink generation.
Ground-A-Video edits videos without training, preserving intended changes.
problem Complex multi-attribute video editing with omitted or wrong changes.
method Grounding-guided video-to-video translation with Cross-Frame Gated Attention.
result Zero-shot multi-attribute video editing with improved accuracy and frame consistency.
A deep historical LSTM model improves tennis shot recognition from RGB videos.
problem Improving action recognition from RGB video input for sports analysis.
method Convolutional Neural Network (CNN) and Weighted Long Short-Term Memory (LSTM) for 3D tennis shot recognition.
result The method achieves better performance than state-of-the-art baselines for tennis shot recognition.
Agents learn action spaces from observation alone.
problem Learning an agent's action space from visual observation.
method Stochastic video prediction and composable action representations.
result Learned representations disentangle action structure.
DDPAE predicts video frames by decomposing and disentangling high-dimensional video data.
problem Predicting future video frames from input sequences is challenging due to high-dimensionality.
method Combines structured probabilistic models and deep networks to decompose and disentangle video components.
result DDPAE learns latent decomposition and disentanglement without supervision.
Survey of deep learning methods for video multi-object tracking.
problem Tracking multiple objects in video sequences.
method Review of deep learning approaches applied to MOT stages.
result Identification of similarities among top-performing methods.
Model learns disentangled representations from natural videos.
problem Disentangling factors of variation in natural data.
method Sparse prior on temporally adjacent observations.
result Model reliably learns disentangled representations on natural data.
A model predicts visual motion by learning from natural videos.
problem Temporal prediction accuracy in visual perception.
method Self-supervised representation learning using Fourier shift theorem.
result Achieves better prediction performance than traditional methods.
CNN-AIRL improves video game performance using expert demonstrations.
problem Designing reward functions for complex video games is difficult.
method Adversarial IRL with CNNs, autoencoder state representation, reward normalization.
result Achieves high-level performance on Catcher game, outperforming baseline.
Proposes a new VAE model with hierarchical nonparametric priors for better data representation.
problem Limited flexibility of standard VAE latent representations.
method Combines tree-structured Bayesian nonparametric priors with VAEs for joint learning of neural parameters and priors.
result Discover highly interpretable activity hierarchies and improved clustering accuracy.
A framework disentangles controllable objects from visual signals for improved RL.
problem Improving sample efficiency and game performance in vision-based RL.
method Action-conditioned video prediction to disentangle controllable objects.
result Improved sample efficiency and game performance in Atari games.
Jointly trains images and videos using residual vectors.
problem Generating high-quality videos from images and vice versa.
method Simultaneously learns latent variables for images and videos using residual vectors.
result Improves sample quality and diversity in video generation and image generation.
Agent learns to play hard games by watching YouTube videos.
problem Sparse rewards in reinforcement learning environments.
method Self-supervised video mapping, YouTube video embedding, imitation reward function.
result Agent achieves human-level performance on hard games.
With the rapid development of social media sharing, people often need to manage the growing volume of multimedia data such as large scale video classification and annotation, especially to organize those videos containing human activities. Recently, manifold regularized semi-supervised learning (SSL), which explores th…
Paper proposes SMFN for high-res spherical video super-resolution.
problem Super-resolution of 360-degree panoramic videos is expensive and challenging.
method Deformable convolutions, mixed attention mechanism, dual learning strategy, weighted mean square error loss function.
result The proposed SMFN method improves super-resolution of equatorial regions in 360-degree videos.
Self-supervised learning helps train deep features without needing lots of labeled data.
problem Annotation bottleneck in deep learning.
method Four main families of self-supervised approaches applied to various data modalities.
result Self-supervised methods can now rival fully supervised pre-training across multiple data types.
SCALOR learns scalable object representations for crowded scenes.
problem Scalability in scenes with many objects.
method Spatially-parallel attention and proposal-rejection mechanisms.
result SCALOR can handle up to a hundred objects in crowded scenes.
The paper tackles video prediction by estimating conditional densities implicitly.
problem Temporal prediction uncertainty and high-dimensional probabilistic inference in natural scenes.
method Score-based conditional density estimation using sequence-to-image networks trained on a resilience-to-noise objective.
result The method handles occlusion boundaries and weights predictive evidence by reliability.
Paper investigates trade-off between video bitrate and CNN accuracy.
problem Network bandwidth and energy consumption issues in video classification.
method Only retain motion vector and select texture information at reduced bitrates.
result Achieved 11%-94% saving in bitrate with marginal accuracy loss.
New deep-unfolded network improves video background separation.
problem Video foreground-background separation.
method Deep unfolding of an iterative RPCA algorithm with adaptive learning.
result Proposed network outperforms state-of-the-art in video foreground-background separation.
New RL method learns from passive data by modeling intentions.
problem Learning from passive data like videos without rewards or actions.
method Model intentions using temporal difference learning, learning representations from raw data.
result Successfully learns features from passive data that accelerate downstream RL tasks.
Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images. We propose to exploit proprioceptive motor signals to provide unsuperv…