Ground-A-Video edits videos without training, preserving intended changes.
problem Complex multi-attribute video editing with omitted or wrong changes.
method Grounding-guided video-to-video translation with Cross-Frame Gated Attention.
result Zero-shot multi-attribute video editing with improved accuracy and frame consistency.
RaMViD uses diffusion models for video prediction and infilling.
problem Predicting and infilling missing information in videos.
method Extends image diffusion models to videos using 3D convolutions and a new conditioning technique.
result Achieves state-of-the-art results on video prediction benchmarks.
Jointly trains images and videos using residual vectors.
problem Generating high-quality videos from images and vice versa.
method Simultaneously learns latent variables for images and videos using residual vectors.
result Improves sample quality and diversity in video generation and image generation.
UDVD uses deep learning to denoise videos without supervision.
problem Lack of clean video data for training deep learning models.
method UDVD is a CNN trained solely on noisy video data, adapting to local motion.
result UDVD performs as well as supervised methods, even with limited training data.
System automates discovery and classification of training videos for career progression.
problem Difficulties in planning and navigating career paths due to changing job requirements and emerging sectors.
method Extracted educational videos, built a machine learning classifier, and optimized probability thresholds.
result Significant improvements in model performance by incorporating video attributes.
Sideways trains video models by overwriting activations as new frames arrive, potentially improving generalization.
problem Training deep video models synchronously slows down and requires storing activations, limiting parallelism.
method Sideways trains video models by overwriting activations as new frames arrive, breaking the precise correspondence between gradients and activations.
result Sideways training can converge and potentially generalize better than standard synchronized backpropagation.
Unified model trained on images and videos using masked autoencoding.
problem Training a single model for multiple visual modalities.
method Masked autoencoding on a Vision Transformer.
result Unified model achieves comparable or better performance than single-modality models.
Current deep learning results on video generation are limited while there are only a few first results on video prediction and no relevant significant results on video completion. This is due to the severe ill-posedness inherent in these three problems. In this paper, we focus on human action videos, and propose a gene…
AMS improves video inference on edge devices by adapting a small model with online knowledge distillation.
problem High computation cost of Deep Neural Networks for real-time video inference on edge devices.
method AMS uses a remote server to continually train and adapt a small model on edge devices, using online knowledge distillation from a large model.
result 0.4--17.8 percent mean Intersection-over-Union improvement in video semantic segmentation.
New algorithm disentangles latent space in GANs using video sequences.
problem Learning disentangled latent spaces in GANs without supervision.
method Adversarial training with video sequences, modifying standard GAN algorithm.
result Disentangled latent space into content and motion attributes.
Improves video search by balancing text and visual modalities.
problem Modality imbalance in video search models, focusing mainly on text matching.
method Proposes MBVR with MS samples and DM to balance modalities.
result Empirically shows significant improvement in modality balance and search effectiveness.
Paper improves video feature learning for better downstream tasks.
problem Improving video feature learning for better performance on downstream tasks.
method Self-supervised learning approach using contrastive bidirectional transformer, extending BERT for real-valued feature vectors.
result Significantly improved performance on video classification, captioning, and segmentation tasks.
New method learns robot actions from videos without explicit labels.
problem Training robots to perform tasks from few demonstrations.
method Uses images and text for task-agnostic and general representation, synthesizes hallucinated actions, and applies dense correspondences.
result Trains robot policies solely from RGB videos, achieving diverse tasks across different robots and environments.
Proposes a defense method against multiple adversarial video types.
problem Lack of multi-perturbation robustness in existing defense approaches.
method Adversarial training with multiple independent BN layers and a BN selection module.
result Demonstrates stronger multi-perturbation robustness against different adversarial video types.
Automated video conferencing system improves user experience with ASD and VC.
problem Improve remote video conferencing experience through automated speaker detection and virtual cinematography.
method Uses 4K wide-FOV camera, depth camera, and microphone array to extract features and train machine learning models for ASD and VC.
result System performs within 0.3 MOS of an expert cinematographer, as rated by users.
We present a solution to "Google Cloud and YouTube-8M Video Understanding Challenge" that ranked 5th place. The proposed model is an ensemble of three model families, two frame level and one video level. The training was performed on augmented dataset, with cross validation.
Detect objects from motion without annotations.
problem Weakly supervised object detection.
method Train model on videos of moving objects and negative scenes.
result Detects objects in single images without annotations.
Learning visual feature representations for video analysis is a daunting task that requires a large amount of training samples and a proper generalization framework. Many of the current state of the art methods for video captioning and movie description rely on simple encoding mechanisms through recurrent neural networ…
Discover equations of motion from distorted video frames.
problem Learning equations of motion from unlabeled, distorted video.
method Train an autoencoder to map frames into latent space, then use symbolic regression to find differential equations.
result The method can discover motion equations even when video is distorted.
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
The usage of deep generative models for image compression has led to impressive performance gains over classical codecs while neural video compression is still in its infancy. Here, we propose an end-to-end, deep generative modeling approach to compress temporal sequences with a focus on video. Our approach builds upon…
Generative model produces high-fidelity video samples.
problem Generating high-fidelity videos from complex datasets.
method Large GAN trained on Kinetics-600 dataset, using a computationally efficient discriminator.
result Achieved state-of-the-art metrics in video synthesis and prediction.
New video super-resolution method robust to multiple degradation models.
problem Current methods are not robust to mismatch between training and testing degradation models.
method Convolutional Neural Network with pseudo-inverse image formation model and perceptual losses.
result Outperforms current state-of-the-art methods and is robust to multiple degradations.
Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that a spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The…
Paper proposes a new LSTM model for spatio-temporal learning.
problem Challenging video tasks require learning long-term spatio-temporal correlations.
method Introduces a higher-order convolutional LSTM model with tensor train decomposition.
result Model achieves state-of-the-art performance with significantly fewer parameters.
This paper presents two unsupervised learning layers (UL layers) for label-free video analysis: one for fully connected layers, and the other for convolutional ones. The proposed UL layers can play two roles: they can be the cost function layer for providing global training signal; meanwhile they can be added to any re…
Paper defends against adversarial videos by detecting and reducing imperceptible perturbations.
problem Adversarial videos can fool well-trained video classification models.
method Temporal consistency between frames and spatial denoising to detect and reduce perturbations.
result The proposed method significantly improves robustness against adversarial attacks.
Proposes a new video attack method that multiplies perturbation to improve model robustness.
problem Challenges existing defense methods for video recognition models against additive adversarial attacks.
method Introduces Multiplicative Adversarial Videos (MultAV) to impose perturbation by multiplication.
result Model trained against additive attacks is less robust to MultAV.
Deep networks are able to learn highly predictive models of video data. Due to video length, a common strategy is to train them on small video snippets. We apply the deep Taylor / LRP technique to understand the deep network's classification decisions, and identify a "border effect": a tendency of the classifier to loo…
New method compresses facial videos using GANs and latent space optimization.
problem Efficiently compressing facial videos at low bit rates.
method Leverages StyleGAN for latent space representation and compression, learns optimal compression through entropy model and perceptual loss.
result Significantly reduces perceptual distortion at low bit rates compared to state-of-the-art codecs.
The extension of image generation to video generation turns out to be a very difficult task, since the temporal dimension of videos introduces an extra challenge during the generation process. Besides, due to the limitation of memory and training stability, the generation becomes increasingly challenging with the incre…
We design a dispatch system to improve the peak service quality of video on demand (VOD). Our system predicts the hot videos during the peak hours of the next day based on the historical requests, and dispatches to the content delivery networks (CDNs) at the previous off-peak time. In order to scale to billions of vide…
Finding compact representation of videos is an essential component in almost every problem related to video processing or understanding. In this paper, we propose a generative model to learn compact latent codes that can efficiently represent and reconstruct a video sequence from its missing or under-sampled measuremen…
Paper introduces DACAL for high-resolution photo and video enhancement.
problem Photo and video enhancement with weak supervision.
method Divide-and-conquer adversarial learning approach with hierarchical decomposition.
result State-of-the-art performance in high-resolution photo and video enhancement.
C3 compresses images and videos with low complexity and high performance.
problem High complexity and low performance in neural compression models.
method Overfits a small model to each image or video separately, improving RD performance with low complexity.
result Matches the RD performance of state-of-the-art neural and video codecs with significantly lower decoding complexity.
Method learns from video demonstrations with human feedback.
problem Teaching autonomous agents using video demonstrations and human feedback.
method Constructs a mapping between standard and visual representations using a neural network.
result Effective in teaching a hopper agent to perform a backflip with minimal human feedback.
Generates coherent 3D scenes from monocular videos without supervision.
problem Lack of 3D scene modeling in video generation models.
method Trains a model to generate 3D scenes with moving objects and a background from monocular videos.
result Trained model generates coherent 3D scenes with multiple moving objects and a background.
Making predictions of future frames is a critical challenge in autonomous driving research. Most of the existing methods for video prediction attempt to generate future frames in simple and fixed scenes. In this paper, we propose a novel and effective optical flow conditioned method for the task of video prediction wit…
Paper presents a new video compression method using autoencoders.
problem Efficient video compression with reduced quality loss.
method 3D autoencoder with discrete latent space and autoregressive prior trained jointly.
result Method outperforms state-of-the-art learned video compression networks.
Is cognition a collection of loosely connected functions tuned to different tasks, or can there be a general learning algorithm? If such an hypothetical general algorithm did exist, tuned to our world, could it adapt seamlessly to a world with different laws of nature? We consider the theory that predictive coding is s…
New model predicts video sequences with latent dynamics.
problem Predicting video sequences with inherent uncertainty.
method Introduces a novel stochastic temporal model with latent dynamics.
result Latent model outperforms prior state-of-the-art methods.
Generating video frames that accurately predict future world states is challenging. Existing approaches either fail to capture the full distribution of outcomes, or yield blurry generations, or both. In this paper we introduce an unsupervised video generation model that learns a prior model of uncertainty in a given en…
Efficiently learns 3D convolutions with less data.
problem High parameter and data costs in 3D convolutions.
method Temporal factorization of 3D kernels.
result Significantly reduces training data requirement and parameter count.
Deep reinforcement learning achieves superhuman performance in a range of video game environments, but requires that a designer manually specify a reward function. It is often easier to provide demonstrations of a target behavior than to design a reward function describing that behavior. Inverse reinforcement learning …
Paper introduces TAP-Vid, a benchmark for tracking any point in videos.
problem Tackles the problem of tracking arbitrary physical points on surfaces over longer video clips.
method Formalizes the problem as TAP, introduces TAP-Vid benchmark, uses crowdsourced pipeline with optical flow estimates, proposes TAP-Net model.
result TAP-Net outperforms all prior methods on TAP-Vid benchmark when trained on synthetic data.
New method tackles anomaly detection in video surveillance using continual learning.
problem Challenges in continual learning for high-dimensional applications like video surveillance.
method Transfer learning and continual learning for online anomaly detection.
result Significantly reduces training complexity and continual learning from recent data.
New method tackles video inverse problems using image diffusion models.
problem Spatio-temporal degradation in video inverse problems.
method Leverages image diffusion models to treat time dimension as batch dimension, introduces batch-consistent diffusion sampling.
result Achieves state-of-the-art reconstructions for various spatio-temporal degradations.
This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering. Multi-modal video question answering is an important task that aims at the joint understanding of vision and language. However, establishing large scale dataset for multi-modal video question answeri…