New approach learns traffic behavior from unlabelled video.
problem Learning from demonstration in settings with unlabelled raw video data.
method Calibrates camera, detects objects, tracks them, and uses trajectories for LfD.
result Can learn traffic behavior from raw video without additional expert knowledge.
V-SysId identifies keypoints and 3D system from unlabeled videos.
problem Identifying keypoints and 3D system from unlabeled videos.
method Alternates between parameter estimation and extrinsic camera calibration, using motion equations as weak supervision.
result Utility of the approach demonstrated across various settings.
Discover equations of motion from distorted video frames.
problem Learning equations of motion from unlabeled, distorted video.
method Train an autoencoder to map frames into latent space, then use symbolic regression to find differential equations.
result The method can discover motion equations even when video is distorted.
Generative models learn from unlabeled videos via object segmentation and scene modeling.
problem Learning generative models from unlabelled videos.
method Decomposed into three subtasks: motion segmentation, background and foreground modeling, and scene sampling.
result Approach allows learning models that generalize beyond occlusions and represent scenes in a modular fashion.
Visual reranking is effective to improve the performance of the text-based video search. However, existing reranking algorithms can only achieve limited improvement because of the well-known semantic gap between low level visual features and high level semantic concepts. In this paper, we adopt interactive video search…
This paper presents two unsupervised learning layers (UL layers) for label-free video analysis: one for fully connected layers, and the other for convolutional ones. The proposed UL layers can play two roles: they can be the cost function layer for providing global training signal; meanwhile they can be added to any re…
Approach to detect and adapt to concept drift in unlabeled streaming data.
problem Detect and adapt to concept drift in high-dimensional, noisy, low-context data.
method Density-based clustering for virtual drift and weak supervision for real drift.
result 90% precision in detecting and adapting to concept drift for 4 years after initial deployment.
Recent works demonstrated the usefulness of temporal coherence to regularize supervised training or to learn invariant features with deep architectures. In particular, enforcing smooth output changes while presenting temporally-closed frames from video sequences, proved to be an effective strategy. In this paper we pro…
In this paper, we present a technique for unsupervised learning of visual representations. Specifically, we train a model for foreground and background classification task, in the process of which it learns visual representations. Foreground and background patches for training come af- ter mining for such patches from …
AV-CPL uses continuous pseudo-labels for AVSR combining labeled and unlabeled data.
problem Improving AVSR performance with labeled and unlabeled data.
method Semi-supervised method using continuous pseudo-labels generated by the same AVSR model.
result Significant improvements in VSR performance on LRS3 dataset.
Paper proposes an active learning method for surgical workflow recognition using long-range temporal dependency.
problem Challenges in automatic surgical workflow recognition due to lack of large-scale labelled datasets.
method NL-RCNet with non-local block for capturing long-range temporal dependency and intra-clip dependency score for selection.
result Our approach outperforms state-of-the-art methods by selecting only 50% of samples for training.
Dreamer 4 learns Minecraft tasks from videos alone.
problem Accurately predicting object interactions in complex environments.
method Reinforcement learning inside a fast, accurate world model.
result Dreamer 4 outperforms previous models in Minecraft, learning from only offline data.
PREREQ learns concept prerequisites from online educational resources.
problem Inferring prerequisite relations between educational concepts.
method PREREQ uses latent representations of concepts from Pairwise Latent Dirichlet Allocation and a Siamese neural network to learn from course prerequisites and labeled data.
result PREREQ outperforms state-of-the-art approaches and can learn from less data.
Dataset for object detection at Oktoberfest beer tent.
problem Realistic object detection in a busy, diverse setting.
method Hand-annotated 1,110 images, provided trained models.
result Challenging dataset for object detection.
BootsTAP uses real-world data to improve TAP tracking performance.
problem Lack of large-scale groundtruth training data for TAP.
method Bootstrapped training with a self-supervised student-teacher setup.
result State-of-the-art performance on TAP-Vid benchmarks.
We propose a framework that learns a representation transferable across different domains and tasks in a label efficient manner. Our approach battles domain shift with a domain adversarial loss, and generalizes the embedding to novel task using a metric learning-based approach. Our model is simultaneously optimized on …
Study quantifies interactions between unlabeled multimodal data.
problem Understanding how modalities combine in semi-supervised settings.
method Information-theoretic definitions and bounds derivation.
result Validated lower and upper bounds accurately track true interactions.
Ground-A-Video edits videos without training, preserving intended changes.
problem Complex multi-attribute video editing with omitted or wrong changes.
method Grounding-guided video-to-video translation with Cross-Frame Gated Attention.
result Zero-shot multi-attribute video editing with improved accuracy and frame consistency.
OpenEDS dataset captures eye images for VR applications.
problem Lack of high-quality eye image datasets for VR applications.
method Captured eye images using VR head-mounted display with synchronized cameras.
result Baseline experiment achieved 98.3% mIoU for pupil, iris, sclera segmentation.
CB-GLNs learn video data's complex dependencies via graph representation.
problem Capturing complex dependency structures in sequential data like videos.
method Represent video data as a graph, find compositional dependencies via graph-cut and message passing.
result CB-GLNs efficiently learn video data's semantic compositional structure.
RaMViD uses diffusion models for video prediction and infilling.
problem Predicting and infilling missing information in videos.
method Extends image diffusion models to videos using 3D convolutions and a new conditioning technique.
result Achieves state-of-the-art results on video prediction benchmarks.
Current deep learning results on video generation are limited while there are only a few first results on video prediction and no relevant significant results on video completion. This is due to the severe ill-posedness inherent in these three problems. In this paper, we focus on human action videos, and propose a gene…
New metric for video generation models, addressing quality, coherence, and diversity.
problem Lack of metrics for evaluating generative models of video.
method Proposed Fréchet Video Distance (FVD) and StarCraft 2 Videos (SCV) benchmark.
result FVD correlates well with human judgment of generated videos.
LumièreNet creates lecture videos from audio narration.
problem Creating high-quality lecture videos from audio narration.
method Modular deep-learning architecture that learns from audio to video.
result Synthesizes high-quality lecture videos from any length of audio.
Paper defends against adversarial videos by detecting and reducing imperceptible perturbations.
problem Adversarial videos can fool well-trained video classification models.
method Temporal consistency between frames and spatial denoising to detect and reduce perturbations.
result The proposed method significantly improves robustness against adversarial attacks.
Paper improves video categorization using temporal coherence.
problem Video categorization in multiple modalities.
method Temporal coherence-based regularization for multimodal models.
result Models with temporal coherence outperform state-of-the-art.
Improves video search by balancing text and visual modalities.
problem Modality imbalance in video search models, focusing mainly on text matching.
method Proposes MBVR with MS samples and DM to balance modalities.
result Empirically shows significant improvement in modality balance and search effectiveness.
A new video prediction model treats videos as continuous processes, reducing sampling steps and improving efficiency.
problem Efficiency and temporal coherence in video prediction models.
method Treats videos as a continuous multi-dimensional process, reducing sampling steps.
result Reduction of 75% sampling steps, state-of-the-art performance on benchmark datasets.
UDVD uses deep learning to denoise videos without supervision.
problem Lack of clean video data for training deep learning models.
method UDVD is a CNN trained solely on noisy video data, adapting to local motion.
result UDVD performs as well as supervised methods, even with limited training data.
Proposes TDNs for learning complex video structures.
problem Complex temporal dependencies in sequential data, especially videos.
method Temporal Dependency Networks (TDNs) using graph representations and graph convolutions.
result Efficiently learns complex semantic structures of video data.
SummaryNet automates video summarisation using deep learning.
problem Creating informative video summaries from videos.
method Two-stream convolutional network for spatial and temporal features, encoder-decoder model for salient features, sigmoid regression with LSTM for frame probability.
result SummaryNet achieves comparable or better results than state-of-the-art methods on benchmark datasets.
Paper proposes SMFN for high-res spherical video super-resolution.
problem Super-resolution of 360-degree panoramic videos is expensive and challenging.
method Deformable convolutions, mixed attention mechanism, dual learning strategy, weighted mean square error loss function.
result The proposed SMFN method improves super-resolution of equatorial regions in 360-degree videos.
Generative model produces high-fidelity video samples.
problem Generating high-fidelity videos from complex datasets.
method Large GAN trained on Kinetics-600 dataset, using a computationally efficient discriminator.
result Achieved state-of-the-art metrics in video synthesis and prediction.
New adversarial method detects abnormal trajectories from unlabeled data.
problem Detecting abnormal events from trajectory data without manual threshold setting.
method Generative adversarial network (GAN) framework for unsupervised learning.
result Best accuracy for abnormal trajectory detection on urban traffic videos.
The problem of feature disentanglement has been explored in the literature, for the purpose of image and video processing and text analysis. State-of-the-art methods for disentangling feature representations rely on the presence of many labeled samples. In this work, we present a novel method for disentangling factors …
Generative model learns compact codes for video recovery.
problem Efficiently represent and reconstruct videos from missing data.
method Generative network trained to map compact latent codes to images, with low-rank and similarity constraints.
result Can recover true video sequences even if not in pretrained network's range.
Paper proposes new principles and framework for AVC learning from user-generated videos.
problem Challenges in learning audio-visual correspondence from short-term user-generated videos.
method Introduced new principles and a framework to facilitate AVC learning from videos' themes.
result Proposed approach outperformed baseline by 23.15% on KWAI-AD-AudVis corpus.
The extension of image generation to video generation turns out to be a very difficult task, since the temporal dimension of videos introduces an extra challenge during the generation process. Besides, due to the limitation of memory and training stability, the generation becomes increasingly challenging with the incre…
Lower-dimensional video discriminators improve GAN performance.
problem High curvature in unconstrained video discriminator loss surfaces.
method Proposed Lower-Dimensional Video Discriminators (LDVD GANs).
result LDVD GANs double Temporal-GAN performance and achieve state-of-the-art performance.
System automates discovery and classification of training videos for career progression.
problem Difficulties in planning and navigating career paths due to changing job requirements and emerging sectors.
method Extracted educational videos, built a machine learning classifier, and optimized probability thresholds.
result Significant improvements in model performance by incorporating video attributes.
With the growth of user-generated content, we observe the constant rise of the number of companies, such as search engines, content aggregators, etc., that operate with tremendous amounts of web content not being the services hosting it. Thus, aiming to locate the most important content and promote it to the users, the…
Scalable system predicts hot videos for peak VOD service.
problem Improving peak service quality of video on demand.
method Two neural networks: clustering and dispatch policy. Clustering reduces video numbers, dispatch policy ranks videos with probabilities. Networks are trained end-to-end.
result Average prediction accuracy of 17% compared to 3% baseline, for same number of dispatches.
Paper proposes graph-based separable transforms for video coding.
problem Improving video coding efficiency by better capturing residual block statistics.
method Derives graph-based separable transforms (GBSTs) from line graphs with weights determined by parameters.
result GBSTs achieve about 0.4% average coding gain over existing transforms in VVC.
Jointly trains images and videos using residual vectors.
problem Generating high-quality videos from images and vice versa.
method Simultaneously learns latent variables for images and videos using residual vectors.
result Improves sample quality and diversity in video generation and image generation.
Paper proposes detecting video manipulation using stream descriptors.
problem Misuse of manipulated video content.
method Binary classifiers on multimedia stream descriptors.
result Scalable approach can detect high-quality manipulations.
Paper introduces adversarial lossy compression for video artifacts reduction.
problem Unpleasant reconstruction artifacts in standard video coding schemes at low bit-rates.
method Adversarial lossy video compression model minimizing an adversarial distortion objective.
result Reduction of perceptual artifacts and detail reconstruction under extreme compression.
Advanced video classification systems decode video frames to derive the necessary texture and motion representations for ingestion and analysis by spatio-temporal deep convolutional neural networks (CNNs). However, when considering visual Internet-of-Things applications, surveillance systems and semantic crawlers of la…
This paper highlights the need for explainable AI in video deep learning models.
problem Lack of explainable AI methods for video deep learning models.
method Illustrates the current state of video deep learning and highlights the need for explainability methods.
result Current explainability methods for video deep learning are insufficient and need improvement.