New EM algorithms for weighted-data clustering improve audio-visual scene analysis.
problem Improving clustering of weighted data in heterogeneous environments.
method Proposed weighted-data Gaussian mixture model and two EM algorithms.
result Validation shows improved clustering in audio-visual scenes.
The paper analyzes how speech enhancement and recognition can be improved in noisy environments.
problem Improving speech recognition in multi-talker scenarios with limited resources.
method Developed and trained two LSTM-based models for speech enhancement and phone recognition, then studied their joint optimization.
result Joint optimization of speech enhancement and recognition leads to a significant reduction in Phone Error Rate (PER).
AV-CPL uses continuous pseudo-labels for AVSR combining labeled and unlabeled data.
problem Improving AVSR performance with labeled and unlabeled data.
method Semi-supervised method using continuous pseudo-labels generated by the same AVSR model.
result Significant improvements in VSR performance on LRS3 dataset.
Curiosity enhanced by audio-visual associations improves learning efficiency.
problem Challenges in reinforcement learning, especially predicting the future.
method Exploits multiple modalities (audio and vision) to predict novel associations.
result Improves exploration and learning efficiency in various environments.
AV-ASR system improves speech recognition with visual context.
problem Improving speech recognition accuracy with visual information.
method Transformer-based architecture with multiresolution and multimodal training.
result Multiresolution training speeds up convergence and improves WER by 18%.
Improved speech recognition with audio-visual fusion.
problem Enhance speech recognition accuracy in noisy conditions.
method Proposes an attention-based audio-visual fusion strategy to align and learn from acoustic and lip motion data.
result Significant improvements in recognition accuracy (7-30%) on TCD-TIMIT dataset.
AVDCNN combines audio and visual data for better speech enhancement.
problem Improving speech quality by reducing noise in audio signals.
method Proposes an AVDCNN model that integrates audio and visual streams into a unified deep CNN network for end-to-end training.
result AVDCNN outperforms audio-only and conventional SE methods in enhancing speech quality.
AVDCNN combines audio and visual data for better speech enhancement.
problem Improving speech quality in noisy environments.
method Multimodal deep CNNs model that integrates audio and visual streams.
result AVDCNN outperforms audio-only and conventional SE methods.
AVEC 2019 challenges AI in detecting depression and cross-cultural emotions.
problem Detecting depression and cross-cultural emotions from audiovisual data.
method Comparison of machine learning methods under standardized conditions.
result Baseline system performance on state-of-mind, depression, and cross-cultural tasks.
The paper tracks multiple speakers using audio and visual data.
problem Tracking multiple speakers with audio and visual data.
method Generative model with variational inference for latent variables.
result The proposed method outperforms baseline methods in real-world scenarios.
Paper proposes new principles and framework for AVC learning from user-generated videos.
problem Challenges in learning audio-visual correspondence from short-term user-generated videos.
method Introduced new principles and a framework to facilitate AVC learning from videos' themes.
result Proposed approach outperformed baseline by 23.15% on KWAI-AD-AudVis corpus.
New method uses overcomplete frames for better acoustic scene analysis.
problem Improving acoustic scene analysis in real-world applications.
method Risk minimization-based overcomplete frame thresholding.
result Validated on bird activity detection task using wavelets.
SCALOR learns scalable object representations for crowded scenes.
problem Scalability in scenes with many objects.
method Spatially-parallel attention and proposal-rejection mechanisms.
result SCALOR can handle up to a hundred objects in crowded scenes.
Two-stream model recognizes affect from audio and video.
problem Human affect recognition in real-world settings.
method Two-stream aural-visual analysis model with separate audio and visual processing.
result Model achieves promising results on Aff-Wild2 database.
A multi-head attention network improves ASC by recognizing overlapping sound patterns.
problem Challenging ASC due to overlapping sound patterns and complex event mixtures.
method Proposes a multi-head attention network to model complex temporal input structures.
result Achieved competitive performance on DCASE 2018 Task 5 dataset.
Tri-modal model predicts personality traits from audio, visual, and text data.
problem Predicting personality traits from multimodal data.
method Stacked Convolutional Neural Networks for each modality, multimodal fusion on decision-level and by concatenation.
result Multimodal fusion outperforms individual modalities, improving personality trait prediction by 9.4%.
A neural scene representation framework enforcing 3D transformations.
problem Learning 3D scene representations from images without 3D supervision.
method Introducing a loss enforcing equivariance of the scene representation with 3D transformations.
result Real-time neural rendering with comparable results to models requiring minutes for inference.
Combines multiple data types to predict emotions in images.
problem Predicting emotions in images using various data types.
method Combines facial features, scene extraction, audio tonality, human pose, text-based tagging, and CNN predictions.
result Improves accuracy in emotion prediction compared to baseline methods.
Polarimetric images enhance object detection in adverse weather conditions.
problem Object detection in road scenes is challenging in adverse weather conditions.
method Combining polarimetric imaging and deep learning.
result Polarimetry improves object detection by 20% to 50% compared to conventional RGB images.
GENESIS generates and samples 3D scenes by capturing object interactions.
problem Lack of models that explicitly capture object interactions in scene generation.
method Object-centric latent variables, spatial GMM, amortized inference, autoregressive prior.
result First object-centric generative model of 3D visual scenes.
Efficient model for foggy scene understanding in vehicles.
problem Challenging scene understanding and segmentation under foggy conditions.
method Domain adaptation and illumination-invariant image transformation.
result Outperforms state-of-the-art models in foggy scene understanding.
Bayesian framework refines scene interpretation with prior and posterior models.
problem Challenges in incorporating contextual relationships into scene recognition.
method Combines prior scene geometry and object spatial arrangement with classifier outputs.
result Progressive refinement of scene interpretation through mutual information maximization.
New framework segments 3D scenes using neural algorithms and sub-Riemannian geometry.
problem Effective scene segmentation in 3D vision.
method Neurogeometric sub-Riemannian model, harmonic analysis, neural-based stereo correspondence.
result Sub-Riemannian metric is central to effective scene segmentation.
Scene text magnifier enhances readability for visually impaired.
problem Helps visually impaired read natural scene text.
method Four CNN-based networks: character erasing, extraction, magnify, synthesis.
result Effective text magnification without background alteration.
New model separates objects in scenes, enabling novel arrangements and depth.
problem Lack of modular, compositional scene modeling in generative models.
method Ensemble of generative models (experts) compete for explaining different parts of a scene.
result Model generates scenes with novel object arrangement and depth ordering.
Sparse manifold transform linearizes non-linear signal transformations.
problem Non-linear signal transformations in sensory data.
method Combines sparse coding, manifold learning, and slow feature analysis.
result Models sparse discreteness and low-dimensional manifold structure in natural scenes.
We address the problems of multi-domain and single-domain regression based on distinct and unpaired labeled training sets for each of the domains and a large unlabeled training set from all domains. We formulate these problems as a Bayesian estimation with partial knowledge of statistical relations. We propose a worst-…
ROOTS learns to represent and render 3D scenes with object-centric models.
problem Learning to represent and render 3D scenes with object-centric compositionality.
method Probabilistic generative model for learning object representations and scene rendering from partial observations.
result The model can infer 3D object representations and render scenes from arbitrary viewpoints.
MONet learns to decompose scenes into meaningful components without supervision.
problem Learning meaningful scene decompositions without labeled data.
method MONet combines a VAE and recurrent attention network to learn decompositions of 3D scenes.
result MONet can learn to represent 3D scenes into meaningful components like objects and background.
SPACE models complex scenes by decomposing objects and backgrounds.
problem Scalability and unsupervised object-oriented scene representation learning.
method Generative latent variable model combining spatial-attention and scene-mixture approaches.
result SPACE achieves factorized object representations and decomposes complex scenes.
NeRF-VAE generates 3D scenes with geometric structure from few images.
problem Generating 3D scenes from few images with geometric consistency.
method Combines NeRF and VAE, incorporating shared geometric structure.
result NeRF-VAE can infer and render geometrically-consistent scenes from unseen environments.
Enhances sound texture in CNN for better acoustic scene classification.
problem Limited understanding of how CNNs perceive audio scenes.
method Used Class Activation Mapping (CAM) to analyze log-Mel features and proposed edge enhancement using DoG and Sobel operators.
result Edge-enhanced log-Mel features improve CNN performance in acoustic scene classification.
Systematic approach to generating diverse 3D scenes and images for training and benchmarking.
problem Training and benchmarking machine learning algorithms for scene understanding.
method Stochastic grammar-based pipeline for generating and rendering photorealistic 3D scenes with detailed ground truth.
result Improves performance in scene understanding tasks and provides controllable benchmarks for model diagnostics.
Improved acoustic scene classification with factorized CNN.
problem Acoustic scene classification in varying environments.
method Large-margin factorized CNN with triplet loss.
result Improved performance and better generalization on unseen data.
A new method for estimating joint value functions in multi-scene reinforcement learning.
problem High variance in samples for policy gradient computations in multi-scene environments.
method Sparse attention mechanism over multiple value function hypotheses to approximate the true joint value function.
result Significant improvements in reward scores and enhanced navigation efficiency across OpenAI ProcGen environments.
Generative models learn from unlabeled videos via object segmentation and scene modeling.
problem Learning generative models from unlabelled videos.
method Decomposed into three subtasks: motion segmentation, background and foreground modeling, and scene sampling.
result Approach allows learning models that generalize beyond occlusions and represent scenes in a modular fashion.
Pix2Shape learns 3D scene representations from single images without supervision.
problem Learning 3D scene information from a single image without supervision.
method Pix2Shape uses an encoder, decoder, and critic network to generate 2.5D surfel-based reconstructions.
result Pix2Shape can generate complex 3D scenes from a single image, scaling with on-screen resolution.
Generative Multisensory Network learns 3D scene representations from multiple modalities.
problem Learning robust 3D scene representations from multiple sensory modalities.
method Amortized Product-of-Experts for efficient inference and cross-modal generation.
result The model can infer modality-invariant 3D scene representations efficiently from various sensory modalities.
Model removes objects from general scenes using weak supervision.
problem Automatic object removal from general scene images with weak supervision.
method Two-stage editor architecture with mask generator and image in-painter; novel GAN prior for mask generator.
result Effectively removes a wide variety of objects from general scenes using weak supervision.
Paper improves reinforcement learning in multi-scene tasks.
problem Reducing sample variance in multi-scene reinforcement learning.
method Sparse dynamic value estimation using Gaussian mixture models.
result Significant improvements in reward scores and navigation efficiency.
RICH models scenes as hierarchical tree to learn and generate complex compositions.
problem Learning compositional structures between parts and objects in natural scenes.
method RICH uses a latent scene graph to organize entities into a tree structure and employs a top-down inference approach.
result RICH learns and generates complex scene hierarchies from unlabeled data.
This paper analyzes sound event detection in synthetic office audio, comparing different systems.
problem Comparing sound event detection systems in synthetic office audio.
method Analysis of systems submitted to DCASE 2016 task, using synthetic office sounds.
result Statistical analysis of results, highlighting system performance under controlled conditions.
Improves reinforcement learning agent's scene-specific value function.
problem High variance in samples for policy gradient computations in multi-scene environments.
method Proposes dynamic value estimation (DVE) for multiple MDPs, clustering value functions across scenes.
result Lower sample variance and more accurate scene-specific value function estimates.
Proposes a deep structured generative model for complex scenes.
problem Difficulty in generating images with complicated structures.
method Boosts GANs with structure information encoded by a stochastic and-or graph (sAOG).
result Successfully captures and generates images with complex structures.
A novel method for visual question answering using scene graphs and reinforcement learning.
problem Answering free-form questions about images with deep linguistic and visual understanding.
method Context-driven, sequential reasoning based on scene graphs and reinforcement learning.
result Our method almost reaches human performance on the GQA dataset.
A new method to decompose audio scenes using deep learning.
problem Understanding audio scenes with random microphone arrangements.
method Formulated a neural network for nonnegative tensor factorization.
result Learned sources' individual spectral dictionaries and activation patterns.
Generative model creates realistic scenes from pixel-wise labels.
problem Creating photo-realistic scenes from pixel-wise labels.
method Semantic bottleneck GAN model combining conditional and unconditional generation networks.
result Model outperforms state-of-the-art models in unsupervised image synthesis.
P3I learns holistic scene representations from a single image.
problem Inferring camera poses, object locations, and global scene structures from a single image.
method Combines search-based and gradient-based algorithms.
result P3I outperforms baselines on various image manipulation tasks.