Paper proposes ICCN to learn correlations between text, audio, and video for multimodal sentiment analysis.
problem Improving multimodal sentiment analysis by learning hidden correlations between text and audio/video features.
method Interaction Canonical Correlation Network (ICCN) using deep canonical correlation analysis (DCCA).
result Empirical results confirm the effectiveness of ICCN in capturing useful information from all three views.
End-to-end probabilistic inference improves audio signal processing.
problem Efficiently processing large audio signals with varying characteristics.
method Formulated a spectral mixture Gaussian process model with nonstationary priors, enabling infinite-horizon Gaussian process regression.
result The method outperforms standard techniques in processing audio signals with hundreds of thousands of data points.
Generative model improves audio synthesis from TF features.
problem Challenges in generating high-quality audio from TF features.
method Used a GAN to generate invertible TF features from short-time Fourier transforms.
result Generated TF features from GAN outperformed direct waveform generation.
Deep learning predicts mental disorders from audio and text samples.
problem Predicting mental disorders from speech samples.
method Multimodal deep learning structure using various pre-trained models for audio and text embeddings, transfer learning, and auxiliary corpora.
result Acceptable accuracy in predicting mental disorders through multimodal analysis.
Two-stream model recognizes affect from audio and video.
problem Human affect recognition in real-world settings.
method Two-stream aural-visual analysis model with separate audio and visual processing.
result Model achieves promising results on Aff-Wild2 database.
Gaussian process (GP) audio source separation is a time-domain approach that circumvents the inherent phase approximation issue of spectrogram based methods. Furthermore, through its kernel, GPs elegantly incorporate prior knowledge about the sources into the separation model. Despite these compelling advantages, the c…
A model classifies music genres from MP3 files using metric learning and feature extraction.
problem Classifying music genres from MP3 files efficiently and accurately.
method Metric learning and feature extraction using MFCC and PCA.
result Promising results in classification accuracy compared to baseline algorithms.
Automates deep model selection for audio source separation.
problem Selecting the right deep learning model for audio source separation.
method Confidence measure based on model output quality estimation.
result Confidence-based ensemble approach significantly outperforms random and oracle selections.
Paper presents an audiovisual model to recognize sounds from weakly labeled video data.
problem Sound recognition from weakly labeled video data.
method Audiovisual fusion model with attention mechanism.
result The model achieves a mean Average Precision (mAP) of 46.16 on AudioSet, outperforming state-of-the-art models.
This paper improves sentiment classification by combining text, audio, and video data using DCCA.
problem Improving sentiment classification accuracy using multi-modal data.
method Deep Canonical Correlation Analysis (DCCA) for combining text, audio, and video embeddings.
result One-Step DCCA outperforms current state-of-the-art in multi-modal embedding learning.
CrossFilter tackles noisy labels in audio tagging.
problem Noisy labels in large audio datasets.
method CrossFilter framework using multiple representations and multi-task learning.
result Improves audio tagging performance on FSDKaggle2018 and FSDKaggle2019 datasets.
Real music signals are highly variable, yet they have strong statistical structure. Prior information about the underlying physical mechanisms by which sounds are generated and rules by which complex sound structure is constructed (notes, chords, a complete musical score), can be naturally unified using Bayesian modell…
Proposes a self-supervised method for generating spatial audio from monaural audio and video.
problem Generating spatial audio from monaural audio and video recordings is challenging and expensive.
method Uses a self-supervised network with an auxiliary classifier to classify video channels and generate spatial audio.
result The proposed method effectively generates spatial audio from monaural audio and video.
Deep audio prior uses neural networks to solve audio problems without data.
problem Challenging audio problems like source separation, editing, and synthesis.
method Randomly-initialized neural network with carefully designed audio prior.
result Superior audio results on Universal-150 benchmark dataset.
Graph neural networks improve music genre classification on audio datasets.
problem Difficulty in applying deep learning on spectrograms due to lack of quality data and augmentation.
method Combination of CNN and Graph Neural Networks (GNN) with Siamese Neural Networks.
result Achieved state-of-the-art results on GTZAN and AudioSet datasets.
Zero-shot audio classification using class label embeddings.
problem Classifying audio without labeled data.
method Bilinear model with audio feature embeddings and class label embeddings.
result Achieved accuracy up to 39.7% for natural audio categories.
The paper analyzes how speech enhancement and recognition can be improved in noisy environments.
problem Improving speech recognition in multi-talker scenarios with limited resources.
method Developed and trained two LSTM-based models for speech enhancement and phone recognition, then studied their joint optimization.
result Joint optimization of speech enhancement and recognition leads to a significant reduction in Phone Error Rate (PER).
We propose a method for recognizing moving vehicles, using data from roadside audio sensors. This problem has applications ranging widely, from traffic analysis to surveillance. We extract a frequency signature from the audio signal using a short-time Fourier transform, and treat each time window as an individual data …
Wearables like smartwatches which are embedded with sensors and powerful processors, provide a strong platform for development of analytics solutions in sports domain. To analyze players' games, while motion sensor based shot detection has been extensively studied in sports like Tennis, Golf, Baseball; Table Tennis and…
Recently, there has been great interest in the field of audio style transfer, where a stylized audio is generated by imposing the style of a reference audio on the content of a target audio. We improve on the current approaches which use neural networks to extract the content and the style of the audio signal and propo…
As part of the 2016 public evaluation challenge on Detection and Classification of Acoustic Scenes and Events (DCASE 2016), the second task focused on evaluating sound event detection systems using synthetic mixtures of office sounds. This task, which follows the `Event Detection - Office Synthetic' task of DCASE 2013,…
Machine learning detects frog calls in audio recordings with high accuracy.
problem Estimating frog presence in audio recordings from tropical environments.
method Gaussian mixture model classification of frog calls with 20 cepstral features.
result The algorithm achieved an average weighted error rate of 0.9% for nine species classification.
This paper explores how different audio signal representations affect topological signatures and their predictive power.
problem The impact of different signal representations on topological signatures and their predictive power.
method The study compares three different signal representations (embedding, spectrogram, and spectrogram zeroes) and evaluates their topological signatures for speaker gender, vowel type, and individual prediction.
result Topological signatures from spectrogram zeroes offer the best improvement for gender prediction, and different representations are complementary.
We propose an efficient end-to-end convolutional neural network architecture, AclNet, for audio classification. When trained with our data augmentation and regularization, we achieved state-of-the-art performance on the ESC-50 corpus with 85:65% accuracy. Our network allows configurations such that memory and compute r…
Method learns audio embeddings with contextualized tags.
problem Align audio and tags for cross-modal tasks.
method Audio autoencoder, word embeddings, multi-head self-attention, contrastive loss.
result Multi-head self-attention improves audio representations.
Study improves radio show segmentation using audio embeddings.
problem Automated segmentation of radio shows.
method Created audio embeddings from multi-class classification tasks on different datasets, evaluated performance against text-only baseline.
result Audio embeddings from non-speech sound event classification significantly outperformed text-only baseline by 32.3% in F1-measure.
Online audio advertising is a particular form of advertising used abundantly in online music streaming services. In these platforms, which tend to host tens of thousands of unique audio advertisements (ads), providing high quality ads ensures a better user experience and results in longer user engagement. Therefore, th…
Computational paralinguistic analysis is increasingly being used in a wide range of cyber applications, including security-sensitive applications such as speaker verification, deceptive speech detection, and medical diagnostics. While state-of-the-art machine learning techniques, such as deep neural networks, can provi…
Bayesian model reconstructs time and frequency data robustly.
problem Missing observations and noise in time/frequency data.
method Probabilistic model, Bayesian update, joint reconstruction.
result Effective joint time/frequency reconstruction with missing data.
Meta-learning improves few-shot acoustic event detection.
problem Detecting new audio events with limited labeled data.
method Formulated few-shot AED problem; explored supervised and meta-learning approaches.
result Meta-learning achieves superior performance in few-shot AED.
A new VAD method uses respiration patterns from video to detect speech.
problem Improving VAD performance in noisy audio recordings.
method Extract respiration patterns from video, use neural models to detect speech.
result Efficacy demonstrated through experiments on real acoustic environments.
Adversarial attacks on spectrograms can fool audio classifiers trained on waveforms.
problem Susceptibility of audio classifiers to adversarial attacks on spectrograms.
method Applying adversarial attacks to spectrograms and reconstructing audio waveforms.
result Perturbed spectrograms can fool 2D CNNs and 1D CNNs trained on audio waveforms.
Study improves animal audio classification using data augmentation.
problem Improving automated animal audio classification accuracy.
method Exploits different data augmentation techniques for training CNNs.
result Best recognition rates on animal audio classification datasets.
MicAugment transfers audio style from few seconds of input to match target conditions.
problem Audio model robustness to diverse acquisition conditions.
method Identifies and applies transformations learned from target audio to input audio.
result MicAugment significantly improves model robustness in downstream tasks.
Detects audio adversarial examples using anomalous pattern detection.
problem Identifies adversarial audio attacks in deep neural networks.
method Applies anomalous pattern detection in activation space of audio models.
result Can detect adversarial examples with up to 0.98 AUC, no degradation on benign samples.
Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…
Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…
Unified model for audio control and style transfer.
problem Explicit control and style transfer in music generation.
method Diffusion autoencoders for semantic feature extraction, disentanglement using adversarial criterion.
result Model generates audio matching timbre targets with specified structure.
Audio fingerprinting, also named as audio hashing, has been well-known as a powerful technique to perform audio identification and synchronization. It basically involves two major steps: fingerprint (voice pattern) design and matching search. While the first step concerns the derivation of a robust and compact audio si…
Transformer model estimates keywords for better audio captioning.
problem Indeterminacy in word selection for audio events/scenes.
method Transformer-based model with keyword estimation.
result Achieved state-of-the-art performance in AAC.
Wavelet Attribution Method (WAM) improves feature attribution for deep models.
problem Inability of pixel-based heatmaps to capture data structure and variability in feature attribution.
method Wavelet domain for feature attribution, leveraging spatial and scale-localized properties of wavelet coefficients.
result WAM provides quantitatively superior explanations across audio, image, and volume modalities.
GANSynth uses GANs to efficiently synthesize high-fidelity audio.
problem Efficient and high-fidelity audio synthesis is challenging.
method Model log magnitudes and instantaneous frequencies with GANs.
result GANSynth outperforms WaveNet on automated and human evaluation metrics.
LumièreNet creates lecture videos from audio narration.
problem Creating high-quality lecture videos from audio narration.
method Modular deep-learning architecture that learns from audio to video.
result Synthesizes high-quality lecture videos from any length of audio.
Review of deep learning techniques for audio signal processing.
problem Improving audio signal processing using deep learning.
method Analysis of various deep learning models and techniques.
result Advancements in speech, music, and environmental sound processing.
This paper describes Task 2 of the DCASE 2018 Challenge, titled "General-purpose audio tagging of Freesound content with AudioSet labels". This task was hosted on the Kaggle platform as "Freesound General-Purpose Audio Tagging Challenge". The goal of the task is to build an audio tagging system that can recognize the c…
AV-CPL uses continuous pseudo-labels for AVSR combining labeled and unlabeled data.
problem Improving AVSR performance with labeled and unlabeled data.
method Semi-supervised method using continuous pseudo-labels generated by the same AVSR model.
result Significant improvements in VSR performance on LRS3 dataset.
MelNet generates high-fidelity audio with long-range structure.
problem Capturing long-range dependencies in audio waveforms.
method Generative model in frequency domain, leveraging spectrograms.
result Improves audio generation in various tasks.
HEAR benchmark evaluates audio representations for diverse tasks.
problem Developing a general-purpose audio representation for various tasks.
method Evaluated 29 models across 19 tasks using 16 datasets.
result No single audio representation performs holistically.