Proposes a self-supervised method for generating spatial audio from monaural audio and video.
problem Generating spatial audio from monaural audio and video recordings is challenging and expensive.
method Uses a self-supervised network with an auxiliary classifier to classify video channels and generate spatial audio.
result The proposed method effectively generates spatial audio from monaural audio and video.
This paper addresses the problem of localizing audio sources using binaural measurements. We propose a supervised formulation that simultaneously localizes multiple sources at different locations. The approach is intrinsically efficient because, contrary to prior work, it relies neither on source separation, nor on mon…
Recently, deep neural network (DNN) has made a breakthrough in monaural source enhancement. Through a training step by using a large amount of data, DNN estimates a mapping between mixed signals and clean signals. At this time, we use an objective function that numerically expresses the quality of a mapping by DNN. In …
We propose an algorithm to separate simultaneously speaking persons from each other, the "cocktail party problem", using a single microphone. Our approach involves a deep recurrent neural networks regression to a vector space that is descriptive of independent speakers. Such a vector space can embed empirically determi…
Deep audio prior uses neural networks to solve audio problems without data.
problem Challenging audio problems like source separation, editing, and synthesis.
method Randomly-initialized neural network with carefully designed audio prior.
result Superior audio results on Universal-150 benchmark dataset.
Zero-shot audio classification using class label embeddings.
problem Classifying audio without labeled data.
method Bilinear model with audio feature embeddings and class label embeddings.
result Achieved accuracy up to 39.7% for natural audio categories.
Recently, there has been great interest in the field of audio style transfer, where a stylized audio is generated by imposing the style of a reference audio on the content of a target audio. We improve on the current approaches which use neural networks to extract the content and the style of the audio signal and propo…
Method learns audio embeddings with contextualized tags.
problem Align audio and tags for cross-modal tasks.
method Audio autoencoder, word embeddings, multi-head self-attention, contrastive loss.
result Multi-head self-attention improves audio representations.
Study improves radio show segmentation using audio embeddings.
problem Automated segmentation of radio shows.
method Created audio embeddings from multi-class classification tasks on different datasets, evaluated performance against text-only baseline.
result Audio embeddings from non-speech sound event classification significantly outperformed text-only baseline by 32.3% in F1-measure.
Online audio advertising is a particular form of advertising used abundantly in online music streaming services. In these platforms, which tend to host tens of thousands of unique audio advertisements (ads), providing high quality ads ensures a better user experience and results in longer user engagement. Therefore, th…
Adversarial attacks on spectrograms can fool audio classifiers trained on waveforms.
problem Susceptibility of audio classifiers to adversarial attacks on spectrograms.
method Applying adversarial attacks to spectrograms and reconstructing audio waveforms.
result Perturbed spectrograms can fool 2D CNNs and 1D CNNs trained on audio waveforms.
Study improves animal audio classification using data augmentation.
problem Improving automated animal audio classification accuracy.
method Exploits different data augmentation techniques for training CNNs.
result Best recognition rates on animal audio classification datasets.
MicAugment transfers audio style from few seconds of input to match target conditions.
problem Audio model robustness to diverse acquisition conditions.
method Identifies and applies transformations learned from target audio to input audio.
result MicAugment significantly improves model robustness in downstream tasks.
Detects audio adversarial examples using anomalous pattern detection.
problem Identifies adversarial audio attacks in deep neural networks.
method Applies anomalous pattern detection in activation space of audio models.
result Can detect adversarial examples with up to 0.98 AUC, no degradation on benign samples.
Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…
Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…
Unified model for audio control and style transfer.
problem Explicit control and style transfer in music generation.
method Diffusion autoencoders for semantic feature extraction, disentanglement using adversarial criterion.
result Model generates audio matching timbre targets with specified structure.
Audio fingerprinting, also named as audio hashing, has been well-known as a powerful technique to perform audio identification and synchronization. It basically involves two major steps: fingerprint (voice pattern) design and matching search. While the first step concerns the derivation of a robust and compact audio si…
Transformer model estimates keywords for better audio captioning.
problem Indeterminacy in word selection for audio events/scenes.
method Transformer-based model with keyword estimation.
result Achieved state-of-the-art performance in AAC.
LumièreNet creates lecture videos from audio narration.
problem Creating high-quality lecture videos from audio narration.
method Modular deep-learning architecture that learns from audio to video.
result Synthesizes high-quality lecture videos from any length of audio.
Review of deep learning techniques for audio signal processing.
problem Improving audio signal processing using deep learning.
method Analysis of various deep learning models and techniques.
result Advancements in speech, music, and environmental sound processing.
This paper describes Task 2 of the DCASE 2018 Challenge, titled "General-purpose audio tagging of Freesound content with AudioSet labels". This task was hosted on the Kaggle platform as "Freesound General-Purpose Audio Tagging Challenge". The goal of the task is to build an audio tagging system that can recognize the c…
AV-CPL uses continuous pseudo-labels for AVSR combining labeled and unlabeled data.
problem Improving AVSR performance with labeled and unlabeled data.
method Semi-supervised method using continuous pseudo-labels generated by the same AVSR model.
result Significant improvements in VSR performance on LRS3 dataset.
MelNet generates high-fidelity audio with long-range structure.
problem Capturing long-range dependencies in audio waveforms.
method Generative model in frequency domain, leveraging spectrograms.
result Improves audio generation in various tasks.
Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditio…
HEAR benchmark evaluates audio representations for diverse tasks.
problem Developing a general-purpose audio representation for various tasks.
method Evaluated 29 models across 19 tasks using 16 datasets.
result No single audio representation performs holistically.
Proposes COALA method for learning audio representations aligned with tags.
problem Lack of annotated data for high-performance audio representation learning.
method Aligns latent representations of audio and tags using a contrastive loss.
result Audio embedding model captures both acoustic and semantic characteristics.
Paper introduces a noisy-labeled audio tagging challenge.
problem Acoustic mismatch and noisy labels in audio tagging.
method Large dataset with minimal supervision, convolutional neural network baseline.
result Demonstrates effectiveness of minimal supervision in noisy conditions.
A new method for audio denoising using deep neural networks.
problem Improving audio quality by removing background noise.
method Combines time and time-frequency domain processing; trains a deep neural network to fit the signal.
result The method effectively disentangles clean audio from noisy signals.
Mic2Mic reduces microphone variability for speech systems.
problem Microphone variability impacts speech system accuracy.
method Cycle-consistent GANs for learning microphone variability reduction.
result Mic2Mic recovers 66% to 89% of lost accuracy due to microphone variability.
As the technology is advancing, audio recognition in machine learning is improved as well. Research in audio recognition has traditionally focused on speech. Living creatures (especially the small ones) are part of the whole ecosystem, monitoring as well as maintaining them are important tasks. Species such as animals …
FSD50K provides an open dataset of over 51k audio clips for sound event recognition.
problem Small and domain-specific sound event recognition datasets.
method Creation of an open dataset with over 51k audio clips manually labeled using 200 classes.
result FSD50K is a new open benchmark for sound event recognition research.
Paper proposes a robust audio classification method against adversarial attacks.
problem Adversarial attacks can fool machine learning models into making incorrect predictions.
method Proposes a novel SVM-based approach using DWT and SURF features.
result The proposed method provides a good balance between accuracy and resilience against adversarial attacks.
We study the problem of semi-supervised singing voice separation, in which the training data contains a set of samples of mixed music (singing and instrumental) and an unmatched set of instrumental music. Our solution employs a single mapping function g, which, applied to a mixed sample, recovers the underlying instrum…
Nonnegative matrix factorization (NMF) is now a common tool for audio source separation. When learning NMF on large audio databases, one major drawback is that the complexity in time is O(FKN) when updating the dictionary (where (F;N) is the dimension of the input power spectrograms, and K the number of basis spectra),…
Self-supervised attention model improves weakly labeled audio event classification.
problem Efficiently classify audio events with minimal labeled data.
method Develops a self-supervised attention model for weakly labeled audio clips.
result Self-supervised attention model performs comparably to strongly supervised model trained with strong labels.
This research tackles backdoor attacks on audio data using a stochastic investment approach.
problem The threat of backdoor attacks on audio data, especially in voice-activated systems.
method A Stochastic investment-based backdoor attack (MarketBack) approach.
result MarketBack can achieve an average attack success rate close to 100% with less than 1% of poisoned data.
WaveGrad generates high-fidelity audio using gradient estimation.
problem Generating high-fidelity audio efficiently.
method Conditional model using score matching and diffusion models, iteratively refining a Gaussian white noise signal.
result WaveGrad can generate high-fidelity audio samples using as few as six iterations.
GACELA fills long gaps in musical audio with a GAN and context conditioning.
problem Restoring long gaps in musical audio with varying complexity and duration.
method Generative adversarial network (GAN) with five parallel discriminators and context conditioning.
result Reduced artifacts in inpaintings from unacceptable to mildly disturbing.
System tackles indeterminacies in automated audio captioning.
problem Word selection and sentence length indeterminacies in automated audio captioning.
method Solves caption generation and sub-indeterminacy problems through multi-task learning to estimate keywords and sentence length.
result Model achieved 20.7 SPIDEr score, significantly outperforming baseline.
A new framework enhances binaural audio for moving talkers.
problem Real-time binaural audio enhancement for moving sound sources.
method Mixture-of-experts framework combining multiple binaural filters.
result Dynamic spatial audio rendering adapts to moving talkers.
Proposes GAAE for high-fidelity audio generation and representation learning.
problem Lack of usable representations and high-fidelity audio generation from unsupervised learning.
method Guided Adversarial Autoencoder (GAAE) leveraging a small percentage of labelled data.
result Generates high-fidelity audio with superior quality and learns powerful representations.
New method allows real-time audio synthesis using non-causal convolutions.
problem Real-time audio synthesis limitations due to offline model constraints.
method Post-training reconfiguration of non-causal models for real-time buffer-based processing.
result Non-causal streaming models can be transformed from offline-trained models without quality loss.
MuLan links music audio to natural language tags.
problem Traditional music tagging systems use rigid attributes; MuLan aims to link audio directly to natural language.
method Joint audio-text embedding model trained on 44 million music recordings and text annotations.
result MuLan's embeddings enable zero-shot functionalities and transfer learning.
Efficient neural network for audio source separation.
problem End-to-end general purpose audio source separation.
method SuDoRMRF structure with simple one-dimensional convolutions for feature aggregation.
result SuDoRMRF achieves high quality audio source separation with minimal computational resources.
Existing automatic music generation approaches that feature deep learning can be broadly classified into two types: raw audio models and symbolic models. Symbolic models, which train and generate at the note level, are currently the more prevalent approach; these models can capture long-range dependencies of melodic st…
Universal audio perturbations fool ASR systems.
problem Speech recognition systems are vulnerable to adversarial attacks.
method Developed an algorithm to create universal perturbations that fool ASR models.
result Universal perturbations can fool ASR systems across different models.
Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative A…