Generative model improves audio synthesis from TF features.
problem Challenges in generating high-quality audio from TF features.
method Used a GAN to generate invertible TF features from short-time Fourier transforms.
result Generated TF features from GAN outperformed direct waveform generation.
Deep audio prior uses neural networks to solve audio problems without data.
problem Challenging audio problems like source separation, editing, and synthesis.
method Randomly-initialized neural network with carefully designed audio prior.
result Superior audio results on Universal-150 benchmark dataset.
Model for directed synthesis of audio textures using multi-scale RNNs.
problem Challenges in modeling complex audio textures with traditional methods.
method Combining multi-scale RNNs with a conditioning strategy for user-directed synthesis.
result Demonstrated improved performance on various audio texture datasets.
The recent success of raw audio waveform synthesis models like WaveNet motivates a new approach for music synthesis, in which the entire process --- creating audio samples from a score and instrument information --- is modeled using generative neural networks. This paper describes a neural music synthesis model with fl…
New method allows real-time audio synthesis using non-causal convolutions.
problem Real-time audio synthesis limitations due to offline model constraints.
method Post-training reconfiguration of non-causal models for real-time buffer-based processing.
result Non-causal streaming models can be transformed from offline-trained models without quality loss.
LumièreNet creates lecture videos from audio narration.
problem Creating high-quality lecture videos from audio narration.
method Modular deep-learning architecture that learns from audio to video.
result Synthesizes high-quality lecture videos from any length of audio.
GANSynth uses GANs to efficiently synthesize high-fidelity audio.
problem Efficient and high-fidelity audio synthesis is challenging.
method Model log magnitudes and instantaneous frequencies with GANs.
result GANSynth outperforms WaveNet on automated and human evaluation metrics.
DiffWave generates high-fidelity audio waveforms efficiently.
problem Conditional and unconditional audio waveform generation.
method Non-autoregressive diffusion model using Markov chain synthesis.
result DiffWave produces high-quality audios in various tasks.
RawNet synthesizes speech without relying on human-designed features.
problem Speech synthesis models rely heavily on human-designed spectral features.
method End-to-end neural vocoder using auto-encoder structure, learning features and audio recovery directly from raw waveform.
result RawNet achieves better speech quality with a simplified model architecture and faster inference speed.
Review of deep learning techniques for audio signal processing.
problem Improving audio signal processing using deep learning.
method Analysis of various deep learning models and techniques.
result Advancements in speech, music, and environmental sound processing.
Synthesizes images from audio and visual data using spike-based autoencoders.
problem Extracting meaningful information from spatio-temporal data for image synthesis.
method Spike-based autoencoders trained to learn spatio-temporal representations of audio and visual data.
result Synthesized images from audio samples with high fidelity, achieving competitive performance.
Improves text-to-speech speed by interleaving character reading and audio synthesis.
problem Latency in text-to-speech models limits their use in time-sensitive tasks.
method Reinforcement learning to train an agent to choose the order of character reading and audio synthesis.
result The proposed method successfully balances latency and audio quality.
This research improves neural synthesizers for music sounds from speech data.
problem Applying speech synthesis techniques to musical instrument sounds.
method Comparison of three neural synthesizers in three scenarios: training, zero-shot learning, and fine-tuning.
result Neural synthesizers trained on speech data and fine-tuned on music data perform better.
Improved visual speech synthesis using adapted ASR acoustic models.
problem Lack of synchronized audio, video, and depth data for speaker-independent speech-driven visual speech synthesis.
method Adapted an ASR acoustic model trained on audio-only data to the visual speech synthesis domain.
result Viewers significantly prefer animations generated from the adapted ASR acoustic model.
This paper proposes a new architecture for speaker adaptation of multi-speaker neural-network speech synthesis systems, in which an unseen speaker's voice can be built using a relatively small amount of speech data without transcriptions. This is sometimes called "unsupervised speaker adaptation". More specifically, we…
Deep learning models generate music with arbitrary control strategies.
problem Lack of efficient methods for generating music with arbitrary control.
method Deep generative models learn to navigate arbitrary sound spaces.
result Deep learning enables high-quality, arbitrary sound synthesis.
New method improves speech synthesis quality.
problem Efficiently train parallel speech synthesis models.
method Spectral energy distance for implicit generative models.
result State-of-the-art generation quality achieved.
Improved human-like speech synthesis using WaveGAN extensions.
problem Generating realistic speech from limited data.
method Extensions to WaveGAN for audio generation, including preprocessing, skip connections, and progressive structures.
result Moderate improvement in human likeness (Cohen's d coefficient of 0.65) in synthetic speech samples.
DDSP integrates signal processing with deep learning for high-fidelity audio synthesis.
problem Efficiently combining signal processing knowledge with deep learning for audio synthesis.
method Integrates classic signal processing elements with deep learning methods.
result High-fidelity audio synthesis without large models or adversarial losses.
In this paper we propose WaveGlow: a flow-based network capable of generating high quality speech from mel-spectrograms. WaveGlow combines insights from Glow and WaveNet in order to provide fast, efficient and high-quality audio synthesis, without the need for auto-regression. WaveGlow is implemented using only a singl…
Paper uses SSL models' uncertainty to predict audio quality efficiently.
problem Efficiently predicting audio quality in low-resource settings.
method Leverages self-supervised learning models' uncertainty measures.
result Uncertainty measures correlate with MOS scores in SSL models.
Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders or MIDI synthesizers for speech or music generation. Despite their successes, current state-of-the-art neural audio synthesizers such as Wav…
RUSLAN is a large Russian speech corpus for text-to-speech.
problem Lack of high-quality annotated Russian speech data for text-to-speech.
method Developed a large annotated Russian speech corpus and trained a neural network for text-to-speech synthesis.
result Synthesized speech quality evaluated with MOS scores: 4.05 for naturalness, 3.78 for intelligibility.
iSTFTNet2 improves iSTFTNet's speed and lightness with 1D-2D CNN.
problem Efficiently synthesizing high-fidelity speech.
method Improved iSTFTNet using 1D-2D CNNs for temporal and spectrogram structures.
result iSTFTNet2 is faster and more lightweight with comparable speech quality.
MelNet generates high-fidelity audio with long-range structure.
problem Capturing long-range dependencies in audio waveforms.
method Generative model in frequency domain, leveraging spectrograms.
result Improves audio generation in various tasks.
Sound synthesis is a complex field that requires domain expertise. Manual tuning of synthesizer parameters to match a specific sound can be an exhaustive task, even for experienced sound engineers. In this paper, we introduce InverSynth - an automatic method for synthesizer parameters tuning to match a given input soun…
Speech Translation has always been about giving source text or audio input and waiting for system to give translated output in desired form. In this paper, we present the Acoustic Dialect Decoder (ADD) - a voice to voice ear-piece translation device. We introduce and survey the recent advances made in the field of Spee…
WaveGrad generates high-fidelity audio using gradient estimation.
problem Generating high-fidelity audio efficiently.
method Conditional model using score matching and diffusion models, iteratively refining a Gaussian white noise signal.
result WaveGrad can generate high-fidelity audio samples using as few as six iterations.
The state-of-the-art in text-to-speech synthesis has recently improved considerably due to novel neural waveform generation methods, such as WaveNet. However, these methods suffer from their slow sequential inference process, while their parallel versions are difficult to train and even more expensive computationally. …
SpecGrad improves neural vocoder sound quality by adapting diffusion noise to log-mel spectrogram.
problem Improving neural vocoder sound quality, especially in high-frequency bands.
method Adapting the diffusion noise distribution to the conditioning log-mel spectrogram through time-varying filtering.
result SpecGrad generates higher-fidelity speech waveform than conventional DDPM-based neural vocoders.
Lipper synthesizes speech from silent videos, improving over single-view methods.
problem Lipreading as text classification is limited; multi-view approach needed.
method Multi-view lipreading as a regression task, producing speech from silent videos.
result Improvement in speech reconstruction with multi-view silent videos.
NanoFlow reduces parameter complexity in normalizing flows.
problem Efficient parameter complexity in flow-based models.
method Single neural density estimator with flow indication embedding.
result Sublinear parameter complexity achieved.
Global Style Tokens (GSTs) are a recently-proposed method to learn latent disentangled representations of high-dimensional data. GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style. In this work, we introduce the T…
Generative compression technique reduces neural network size and improves performance on microcontrollers.
problem Memory constraints on microcontrollers limit the size of neural networks, especially for 1x1 pointwise (PW) mixers.
method HYPER-TINYPW uses a shared micro-MLP to generate PW kernels from tiny per-layer codes, reducing memory usage.
result HYPER-TINYPW achieves comparable performance to larger models while being significantly smaller (225 kB vs 1.4 MB).
End-to-end speech synthesis is a promising approach that directly converts raw text to speech. Although it was shown that Tacotron2 outperforms classical pipeline systems with regards to naturalness in English, its applicability to other languages is still unknown. Japanese could be one of the most difficult languages …
iSTFTNet speeds up mel-spectrogram vocoders without sacrificing quality.
problem Efficiently converting mel-spectrograms to speech with minimal computation.
method Replaces convolutional layers with iSTFT after frequency dimension reduction.
result Significant reduction in computational cost with comparable quality.
A new method controls synthesizers with normalizing flows for easier exploration and creation.
problem Complexity and parameters in modern synthesizers make them hard to master.
method Formalizing synthesizer control as latent audio space mapping with VAE and NF, disentangling flows.
result Superior performance in parameter inference and audio reconstruction, disentangling major audio variations.
SED integrates synthesis, execution, and debugging for neural program synthesis.
problem Challenges in synthesizing complex programs that match specifications.
method SED combines synthesis, execution, and debugging to improve neural program generation.
result SED reduces error rates and outperforms standard decoding methods.
Detects audio adversarial examples using anomalous pattern detection.
problem Identifies adversarial audio attacks in deep neural networks.
method Applies anomalous pattern detection in activation space of audio models.
result Can detect adversarial examples with up to 0.98 AUC, no degradation on benign samples.
Efficient neural network for audio source separation.
problem End-to-end general purpose audio source separation.
method SuDoRMRF structure with simple one-dimensional convolutions for feature aggregation.
result SuDoRMRF achieves high quality audio source separation with minimal computational resources.
Neural network generates music scores directly from polyphonic audio.
problem Transcribing music scores directly from polyphonic audio.
method Convolutional Recurrent Neural Network (CRNN) with CTC loss function.
result Model can learn to transcribe scores directly from audio signals.
Recently, there has been great interest in the field of audio style transfer, where a stylized audio is generated by imposing the style of a reference audio on the content of a target audio. We improve on the current approaches which use neural networks to extract the content and the style of the audio signal and propo…
Study improves animal audio classification using data augmentation.
problem Improving automated animal audio classification accuracy.
method Exploits different data augmentation techniques for training CNNs.
result Best recognition rates on animal audio classification datasets.
New neural network improves audio classification accuracy.
problem Challenging audio classification problem in pattern recognition.
method Introduces a Classifier-Attention-Based Convolutional Neural Network (CAB-CNN) with an attention mechanism to reduce classifier complexity.
result Significantly improves audio classification performance, achieving more than 10% improvements.
BOFFIN TTS optimizes hyper-parameters for new speaker adaptation.
problem Fine-tuning a pre-trained TTS model for a new speaker with limited data.
method Bayesian optimization to efficiently find optimal hyper-parameters.
result Average 30% improvement in speaker similarity over standard techniques.
A new method for audio denoising using deep neural networks.
problem Improving audio quality by removing background noise.
method Combines time and time-frequency domain processing; trains a deep neural network to fit the signal.
result The method effectively disentangles clean audio from noisy signals.
Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…
Speech enhancement (SE) aims to reduce noise in speech signals. Most SE techniques focus only on addressing audio information. In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual de…