Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditio…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “audio synthesis”
Model for directed synthesis of audio textures using multi-scale RNNs.
problem Challenges in modeling complex audio textures with traditional methods.
method Combining multi-scale RNNs with a conditioning strategy for user-directed synthesis.
result Demonstrated improved performance on various audio texture datasets.
New method allows real-time audio synthesis using non-causal convolutions.
problem Real-time audio synthesis limitations due to offline model constraints.
method Post-training reconfiguration of non-causal models for real-time buffer-based processing.
result Non-causal streaming models can be transformed from offline-trained models without quality loss.
Deep Audio Priorcs.SD
Deep audio prior uses neural networks to solve audio problems without data.
problem Challenging audio problems like source separation, editing, and synthesis.
method Randomly-initialized neural network with carefully designed audio prior.
result Superior audio results on Universal-150 benchmark dataset.
DiffWave generates high-fidelity audio waveforms efficiently.
problem Conditional and unconditional audio waveform generation.
method Non-autoregressive diffusion model using Markov chain synthesis.
result DiffWave produces high-quality audios in various tasks.
Deep learning models generate music with arbitrary control strategies.
problem Lack of efficient methods for generating music with arbitrary control.
method Deep generative models learn to navigate arbitrary sound spaces.
result Deep learning enables high-quality, arbitrary sound synthesis.
The recent success of raw audio waveform synthesis models like WaveNet motivates a new approach for music synthesis, in which the entire process --- creating audio samples from a score and instrument information --- is modeled using generative neural networks. This paper describes a neural music synthesis model with fl…
Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative A…
Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Modelseess.AS
Speech-driven visual speech synthesis involves mapping features extracted from acoustic speech to the corresponding lip animation controls for a face model. This mapping can take many forms, but a powerful approach is to use deep neural networks (DNNs). However, a limitation is the lack of synchronized audio, video, an…
Given the recent surge in developments of deep learning, this article provides a review of the state-of-the-art deep learning techniques for audio signal processing. Speech, music, and environmental sound processing are considered side-by-side, in order to point out similarities and differences between the domains, hig…
In this paper we propose WaveGlow: a flow-based network capable of generating high quality speech from mel-spectrograms. WaveGlow combines insights from Glow and WaveNet in order to provide fast, efficient and high-quality audio synthesis, without the need for auto-regression. WaveGlow is implemented using only a singl…
Paper uses SSL models' uncertainty to predict audio quality efficiently.
problem Efficiently predicting audio quality in low-resource settings.
method Leverages self-supervised learning models' uncertainty measures.
result Uncertainty measures correlate with MOS scores in SSL models.
Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learningeess.AS
Improves text-to-speech speed by interleaving character reading and audio synthesis.
problem Latency in text-to-speech models limits their use in time-sensitive tasks.
method Reinforcement learning to train an agent to choose the order of character reading and audio synthesis.
result The proposed method successfully balances latency and audio quality.
We present LumièreNet, a simple, modular, and completely deep-learning based architecture that synthesizes, high quality, full-pose headshot lecture videos from instructor's new audio narration of any length. Unlike prior works, LumièreNet is entirely composed of trainable neural network modules to learn mapping functi…
This paper proposes a new architecture for speaker adaptation of multi-speaker neural-network speech synthesis systems, in which an unseen speaker's voice can be built using a relatively small amount of speech data without transcriptions. This is sometimes called "unsupervised speaker adaptation". More specifically, we…
Neural network-based vocoders have recently demonstrated the powerful ability to synthesize high-quality speech. These models usually generate samples by conditioning on spectral features, such as Mel-spectrogram and fundamental frequency, which is crucial to speech synthesis. However, the feature extraction procession…
New method improves speech synthesis quality.
problem Efficiently train parallel speech synthesis models.
method Spectral energy distance for implicit generative models.
result State-of-the-art generation quality achieved.
DDSP integrates signal processing with deep learning for high-fidelity audio synthesis.
problem Efficiently combining signal processing knowledge with deep learning for audio synthesis.
method Integrates classic signal processing elements with deep learning methods.
result High-fidelity audio synthesis without large models or adversarial losses.
Spiking neural networks (SNNs) offer a promising alternative to current artificial neural networks to enable low-power event-driven neuromorphic hardware. Spike-based neuromorphic applications require processing and extracting meaningful information from spatio-temporal data, represented as series of spike trains over …
Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations suc…
Speech Translation has always been about giving source text or audio input and waiting for system to give translated output in desired form. In this paper, we present the Acoustic Dialect Decoder (ADD) - a voice to voice ear-piece translation device. We introduce and survey the recent advances made in the field of Spee…
Recent neural waveform synthesizers such as WaveNet, WaveGlow, and the neural-source-filter (NSF) model have shown good performance in speech synthesis despite their different methods of waveform generation. The similarity between speech and music audio synthesis techniques suggests interesting avenues to explore in te…
WaveGrad generates high-fidelity audio using gradient estimation.
problem Generating high-fidelity audio efficiently.
method Conditional model using score matching and diffusion models, iteratively refining a Gaussian white noise signal.
result WaveGrad can generate high-fidelity audio samples using as few as six iterations.
Generative Adversarial Networks (GANs) have become exceedingly popular in a wide range of data-driven research fields, due in part to their success in image generation. Their ability to generate new samples, often from only a small amount of input data, makes them an exciting research tool in areas with limited data re…
Global Style Tokens (GSTs) are a recently-proposed method to learn latent disentangled representations of high-dimensional data. GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style. In this work, we introduce the T…
Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders or MIDI synthesizers for speech or music generation. Despite their successes, current state-of-the-art neural audio synthesizers such as Wav…
End-to-end speech synthesis is a promising approach that directly converts raw text to speech. Although it was shown that Tacotron2 outperforms classical pipeline systems with regards to naturalness in English, its applicability to other languages is still unknown. Japanese could be one of the most difficult languages …
Sound synthesis is a complex field that requires domain expertise. Manual tuning of synthesizer parameters to match a specific sound can be an exhaustive task, even for experienced sound engineers. In this paper, we introduce InverSynth - an automatic method for synthesizer parameters tuning to match a given input soun…
The state-of-the-art in text-to-speech synthesis has recently improved considerably due to novel neural waveform generation methods, such as WaveNet. However, these methods suffer from their slow sequential inference process, while their parallel versions are difficult to train and even more expensive computationally. …
iSTFTNet2 improves iSTFTNet's speed and lightness with 1D-2D CNN.
problem Efficiently synthesizing high-fidelity speech.
method Improved iSTFTNet using 1D-2D CNNs for temporal and spectrogram structures.
result iSTFTNet2 is faster and more lightweight with comparable speech quality.
We present RUSLAN -- a new open Russian spoken language corpus for the text-to-speech task. RUSLAN contains 22200 audio samples with text annotations -- more than 31 hours of high-quality speech of one person -- being the largest annotated Russian corpus in terms of speech duration for a single speaker. We trained an e…
Paper explores how generative models can be made more creative.
problem Limitation of generative models in diverging from original data distribution.
method Proposes a novel training objective called Bounded Adversarial Divergence (BAD) to enable creative divergence.
result Preliminary results suggest BAD can enable creative divergence in generative models.
iSTFTNet speeds up mel-spectrogram vocoders without sacrificing quality.
problem Efficiently converting mel-spectrograms to speech with minimal computation.
method Replaces convolutional layers with iSTFT after frequency dimension reduction.
result Significant reduction in computational cost with comparable quality.
Non-negative blind source separation (non-negative BSS), which is also referred to as non-negative matrix factorization (NMF), is a very active field in domains as different as astrophysics, audio processing or biomedical signal processing. In this context, the efficient retrieval of the sources requires the use of sig…
Generative compression technique reduces neural network size and improves performance on microcontrollers.
problem Memory constraints on microcontrollers limit the size of neural networks, especially for 1x1 pointwise (PW) mixers.
method HYPER-TINYPW uses a shared micro-MLP to generate PW kernels from tiny per-layer codes, reducing memory usage.
result HYPER-TINYPW achieves comparable performance to larger models while being significantly smaller (225 kB vs 1.4 MB).
Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as video and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often …
This paper proposes a zero-shot learning approach for audio classification based on the textual information about class labels without any audio samples from target classes. We propose an audio classification system built on the bilinear model, which takes audio feature embeddings and semantic class label embeddings as…
Detects AI-synthesized speech using cepstral and bispectral analysis.
problem Validating the authenticity of speech from AI-generated content.
method Integrates cepstral and bispectral analysis for distinguishing human from AI-synthesized speech.
result Higher-order statistics show less correlation for human speech compared to AI-synthesis, and cepstral analysis reveals unique power components.
NanoFlow reduces parameter complexity in normalizing flows.
problem Efficient parameter complexity in flow-based models.
method Single neural density estimator with flow indication embedding.
result Sublinear parameter complexity achieved.
Recently, there has been great interest in the field of audio style transfer, where a stylized audio is generated by imposing the style of a reference audio on the content of a target audio. We improve on the current approaches which use neural networks to extract the content and the style of the audio signal and propo…
Spatial audio is an essential medium to audiences for 3D visual and auditory experience. However, the recording devices and techniques are expensive or inaccessible to the general public. In this work, we propose a self-supervised audio spatialization network that can generate spatial audio given the corresponding vide…
Method learns audio embeddings with contextualized tags.
problem Align audio and tags for cross-modal tasks.
method Audio autoencoder, word embeddings, multi-head self-attention, contrastive loss.
result Multi-head self-attention improves audio representations.
Study improves radio show segmentation using audio embeddings.
problem Automated segmentation of radio shows.
method Created audio embeddings from multi-class classification tasks on different datasets, evaluated performance against text-only baseline.
result Audio embeddings from non-speech sound event classification significantly outperformed text-only baseline by 32.3% in F1-measure.
Online audio advertising is a particular form of advertising used abundantly in online music streaming services. In these platforms, which tend to host tens of thousands of unique audio advertisements (ads), providing high quality ads ensures a better user experience and results in longer user engagement. Therefore, th…
We study approximations of non-Gaussian stationary processes having long range correlations with microcanonical models. These models are conditioned by the empirical value of an energy vector, evaluated on a single realization. Asymptotic properties of maximum entropy microcanonical and macrocanonical processes and the…
SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shapingeess.AS
SpecGrad improves neural vocoder sound quality by adapting diffusion noise to log-mel spectrogram.
problem Improving neural vocoder sound quality, especially in high-frequency bands.
method Adapting the diffusion noise distribution to the conditioning log-mel spectrogram through time-varying filtering.
result SpecGrad generates higher-fidelity speech waveform than conventional DDPM-based neural vocoders.
Generative models produce realistic objects in many domains, including text, image, video, and audio synthesis. Most popular models---Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs)---usually employ a standard Gaussian distribution as a prior. Previous works show that the richer family of pri…
Paper presents a novel waveform-to-waveform model for music source separation.
problem Isolating individual instruments from mixed music recordings.
method Adapted Conv-Tasnet to waveform domain and developed Demucs with U-Net and bidirectional LSTM.
result Demucs achieves superior performance on music source separation tasks, surpassing existing state-of-the-art.