Improved BER with reduced power in time-domain digital backpropagation.
problem Improving BER performance in time-domain digital backpropagation.
method Jointly optimized and quantized chromatic dispersion filters using machine learning.
result Improved BER performance and power dissipation reductions.
Improved signal processing for long-distance optical signals.
problem Compensating walk-off effect in long-distance optical signals.
method Sub-banded DSP architecture with deep learning for walk-off compensation.
result 2.8 dB SNR improvement over linear equalization.
A machine learning model for PMD compensation in dual-polarization systems.
problem Compensating for polarization-mode dispersion (PMD) in dual-polarization systems.
method Model-based machine learning approach using the split-step Fourier method for the Manakov-PMD equation.
result The model converges to within 1% of peak dB performance after 428 iterations, achieving a 0.30 dB reduction in effective signal-to-noise ratio compared to PMD-free case.
Optical co-processor speeds up neural network training.
problem Expensive training costs for large neural networks.
method Direct feedback alignment, optical error projection.
result Optical co-processor trains neural networks for handwritten digit recognition.
A neural-network-based approach is presented to efficiently implement digital backpropagation (DBP). For a 32x100 km fiber-optic link, the resulting "learned" DBP significantly reduces the complexity compared to conventional DBP implementations.
A method for learning sparse transformations through backpropagation.
problem Sparse transformations in deep learning architectures are hard to design and often represented densely during learning.
method Adaptive, sparse hyperlayer with randomly sampled connections to overcome gradient issues.
result Trained models achieve competitive performance on real data.
Paper proposes an efficient method to optimize neural networks without backpropagation.
problem Computational inefficiency and scalability issues in neural network optimization.
method Derives explicit solutions to optimize neural networks, reducing computational costs.
result Explicit solutions achieve near-optimality and can discover better optima than backpropagation.
Study introduces constrained neural units over time for learning problems.
problem Learning problems with Neural Networks using the principle of least cognitive action.
method Defines Neural Networks with special constraints on structure and data interaction.
result The approach can degenerate to Backpropagation under certain conditions.
Machine learning improves PMD compensation in multiplexed systems.
problem Improving performance in multiplexed systems with PMD.
method Model-based machine learning parameterizing the Manakov-PMD equation.
result Performance close to PMD-free case achieved with hardware-friendly DBP and PMD compensation.
In recent years, there have been numerous developments towards solving multimodal tasks, aiming to learn a stronger representation than through a single modality. Certain aspects of the data can be particularly useful in this case - for example, correlations in the space or time domain across modalities - but should be…
BinaryGAN uses binary neurons for GAN training.
problem Training GANs with binary outputs.
method End-to-end backpropagation with sigmoid-adjusted straight-through estimators.
result BinaryGAN can generate binary-valued predictions.
New loss function connects learning rate and momentum.
problem Finding optimal learning rate and momentum empirically.
method Proposes a new information-theoretical loss function.
result Loss, learning rate, and momentum are closely connected.
This study examines how hidden layers affect CNN performance on handwritten digit recognition.
problem Impact of hidden layers on CNN performance in handwritten digit recognition.
method Applied CNN with varying hidden layers on MNIST dataset, trained with stochastic gradient and backpropagation, tested with feedforward.
result Variations in accuracies for different hidden layers and epochs.
Efficient sparse GP model improves audio source separation.
problem Sparse Gaussian Process (GP) inference is computationally expensive for long audio frames.
method Used GP regression, spectral mixture kernels, and variational sparse GPs.
result Proposed method outperforms LD-PSDTF, KL-NMF, and IS-NMF.
Physics-based deep learning improves fiber-optic communication efficiency.
problem Improving signal propagation in fiber-optic communication systems.
method Parameterizing the split-step method of solving the nonlinear Schrödinger equation as a deep neural network.
result Filters can be pruned to as few as 3 taps/step without sacrificing performance.
Improves speech separation by integrating time and frequency domains.
problem Speech separation using deep learning techniques.
method Proposes a framework that combines time and frequency domain features, using an embedding network and clustering.
result Obtained state-of-the-art results on WSJ0-2mix dataset.
In a previous work we have detailed the requirements to obtain a maximal performance benefit by implementing fully connected deep neural networks (DNN) in form of arrays of resistive devices for deep learning. This concept of Resistive Processing Unit (RPU) devices we extend here towards convolutional neural networks (…
Wave-U-Net with MHE regularization improves singing voice separation.
problem Singing voice separation from mixed music recordings.
method Wave-U-Net architecture with MHE regularization applied to 1D filters.
result Adding MHE regularization to the loss function consistently improves singing voice separation.
Paper uses ANFIS to predict cryptocurrency prices.
problem Predicting cryptocurrency prices for seven days.
method Adaptive Network Based Fuzzy Inference System (ANFIS) with hybrid and backpropagation algorithms.
result The method can predict cryptocurrency prices in a short time.
TTW aligns time-series faster and more accurately than existing methods.
problem Efficiently aligning multiple time-series signals with varying lengths.
method TTW uses a sinc convolutional kernel and gradient-based optimization for linear time and sequence complexity.
result TTW outperforms existing methods in time-series averaging and classification tasks.
Shallow networks with local learning rules can match deep learning performance.
problem Training deep neural networks is biologically implausible; the goal is to achieve similar performance with shallow networks.
method Investigated shallow networks with one hidden layer and a single readout layer, using various local learning rules for the hidden layer and supervised learning for the readout layer.
result Shallow networks can achieve test accuracy comparable to deep learning models, suggesting the use of different datasets for testing.
Belief propagation recovers backpropagation results.
problem Connection between backpropagation and belief propagation poorly understood.
method Converted backpropagation input to belief propagation input and showed results.
result Backpropagation is a special case of belief propagation.
A two-layer classifier improves smartphone transportation mode recognition.
problem Improving accuracy of transportation mode classification.
method Two-layer hierarchical classifier combining time and frequency domain features.
result Maximum classification accuracy of 97.02%.
Paper proposes DAG-DB for learning discrete DAGs via backpropagation.
problem Learning Directed Acyclic Graphs (DAGs) from data.
method DAG-DB uses Discrete Backpropagation with I-MLE and Straight-Through Estimation.
result DAG-DB learns DAGs effectively using probabilistic sampling and backpropagation.
Synthesizes images from audio and visual data using spike-based autoencoders.
problem Extracting meaningful information from spatio-temporal data for image synthesis.
method Spike-based autoencoders trained to learn spatio-temporal representations of audio and visual data.
result Synthesized images from audio samples with high fidelity, achieving competitive performance.
Soft-constrained PINN solves ODEs with minimal data, improving efficiency and robustness.
problem Sparse and noisy data in experiments and simulations.
method Soft-constrained Physics-informed Neural Network (PINN) with minimal labeled data.
result Soft-constrained PINN reduces need for labeled data and achieves strong generalization.
Analog method solves portfolio optimization problems faster and more efficiently.
problem Accurate covariance matrix estimation and fast optimal portfolio selection for financial applications.
method Two-step process using equilibrium propagation and analog Hopfield networks.
result Fully analog pipeline calculates optimal portfolios in energy-efficient manner.
GAIT-prop derives a biologically plausible learning rule from backpropagation.
problem Biological implausibility in traditional backpropagation for neural networks.
method GAIT-prop uses a top-down model to convert output error into plausible targets for weight updates.
result GAIT-prop and backpropagation give identical weight updates under certain conditions.
This work shows synthetic gradients can outperform backpropagation in sample efficiency.
problem The efficiency of backpropagation in training neural networks.
method Unified vectorized feedback framework for loss-based and reward-based learning, introducing synthetic gradients.
result Synthetic gradients can achieve lower gradient-estimation mean squared error than backpropagation under certain conditions.
A new method computes gradients without backpropagation.
problem Optimization of machine learning models.
method Forward mode automatic differentiation to compute gradients.
result Forward gradient is an unbiased estimate of the gradient, eliminating the need for backpropagation.
Unified framework for faster neural network training with less information loss.
problem Time-consuming backpropagation and loss of unpropagated gradient information.
method Unified sparse backpropagation framework and memorized sparse backpropagation algorithm.
result Convergence in probability with certain conditions and effective information loss mitigation.
Novel unsupervised audio source separation using generative priors.
problem Challenges of supervised methods in audio source separation.
method Generative priors trained on individual sources using projected gradient descent optimization.
result Effective recovery of constituent sources with good-quality estimates.
Backprop-Q extends standard backpropagation for stochastic computation graphs.
problem Applying standard backpropagation to stochastic computation graphs is challenging.
method Construct Q-functions for each stochastic node and use them to train the SCG with standard backpropagation.
result Generalized backpropagation for stochastic computation graphs is feasible and extends learning signals beyond gradients.
Wave-U-Net improves audio source separation by modeling phase information.
problem Fixed spectral transformations and high sampling rates limit audio source separation performance.
method Wave-U-Net adapts U-Net to time-domain, using repeated resampling to capture different time scales.
result Wave-U-Net achieves comparable performance to spectrogram-based U-Net on singing voice separation.
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
Skip connections improve biologically-inspired learning rules.
problem Biologically-inspired learning rules often underperform compared to backpropagation.
method Introduced skip connections between intermediate layers in biologically-motivated learning rules.
result Skip connections can match the performance of backpropagation and are robust to hyper-parameters.
Pipelined Backpropagation trains large models without batches efficiently.
problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.
A new method suppresses echo more effectively with lower latency.
problem Echo cannot be fully removed by linear filters due to nonlinear relationships.
method Uses a modified Conv-TasNet with multiple streams of input signals.
result Efficacy validated in both single-talk and double-talk situations.
Spectral backpropagation optimizes implicit likelihoods without explicit assumptions.
problem Training systems without explicit goals and likelihood functions.
method Spectral backpropagation for implicit likelihood optimization.
result Identified two novel properties of GAN generators: aberrant outputs and quasi-disentangled factors.
ZORB speeds up neural network training without sacrificing accuracy.
problem Slow and strenuous training of neural networks using gradient descent.
method Uses pseudoinverse of targets instead of gradients for backpropagation.
result ZORB converges 300 times faster than Adam on MNIST without hyperparameter tuning.
Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully l…
HSIC bottleneck trains deep networks without backpropagation.
problem Training deep neural networks with exploding and vanishing gradients.
method HSIC bottleneck, alternative to cross-entropy loss and backpropagation.
result HSIC bottleneck achieves comparable performance to backpropagation.
Analog deep learning shows promise but faces scalability challenges.
problem Bottlenecks in deep neural networks' calculation and optimization.
method Evaluation of eight analog deep learning methodologies.
result Analog deep learning has potential for consumer-level applications but scalability remains an issue.
In this paper, we address the problem of reconstructing a time-domain signal (or a phase spectrogram) solely from a magnitude spectrogram. Since magnitude spectrograms do not contain phase information, we must restore or infer phase information to reconstruct a time-domain signal. One widely used approach for dealing w…
Mean field theory explains gradient backpropagation in deep dropout networks.
problem Understanding gradient backpropagation in deep dropout networks.
method Applied mean field theory to dropout networks, considering realistic training conditions.
result Gradient backpropagation length is limited by depth scales, not just independence assumption.
Generalizes double backpropagation for neural networks in Hilbert space.
problem Lack of a general description of derivatives in neural network loss functions.
method Develops a Hilbert space framework to cover various special cases of double backpropagation.
result Optimized backpropagation rules for real-world scenarios, reducing calculations by up to 33%.
Backpropagation-free RL method trains layers using local signals.
problem Vanishing or exploding gradients in backpropagation-based RL.
method Local pairwise distance matching for layer-wise training without backpropagation.
result Backpropagation-free method achieves competitive performance and stability.
MelNet generates high-fidelity audio with long-range structure.
problem Capturing long-range dependencies in audio waveforms.
method Generative model in frequency domain, leveraging spectrograms.
result Improves audio generation in various tasks.