Neural nets trained with linear discriminant initialization converge faster and more accurately.
problem Training feed-forward neural networks efficiently and accurately.
method Initialize first layer weights with linear discriminants.
result Asymptotic higher accuracy and faster convergence.
This work optimizes reservoir computing models by linking recurrence and non-linear dynamics.
problem Understanding how recurrence and non-linear dynamics in cortical networks contribute to their function.
method Transformed time-continuous, recurrent dynamics into an effective feed-forward structure of linear and non-linear temporal kernels.
result Optimal time-series classifiers can be built from random reservoir networks, demonstrating significant performance gains.
Proposes a feed-forward method for uncertainty propagation in neural networks.
problem Uncertainty in neural network inputs and activations.
method Feed-forward inference method for belief and neural networks, considering both mean and variance.
result Propagates uncertainty through neural network layers and parameters.
Feed-forward nets fail to learn equality relations, but adding DR units helps.
problem Feed-forward neural networks struggle to learn equality relations reliably.
method Introduced differential rectifier (DR) units to create an inductive bias.
result DR units enable feed-forward nets to learn equality relations reliably.
Proposes RMN for learning long-term dependencies in feed-forward networks.
problem Complicated training of deep RNN architectures.
method Residual Memory Neural Network (RMN) with residual and time-delayed connections.
result RMN and BRMN outperform LSTM and BLSTM networks in learning long-term and hierarchical information.
ForecastNet uses a time-variant deep feed-forward neural network for better multi-step-ahead time series forecasting.
problem Time-invariant architectures limit multi-step-ahead forecasting.
method ForecastNet employs a deep feed-forward architecture with time-variant parameters and interleaved outputs.
result ForecastNet outperforms other models on multi-step-ahead time series forecasting tasks.
Stable recurrent models perform similarly to unstable ones, proving useful for sequence tasks.
problem The lack of stability in recurrent neural networks hinders their practical application.
method Theoretical analysis and empirical testing of stable recurrent models.
result Stable recurrent models can be well approximated by feed-forward networks, performing similarly to unstable ones.
Paper characterizes and constructs universal approximators for neural networks.
problem Limited understanding of universal approximation in neural networks.
method Characterization, representation, construction method, existence result for any universal approximator.
result Improved capabilities of feed-forward architecture to approximate continuous functions.
Adapting robust statistics to neural networks, researchers found neural networks can be more robust with certain loss functions.
problem The robustness of neural networks in complex learning tasks.
method Adapting the regression breakdown point from robust statistics to neural networks and comparing different configurations and contamination settings.
result Neural networks can benefit from robust loss functions, as demonstrated in extensive simulations.
Efficient neural network optimization reduces costs and improves model performance.
problem High computational costs in optimizing neural networks, especially at scale.
method Introduces self-attentive feed-forward neural units (SAFFU) for efficient optimization.
result Explicit solutions outperform models optimized by backpropagation alone, and further training with backpropagation leads to better optima from smaller data sets.
SSFN self-estimates network size with low complexity and consistent performance.
problem Designing a self-estimating feed-forward network with low complexity and consistent performance.
method Joint optimization for layer and node estimation, low computational complexity, and use of lossless flow property and convex optimization.
result Consistent performance across Monte-Carlo trials and monotonically non-increasing cost with network growth.
New model adds persistent memory to self-attention layers for improved performance.
problem Improving transformer performance by removing feed-forward layers.
method Augmenting self-attention layers with persistent memory vectors.
result The model outperforms standard transformers on language modeling benchmarks.
Paper uses neural networks to analyze oil price impact on Iranian stock and industry indices.
problem Impact of oil price volatility on Tehran stock and industry indices.
method Feed-forward neural networks analysis of two periods: sanctions and post-sanctions.
result Neural networks predict stock and industry indices well, showing significant oil price volatility impact.
New connection between FFNs and BNs improves performance and generalization.
problem Improving performance and generalization in classification and segmentation tasks.
method Characterizing FFNs as approximations of BNs and developing a new learning algorithm.
result Statistically learned BNs outperform FFNs in classification and segmentation tasks.
We investigate the capacity, convexity and characterization of a general family of norm-constrained feed-forward networks.
New study shows neural networks need many samples for training.
problem How much data is needed to train a ReLU feed-forward neural network?
method Theoretical and empirical analysis of ReLU feed-forward neural networks.
result Generalization error scales at 1 / n 1/\sqrt{n} 1/ n in sample size n n n . New architecture avoids forgetting old tasks while learning new ones.
problem Catastrophic forgetting in neural networks when learning multiple tasks.
method Sequential learning paradigm with reusing features from previous tasks.
result Reduces forgetting and requires fewer resources for new task learning.
New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
This paper optimizes autonomous vehicle controllers using data-driven methods.
problem Designing robust controllers for autonomous vehicles that handle external and internal disturbances.
method Data-driven approach using principal component analysis and time delay neural networks.
result Improved controller performance through a feed-forward compensator.
Neural networks with bounded weights converge to unique fixed points.
problem Convergence and fixed points of neural networks with bounded weights.
method Analysis of iterative neural networks in Hilbert spaces with a single mild condition on weights.
result Guaranteed convergence to a unique fixed point with a bound on its norm.
General lower bounds on neural network approximation in L^p norm.
problem Fundamental limits of neural network expressivity.
method General lower bound proof on approximation in L^p norm, applied to feed-forward neural networks.
result Neural networks can't approximate certain functions as well as previously thought.
Deep network learns Obstacle Tower challenge without human demonstrations.
problem Master procedurally generated levels that get progressively harder.
method Deep Reinforcement Learning with a simple feed-forward network.
result Performed competitively in a reinforcement learning competition.
The study compares feed-forward and attention layers in language models.
problem Understanding the role of feed-forward and attention layers in language models.
method Empirical and theoretical analysis in a synthetic setting.
result Feed-forward layers learn simple distributional associations, while attention layers focus on in-context reasoning.
The study connects deep neural networks with statistical mechanics, revealing natural activation functions.
problem Understanding the activation functions in deep neural networks.
method Statistical Mechanics model of deep neural networks, focusing on encoding, validation, and propagation steps.
result A set of natural activations including Sigmoid, tanh, ReLU, and Swish are identified.
Edge devices adapt pre-trained models to local data without backpropagation.
problem Adapting pre-trained models to edge devices' local data distributions.
method Feed-forward latent domain adaptation using cross-attention.
result Consistent improvements over ERM baselines and domain-supervised adaptation.
Improved bounds on neural network regions using activation histograms.
problem Bounding the number of affine regions in ReLU networks.
method Analysis of algebraic topology problem, extension of framework to subnetwork composition.
result Slightly tighter bounds and insights into parameter initialization.
Proposes a new CG interpretation of neural networks for better theoretical analysis.
problem Lack of theoretical analysis in neural networks interpretation.
method Interprets neural networks as chain graphs and feed-forward as approximate inference.
result Provides novel theoretical support and insights for various neural network techniques.
DeepPhaseCut uses neural networks to improve Fourier phase retrieval.
problem Fourier phase retrieval from magnitude data.
method Unsupervised feed-forward neural network with cycleGAN training.
result Outperforms existing methods in Fourier phase retrieval.
Proposes a neural network for contextual regression.
problem Improving model efficiency and interpretability in regression with contextual features.
method Simple contextual neural network (SCtxtNN) that separates context identification from context-specific regression.
result SCtxtNN achieves lower excess mean squared error and more stable performance than feed-forward neural networks.
FEAST improves FNN's range accuracy in ocean waveguides.
problem Evaluating FNN's range accuracy on unlabeled test data in ocean waveguides.
method Introducing FEAST to evaluate and improve FNN's range accuracy.
result FEAST helps improve FNN's ranging accuracy on test data.
The paper develops a neural network-based method for detecting change points in large-scale time-evolving data.
problem Detecting and locating change points in multivariate time-evolving data.
method Two-step procedure involving neural network training and test error function calibration over moving windows.
result Consistent estimates for the number and locations of change points under temporal dependence.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
Universal Transformer improves sequence modeling accuracy.
problem Feed-forward sequence models fail on simple tasks.
method Parallel-in-time self-attentive recurrent model combining feed-forward and RNN properties.
result UTs outperform standard Transformers on various tasks.
Adaptive Group Lasso selects important features in neural networks.
problem Lack of interpretability in neural networks.
method Adaptive Group Lasso for feature selection.
result Consistent feature selection for neural networks with theoretical guarantee.
This study examines how neural network architecture parameters affect loss surface modality.
problem Understanding the relationship between neural architecture parameters and loss surface modality.
method Fitness landscape analysis of neural network loss surfaces under various architecture settings.
result An increase in problem dimensionality, hidden layer width, and architecture depth affects the modality of loss surfaces.
Stable processes emerge as limits of deep neural networks with symmetric stable distributions.
problem Understanding the behavior of deep neural networks as they become infinitely wide.
method Analyzing fully connected feed-forward deep neural networks with symmetric stable distributions and showing the limit as a stable process.
result The infinite wide limit of the network is a stable process with multivariate stable distributions.
New method adapts without backprop, faster and better.
problem Efficient domain adaptation without source data.
method Computing class prototypes from pre-trained model.
result Significant accuracy improvements over pre-trained model.
We introduce Dynamic Deep Neural Networks (D2NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D2NN neurons are executed, and the particular subset is determined by the D2NN itself. By pruning unnecessary computation depending on input, D2NNs provide a…
Develops hyperparameter transfer methods for Dense Associative Memories.
problem Challenges in transferring hyperparameters for DenseAMs due to unique architecture and activation functions.
method Derives explicit prescriptions for hyperparameter transfer from small to large models.
result Excellent agreement between theoretical and empirical results.
New method uses neural networks to forecast spatial-temporal data.
problem Probabilistic forecasting of spatio-temporal data with causal structure.
method MMAF-guided learning with ensemble of stochastic feed-forward neural networks.
result Forecasting remains calibrated across multiple time horizons.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
TFiLM expands convolutional models' receptive field with minimal overhead.
problem Capturing long-range dependencies in sequential data.
method A novel architectural component using a recurrent neural network to modulate convolutional model activations.
result TFiLM significantly improves learning speed and accuracy on various tasks.
Single-pass method estimates neural network uncertainty.
problem Uncertainty estimation in deep learning requires multiple passes.
method Probabilistic reasoning over neural network depths.
result Single forward pass for uncertainty estimation.
A new algorithm speeds up neural network derivative calculations.
problem Exponential runtime of autodifferentiation for high-order derivatives in neural networks.
method n-TangentProp, a quasilinear algorithm for computing higher-order derivatives.
result Computes exact derivatives in quasilinear time, not exponential.
New methods for uncertainty in neural networks with leaky ReLU activations.
problem Uncertainty in feed-forward neural networks with random input perturbations.
method Analytical expressions for PDF and moments of neural network output, linearization of leaky ReLU, Gaussian copula surrogate models.
result Accurate statistical results for large input perturbations, excellent agreement with Monte Carlo simulations.
We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.
New method uses MMAF-guided learning for spatio-temporal probabilistic forecasts.
problem Probabilistic forecasting of spatio-temporal data with causal structure.
method Generalized Bayesian methodology, MMAF-guided learning, ensemble of stochastic feed-forward neural networks.
result Forecast performance comparable to, and sometimes better than, deep learning architectures.
MTL improves multi-dimensional regression in luminescence sensing.
problem Challenges in modeling multi-dimensional regression problems with classical methods.
method Multi-task learning (MTL) with feed-forward neural networks (FFNNs).
result MTL allows predicting multiple parameters from a single set of measurements.