A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Neural nets trained with linear discriminant initialization converge faster and more accurately.
The study compares feed-forward and attention layers in language models.
This work optimizes reservoir computing models by linking recurrence and non-linear dynamics.
New method adapts without backprop, faster and better.
Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of…
Efficient neural network optimization reduces costs and improves model performance.
Edge devices adapt pre-trained models to local data without backpropagation.
Basic binary relations such as equality and inequality are fundamental to relational data structures. Neural networks should learn such relations and generalise to new unseen data. We show in this study, however, that this generalisation fails with standard feed-forward networks on binary vectors. Even when trained wit…
Adapting robust statistics to neural networks, researchers found neural networks can be more robust with certain loss functions.
The universal approximation property of various machine learning models is currently only understood on a case-by-case basis, limiting the rapid development of new theoretically justified neural network architectures and blurring our understanding of our current models' potential. This paper works towards overcoming th…
We propose a feed-forward inference method applicable to belief and neural networks. In a belief network, the method estimates an approximate factorized posterior of all hidden units given the input. In neural networks the method propagates uncertainty of the input through all the layers. In neural networks with inject…
We investigate the capacity, convexity and characterization of a general family of norm-constrained feed-forward networks.
A reliable controller is critical and essential for the execution of safe and smooth maneuvers of an autonomous vehicle.The controller must be robust to external disturbances, such as road surface, weather, and wind conditions, and so on.It also needs to deal with the internal parametric variations of vehicle sub-syste…
When a feed-forward neural network (FNN) is trained for source ranging in an ocean waveguide, it is difficult evaluating the range accuracy of the FNN on unlabeled test data. A fitting-based early stopping (FEAST) method is introduced to evaluate the range error of the FNN on test data where the distance of source is u…
In this paper, we model the impact of oil price volatility on Tehranstock and industry indices in two periods of international sanctions and post-sanction. To analyse the purpose of study, we use Feed-forward neural net-works. The period of study is from 2008 to 2018 that is split in two periods during international en…
New study shows neural networks need many samples for training.
New insights into how encoder-decoder networks generate attention matrices.
Deep network learns Obstacle Tower challenge without human demonstrations.
General lower bounds on neural network approximation in L^p norm.
Training deep recurrent neural network (RNN) architectures is complicated due to the increased network complexity. This disrupts the learning of higher order abstracts using deep RNN. In case of feed-forward networks training deep structures is simple and faster while learning long-term temporal information is not poss…
Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward …
Proposes a new CG interpretation of neural networks for better theoretical analysis.
We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimension…
Learning representations that accurately capture long-range dependencies in sequential inputs -- including text, audio, and genomic data -- is a key problem in deep learning. Feed-forward convolutional models capture only feature interactions within finite receptive fields while recurrent architectures can be slow and …
DeepPhaseCut uses neural networks to improve Fourier phase retrieval.
Improved bounds on neural network regions using activation histograms.
It has been argued in the past that high-dimensional neural networks do not exhibit local minima capable of trapping an optimisation algorithm. However, the relationship between loss surface modality and the neural architecture parameters, such as the number of hidden neurons per layer and the number of hidden layers, …
We design a self size-estimating feed-forward network (SSFN) using a joint optimization approach for estimation of number of layers, number of nodes and learning of weight matrices. The learning algorithm has a low computational complexity, preferably within few minutes using a laptop. In addition the algorithm has a l…
Recurrent and convolutional neural networks are the most common architectures used for time series forecasting in deep learning literature. These networks use parameter sharing by repeating a set of fixed architectures with fixed parameters over time or space. The result is that the overall architecture is time-invaria…
Two of the most popular modelling paradigms in computer vision are feed-forward neural networks (FFNs) and probabilistic graphical models (GMs). Various connections between the two have been studied in recent works, such as e.g. expressing mean-field based inference in a GM as an FFN. This paper establishes a new conne…
The paper develops a neural network-based method for detecting change points in large-scale time-evolving data.
Proposes a neural network for contextual regression.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been show…
Develops hyperparameter transfer methods for Dense Associative Memories.
MTL improves multi-dimensional regression in luminescence sensing.
New method uses neural networks to forecast spatial-temporal data.
Adaptive Group Lasso selects important features in neural networks.
We introduce Dynamic Deep Neural Networks (D2NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D2NN neurons are executed, and the particular subset is determined by the D2NN itself. By pruning unnecessary computation depending on input, D2NNs provide a…
New method uses MMAF-guided learning for spatio-temporal probabilistic forecasts.
We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…
A new algorithm speeds up neural network derivative calculations.
New methods for uncertainty in neural networks with leaky ReLU activations.
This paper improves speech synthesis using a DGP with SRU for naturalness.
NN2Poly converts deep neural networks into polynomial models for better understanding.
This paper removes the finite variance assumption for deep convolutional neural networks.