A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
GNIs induce a regulariser that penalizes high-frequency components in neural network activations.
New framework monitors neural network training and reveals regularisation mechanisms.
Study on neural networks with regularisation and its impact on training dynamics.
Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs). The behaviour of these models depends on the initialisation of the corresponding network. In this work, we consider the impact of noise regularisation …
Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will h…
New insights into how neural networks learn features, especially when they are very wide.
This work shows how penalising bias terms in norm regularisation leads to sparse solutions.
Study on how noise and variation-norm regularisation help shallow ReLU networks use fewer neurons.
We develop a new method for regularising neural networks. We learn a probability distribution over the activations of all layers of the model and then insert imputed values into the network during training. We obtain a posterior for an arbitrary subset of activations conditioned on the remainder. This is a generalisati…
Regularization preserves topological data structure in autoencoders.
Noise-driven neural networks emerge modular structures, improving robustness and generalization.
Novel method decorrelates neurons for better deep learning model generalization.
This paper improves inverse problem solving with weakly convex regularisers and proves convergence.
The paper extends infinite-width analysis to neural network Jacobians, revealing convergence to Gaussian processes and linear ODEs.
Deep learning using multi-layer neural networks (NNs) architecture manifests superb power in modern machine learning systems. The trained Deep Neural Networks (DNNs) are typically large. The question we would like to address is whether it is possible to simplify the NN during training process to achieve a reasonable pe…
Investigates gradient descent dynamics and introduces new regularisation methods.
This paper introduces GEMINI, a new mutual information metric for unsupervised neural network training.
Deep Learning Accelerators are prone to faults which manifest in the form of errors in Neural Networks. Fault Tolerance in Neural Networks is crucial in real-time safety critical applications requiring computation for long durations. Neural Networks with high regularisation exhibit superior fault tolerance, however, at…
This paper introduces GEMINI, a new metric for unsupervised neural network training that avoids the need for regularizations.
Improves CNN stability by translating classical signal denoising methods.
We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the ma…
We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant---for multiple -norms---of a feed forward neural network composed of commonly used layer types.…
Bayesian neural networks improve uncertainty quantification with unlabelled data.
We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network. This method, referred to as functional regularisation for Continual Learning, avoids forgetting a previous task by constructing and memorising an approximate post…
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new frame…
Improved deep kernel machine achieves 94.5% test accuracy on CIFAR-10.
QSD enhances deep network performance through biologically plausible dropout.
This paper examines the assumptions of the derived equivalence between dropout noise injection and regularisation for logistic regression with negative log loss. We show that the approximation method is based on a divergent Taylor expansion, making, subsequent work using this approximation to compare the dropout …
A new method constrains deep networks during fine-tuning to improve generalization.
Convolutional neural networks have had a great success in numerous tasks, including image classification, object detection, sequence modelling, and many more. It is generally assumed that such neural networks are translation invariant, meaning that they can detect a given feature independent of its location in the inpu…
Neural networks have been used as a nonparametric method for option pricing and hedging since the early 1990s. Far over a hundred papers have been published on this topic. This note intends to provide a comprehensive review. Papers are compared in terms of input features, output variables, benchmark models, performance…
Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.
ProSelfLC improves robustness of deep neural networks by automatically deciding trust in predictions.
Bayesian interpretations of neural network have a long history, dating back to early work in the 1990's and have recently regained attention because of their desirable properties like uncertainty estimation, model robustness and regularisation. We want to discuss here the application of Bayesian models to knowledge sha…
Regularising for invariance to data augmentation improves machine learning models.
We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible inp…
nUDEs use neural networks to model biology without negative values.
Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimisation tools developed for potential energy landscapes in molecular science. The number of local minima and transition sta…
Robust risk minimisation has several advantages: it has been studied with regards to improving the generalisation properties of models and robustness to adversarial perturbation. We bound the distributionally robust risk for a model class rich enough to include deep neural networks by a regularised empirical risk invol…
Generalisation of a deep neural network (DNN) is one major concern when employing the deep learning approach for solving practical problems. In this paper we propose a new technique, named approximated orthonormal normalisation (AON), to improve the generalisation capacity of a DNN model. Considering a weight matrix W …
Given the potential difficulties in obtaining large quantities of labelled data, many works have explored the use of deep semi-supervised learning, which uses both labelled and unlabelled data to train a neural network architecture. The vast majority of SSL approaches focus on implementing the low-density separation as…
Unified understanding of three continual learning regularisation methods.
Deep neural networks are typically too computationally expensive to run in real-time on consumer-grade hardware and low-powered devices. In this paper, we investigate reducing the computational and memory requirements of neural networks through network pruning and quantisation. We examine their efficacy on large networ…
Enhanced feature learning using neural networks and kernel methods with improved robustness.
New method prevents deep learning forgetting past by remembering key examples.
This research improves multimodal systems by adding a second objective and regularisation methods.
Dropout is a regularisation technique in neural network training where unit activations are randomly set to zero with a given probability \emph{independently}. In this work, we propose a generalisation of dropout and other multiplicative noise injection schemes for shallow and deep neural networks, where the random noi…
New method stabilizes deep learning models for clinical risk prediction.