GNIs induce a regulariser that penalizes high-frequency components in neural network activations.
problem Understanding the regularizing effect of Gaussian noise injections on neural network activations.
method Deriving the explicit regularizer by marginalizing out injected noise and analyzing its effect in the Fourier domain.
result GNIs induce a regularizer that produces calibrated classifiers with large margins.
New framework monitors neural network training and reveals regularisation mechanisms.
problem Overfitting in neural networks and the need for explicit regularizers.
method Model Gradient Similarity (MGS) framework to measure and monitor regularisation.
result Explicit regularizers increase Model Gradient Similarity (MGS).
Study on neural networks with regularisation and its impact on training dynamics.
problem Understanding the dynamics of neural networks with regularization.
method Established explicit dynamics for neural networks with a regularizing term, linearizing around initialisation.
result The regularisation term modifies the standard NTK dynamics, leading to new insights into network training.
Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs). The behaviour of these models depends on the initialisation of the corresponding network. In this work, we consider the impact of noise regularisation …
Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will h…
New insights into how neural networks learn features, especially when they are very wide.
problem Understanding how gradient flow in wide neural networks selects solutions, especially in the feature-learning regime.
method Axiomatizing the canonical regularizer as a function-space energy and lift, and deriving geodesic ridge for the feature-learning regime.
result Gradient flow in feature-learning networks biases towards ridge regularization, distorting the inductive bias and damaging pretrained networks.
This work shows how penalising bias terms in norm regularisation leads to sparse solutions.
problem Understanding the relation between parameter norm regularization and the sparsity of neural network solutions.
method Analyzes one hidden ReLU layer networks with unidimensional data, showing the norm required for function representation and the importance of the bias term's norm.
result Penalising the bias terms in regularisation leads to sparse solutions, enforcing the uniqueness and sparsity of the minimal norm interpolator.
Study on how noise and variation-norm regularisation help shallow ReLU networks use fewer neurons.
problem Understanding how shallow ReLU networks use a finite number of neurons in the infinitely wide limit.
method Analysis of two regularisation strategies: noise injection and variation-norm.
result Both regularisation methods minimize functions with a finite number of neurons, regardless of overparametrisation.
We develop a new method for regularising neural networks. We learn a probability distribution over the activations of all layers of the model and then insert imputed values into the network during training. We obtain a posterior for an arbitrary subset of activations conditioned on the remainder. This is a generalisati…
Regularization preserves topological data structure in autoencoders.
problem Ensuring topological data structure preservation in autoencoders.
method Regularization using Legendre nodes to preserve manifold embedding.
result Regularized autoencoders ensure one-to-one embedding of data manifolds.
Noise-driven neural networks emerge modular structures, improving robustness and generalization.
problem Artificial neural networks struggle with modular solutions, leading to poor generalization and robustness.
method Inspired by brain's modular architecture, the study uses neural noise and nonlinear responses to drive the emergence of modular solutions.
result Noise-driven modularisation improves robustness and generalization in neural networks.
Novel method decorrelates neurons for better deep learning model generalization.
problem High correlations between neurons limit deep learning model generalization.
method Regularization terms from minimum spanning tree of neuron cliques, using correlation dissimilarities.
result Our regularizers outperform existing methods and minimize neuron redundancies.
This paper improves inverse problem solving with weakly convex regularisers and proves convergence.
problem Improving solution methods for inverse problems.
method Generalised formulation of convergent regularisation using weakly convex regularisers, and proof of convergence for primal-dual hybrid gradient method.
result Proves convergence of primal-dual hybrid gradient method for variational problems and shows improved performance with IWCNNs.
The paper extends infinite-width analysis to neural network Jacobians, revealing convergence to Gaussian processes and linear ODEs.
problem Understanding the training dynamics of neural networks in the infinite-width limit.
method Extending infinite-width analysis to Jacobians, characterizing convergence to Gaussian processes and linear ODEs.
result The evolution of MLPs under robust training in the infinite-width limit is described by a linear ODE.
Deep learning using multi-layer neural networks (NNs) architecture manifests superb power in modern machine learning systems. The trained Deep Neural Networks (DNNs) are typically large. The question we would like to address is whether it is possible to simplify the NN during training process to achieve a reasonable pe…
Investigates gradient descent dynamics and introduces new regularisation methods.
problem Understanding and mitigating gradient descent instabilities and interactions with smoothness regularisation.
method Derives continuous-time flows to account for discretisation drift, constructs learning rate schedules and regularisers.
result New regularisation methods improve performance in reinforcement learning.
This paper introduces GEMINI, a new mutual information metric for unsupervised neural network training.
problem The mutual information (MI) as a clustering objective does not lead to satisfactory clusters.
method The authors generalised MI by changing its core distance, introducing GEMINIs that do not require regularizations and can automatically select the number of clusters.
result GEMINIs can automatically select the number of clusters without requiring a priori knowledge of the number of clusters.
Deep Learning Accelerators are prone to faults which manifest in the form of errors in Neural Networks. Fault Tolerance in Neural Networks is crucial in real-time safety critical applications requiring computation for long durations. Neural Networks with high regularisation exhibit superior fault tolerance, however, at…
This paper introduces GEMINI, a new metric for unsupervised neural network training that avoids the need for regularizations.
problem The mutual information (MI) as a clustering objective does not lead to satisfactory clusters.
method The authors generalised the mutual information by changing its core distance, introducing the Generalised Mutual Information (GEMINI).
result Some GEMINIs do not require regularizations when training and can automatically select the number of clusters.
Improves CNN stability by translating classical signal denoising methods.
problem Stability of CNNs is poorly understood.
method Interprets classical signal denoising methods as ResNet architectures.
result Translates diffusivities, shrinkage functions, and regularizers into CNN activation functions.
This paper addresses the issue of feature importance landscapes in complex images and proposes a regularisation technique to improve network performance.
problem Feature importance landscapes in complex images are not as uniform as assumed, affecting network performance.
method Developed the PILCRO objective to regularize weight configurations, making importance landscapes smoother and more data-driven.
result P-regularised networks have a flat importance landscape, train faster, and perform better in accuracy and robustness.
We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the ma…
We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant---for multiple p-norms---of a feed forward neural network composed of commonly used layer types.…
Bayesian neural networks improve uncertainty quantification with unlabelled data.
problem Over-confidence in predictions on covariate-shifted data.
method Approximate Bayesian inference using posterior regularisation with pseudo-labels from unlabelled data.
result Significant improvement in uncertainty quantification accuracy on covariate-shifted data.
We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network. This method, referred to as functional regularisation for Continual Learning, avoids forgetting a previous task by constructing and memorising an approximate post…
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new frame…
Improved deep kernel machine achieves 94.5% test accuracy on CIFAR-10.
problem Achieving high accuracy on complex tasks like image classification.
method Stochastic kernel regularisation to add noise to learned Gram matrices.
result 94.5% test accuracy on CIFAR-10.
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
This paper examines the assumptions of the derived equivalence between dropout noise injection and L2 regularisation for logistic regression with negative log loss. We show that the approximation method is based on a divergent Taylor expansion, making, subsequent work using this approximation to compare the dropout …
A new method constrains deep networks during fine-tuning to improve generalization.
problem Improving generalization of fine-tuned deep networks.
method A neural network generalisation bound based on distance from initial weights constrains the hypothesis class to a small sphere.
result Empirical evaluation shows superior generalization performance compared to existing methods.
CycleCluster uses clustering to improve deep semi-supervised learning.
problem Difficulty in obtaining labelled data for deep learning.
method Proposes a new framework using clustering regularisation and graph-based pseudo-labels.
result Demonstrates improved predictive capability through numerical results.
Neural networks have been used as a nonparametric method for option pricing and hedging since the early 1990s. Far over a hundred papers have been published on this topic. This note intends to provide a comprehensive review. Papers are compared in terms of input features, output variables, benchmark models, performance…
Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.
problem Understanding the impact of stochasticity and large stepsizes on gradient descent and SGD solutions.
method Investigation of GD and SGD over diagonal linear networks with macroscopic stepsizes, proving convergence and characterizing solutions.
result Large stepsizes consistently benefit SGD for sparse regression problems, but can hinder GD recovery of sparse solutions, especially in the edge of stability regime.
ProSelfLC improves robustness of deep neural networks by automatically deciding trust in predictions.
problem Training robust deep neural networks requires addressing issues like label noise and low entropy predictions.
method ProSelfLC progressively increases trust in predicted labels over time, considering entropy and learning time.
result ProSelfLC demonstrates improved robustness in both clean and noisy settings through empirical validation.
Bayesian interpretations of neural network have a long history, dating back to early work in the 1990's and have recently regained attention because of their desirable properties like uncertainty estimation, model robustness and regularisation. We want to discuss here the application of Bayesian models to knowledge sha…
Regularising for invariance to data augmentation improves machine learning models.
problem Improving generalization in machine learning models through data augmentation.
method Explicit regularisation to encourage invariance at the level of individual model predictions.
result Explicit regularisation improves generalization and equalizes performance differences between objectives.
We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible inp…
nUDEs use neural networks to model biology without negative values.
problem Unrealistic negative values in hybrid models of biology.
method Developed non-negative UDEs (nUDEs) with regularization techniques.
result nUDEs provide realistic solutions for biological models.
Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimisation tools developed for potential energy landscapes in molecular science. The number of local minima and transition sta…
Robust risk minimisation has several advantages: it has been studied with regards to improving the generalisation properties of models and robustness to adversarial perturbation. We bound the distributionally robust risk for a model class rich enough to include deep neural networks by a regularised empirical risk invol…
Generalisation of a deep neural network (DNN) is one major concern when employing the deep learning approach for solving practical problems. In this paper we propose a new technique, named approximated orthonormal normalisation (AON), to improve the generalisation capacity of a DNN model. Considering a weight matrix W …
Paper explores pruning and quantisation to compress neural networks.
problem Reduces computational and memory costs of deep neural networks.
method Investigates network pruning and quantisation for AlexNet, ShuffleNet, and MobileNet.
result Pruning and quantisation compress networks to less than half their size and improve efficiency.
Unified understanding of three continual learning regularisation methods.
problem Maintaining knowledge of earlier tasks without re-accessing them.
method Three regularisation approaches: Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI), and Memory Aware Synapses (MAS).
result EWC, SI, and MAS are linked to the same theoretical quantity, the square root of the Fisher Information.
Enhanced feature learning using neural networks and kernel methods with improved robustness.
problem Improving feature learning and function estimation in supervised learning.
method Regularised empirical risk minimisation with a new kernel approach.
result The proposed method, BKerNN, converges to the minimal risk with explicit high-probability rates.
New method prevents deep learning forgetting past by remembering key examples.
problem Catastrophic forgetting in continual learning.
method Functional regularisation using Gaussian Process formulation.
result Achieves state-of-the-art performance on benchmarks.
This research improves multimodal systems by adding a second objective and regularisation methods.
problem Improving performance of multimodal systems with multiple objectives and regularisation.
method Introduces a second objective over multimodal fusion using variational inference and regularisation methods.
result Demonstrates potential for multiple objectives and probabilistic methods to lower variance and improve generalisation.
Dropout is a regularisation technique in neural network training where unit activations are randomly set to zero with a given probability \emph{independently}. In this work, we propose a generalisation of dropout and other multiplicative noise injection schemes for shallow and deep neural networks, where the random noi…
New method stabilizes deep learning models for clinical risk prediction.
problem Stability issues in deep learning models for clinical risk prediction.
method Bootstrapping-based regularisation framework embedded in deep neural networks.
result Improved prediction stability across multiple datasets.