Critical initialisation strategies are identified for noisy ReLU networks.
problem Understanding signal propagation in noisy rectifier neural networks.
method Developed a new framework for signal propagation in stochastic regularized neural networks, incorporating various noise distributions.
result Critical initialisation strategies for multiplicative noise (e.g. dropout) are identified, but not for additive noise.
Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.
problem Pessimistic initialisation of Q-values in deep RL leads to poor exploration performance.
method Augments pessimistically initialised Q-values with count-based bonuses to ensure optimism.
result OPIQ outperforms non-optimistic DQN variants in hard exploration tasks.
New weight initialisation for ICNNs accelerates learning and improves generalization.
problem Lack of effective initialisation strategies for ICNNs due to their unique weight and activation properties.
method Derived a principled weight initialisation by generalizing signal propagation theory for ICNNs with non-negative weights.
result Principled initialisation effectively accelerates learning and leads to better generalization in ICNNs.
Study shows critical initialisation not crucial for ReLU networks under dropout limits.
problem Effect of initialisation on training speed and generalisation in ReLU networks.
method Large-scale statistical analysis of over 12,000 trained networks.
result Non-critical initialisations perform similarly to critical initialisations in terms of performance.
Study shows proper initialisation of binary weights is crucial for deep neural networks.
problem Training stochastic binary neural networks with continuous surrogates is challenging.
method Developed new surrogates based on Markov chain theory and mean field analysis.
result Critical initialisations are necessary for training deep networks with binary weights.
Unified theory explains two failure modes of deep transformers and provides initialisation guidelines.
problem Two failure modes (rank collapse and entropy collapse) of self-attention layers in deep transformers.
method Analytical theory of signal propagation through deep transformers, using the Random Energy Model analogy.
result Simple algorithm to compute trainability diagrams for correct initialisation hyper-parameters.
New method for k-modes algorithm improves clustering performance.
problem Improving initial solution selection for k-modes algorithm.
method Uses Hospital-Resident Assignment Problem to find initial cluster centroids.
result Outperforms other initialisations in most cases, especially for low-density data.
K-Means clustering improved with sophisticated initialisation techniques.
problem K-Means algorithm's sensitivity to initial centroid positions and local minima.
method Comparison of deterministic and stochastic initialisation techniques for K-Means variations.
result Deterministic methods outperform stochastic methods in clustering quality.
The paper develops a new mathematical framework for group-equivariant operators in machine learning.
problem Developing a robust mathematical framework for group-equivariant operators in machine learning.
method The paper introduces group-equivariant non-expansive operators (GENEOs) and studies their topological and metric properties.
result The space of GENEOs is compact and convex, providing fundamental guarantees for machine learning.
Graph classification models are sensitive to initialisation and structure, but simple models perform well.
problem Graph classification models' performance is sensitive to initialisation and structure.
method Examined recent graph coarsening architectures and their performance sensitivity.
result Simple models like MLP, single-layer GCN, and fixed-weight GCN achieve competitive performance.
New approach connects quantum phases to VQA trainability, enabling better scaling.
problem Scalability issues in VQAs, especially barren plateaus.
method Analog VQA ansätze composed of quenches of a disordered Ising chain, tuning disorder strength.
result Thermalized and MBL phases reach maximal expressivity at large M, but barren plateaus emerge at smaller M in the thermalized phase. Complex-valued neural networks perform similarly to real-valued models for real-valued classification tasks.
problem Comparing real-valued and complex-valued neural networks for real-valued classification tasks.
method Comparison of neural networks with similar capacity sizes, using various activation functions and weight initialisation strategies.
result Complex-valued neural networks perform equal to or slightly worse than real-valued models for real-valued classification tasks.
A new method for fitting mixture models using Boltzmann exploration.
problem Challenges in learning mixture models, especially with good initialisation.
method Boltzmann exploration expectation-maximisation (BEEM) algorithm.
result BEEM can escape local optima and is insensitive to parameter initialisation.
Early alignment in neural networks leads to sparse representations but hinders convergence.
problem The implicit bias of gradient descent during early training phases.
method Quantitative description of early alignment phase in small initialisation, one hidden layer networks.
result Early alignment induces a sparse representation but also hinders convergence to global minima.
Non-determinism from GPUs dominates ResNet training accuracy variability.
problem Variability in ResNet training accuracy due to GPU non-determinism.
method Analysis of TensorFlow ResNet training on GPUs, comparing fixed seeds vs. different seeds.
result 74% of ResNet model variability is due to GPU non-determinism.
In this paper we make two novel contributions to hierarchical clustering. First, we introduce an anomalous pattern initialisation method for hierarchical clustering algorithms, called A-Ward, capable of substantially reducing the time they take to converge. This method generates an initial partition with a sufficiently…
Analyzes deep neural networks training errors with SGD and random init.
problem Lack of rigorous understanding of deep learning algorithms.
method Mathematical analysis of deep learning with SGD and random init.
result First full error analysis for deep learning with SGD and random init.
This study explains gradient flow dynamics in neural networks for small initialisation.
problem Understanding the training dynamics of neural networks for small initialisation.
method Analysis of gradient flow dynamics for one-hidden layer ReLU networks with orthogonal inputs.
result Gradient flow converges to zero loss and characterizes implicit bias towards minimum variation norm.
Two algorithms converge to dictionary learning with geometric rate for non-uniform data.
problem Dictionary learning for non-uniform data models.
method Derivation of convergence conditions for MOD and ODL.
result Both algorithms converge to the generating dictionary with geometric rate under certain conditions.
We present a probabilistic framework for both (i) determining the initial settings of kernel adaptive filters (KAFs) and (ii) constructing fully-adaptive KAFs whereby in addition to weights and dictionaries, kernel parameters are learnt sequentially. This is achieved by formulating the estimator as a probabilistic mode…
Noise regularisation in deep nets makes them behave like Gaussian processes.
problem Understanding the behavior of noise-regularized deep neural networks as Gaussian processes.
method Analyzing the impact of noise regularisation on neural network Gaussian processes (NNGPs) and relating their behavior to signal propagation theory.
result Best performing NNGPs have kernel parameters corresponding to a specific initialisation scheme.
Batch normalization prevents rank collapse in deep networks, improving training stability.
problem Rank collapse in randomly initialized deep networks with increasing depth.
method Investigates spectral instabilities in random matrices and uses batch normalization to avoid rank collapse.
result Batch normalization prevents rank collapse in both linear and ReLU networks, improving training stability.
A new method averages neural network parameters to rank features robustly.
problem Neural networks' sensitivity to random initialization affects feature ranking robustness.
method Parameter averaging of multiple shallow networks trained with different random seeds.
result The averaged model discovers ground-truth feature importance consistently.
This paper studies the convergence behaviour of dictionary learning via the Iterative Thresholding and K-residual Means (ITKrM) algorithm. On one hand it is proved that ITKrM is a contraction under much more relaxed conditions than previously necessary. On the other hand it is shown that there seem to exist stable fixe…
Study improves Bayesian optimisation with ensemble transfer learning.
problem Improving sample efficiency in Bayesian optimisation of expensive functions.
method Empirical analysis of ensemble-based transfer learning methods and pipeline components.
result Two components (warm start initialisation and positive weight constraint) improve transfer learning Bayesian optimisation performance.
Overparameterisation can limit the benefits of curriculum learning in neural networks.
problem The ineffectiveness of curriculum learning in deep learning applications.
method Analytical study connecting curriculum learning and overparameterisation in an online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem.
result High degree of overparameterisation can limit the benefit from curricula.
One-pass optimisation for high-dimensional hyperparameters.
problem Efficient optimisation of hyperparameters in machine learning models.
method Approximate hypergradient-based optimisation for any continuous hyperparameter, requiring only one training episode.
result Competitive performance on various datasets without hyperparameter restarts.
The paper extends infinite-width analysis to neural network Jacobians, revealing convergence to Gaussian processes and linear ODEs.
problem Understanding the training dynamics of neural networks in the infinite-width limit.
method Extending infinite-width analysis to Jacobians, characterizing convergence to Gaussian processes and linear ODEs.
result The evolution of MLPs under robust training in the infinite-width limit is described by a linear ODE.
Adversarial adaptation improves reinforcement learning performance.
problem Slow training and exploration in reinforcement learning.
method Adversarial domain adaptation for feature initialization.
result Significant improvement in reinforcement learning performance.
GENESIS-V2 infers unordered object representations without iterative refinement.
problem Unsupervised learning of unordered object representations for complex images.
method Stochastic stick-breaking process for clustering pixel embeddings.
result GENESIS-V2 outperforms recent baselines in unsupervised image segmentation and scene generation.
einspace expands NAS search space to include diverse neural architectures.
problem NAS results are often limited to existing structures; new designs are rare.
method einspace uses a probabilistic context-free grammar to create a versatile search space.
result einspace discovers novel and improved architectures, including convolutions and attention.
A new clustering model for mixed datasets combines continuous and non-continuous data.
problem Challenges in clustering mixed data due to heterogeneous variables.
method Mixed Deep Gaussian Mixture Model (MDGMM) with multilayer architecture.
result Automatic selection of model specifications and optimal number of clusters.
In this paper we propose a model that combines the strengths of RNNs and SGVB: the Variational Recurrent Auto-Encoder (VRAE). Such a model can be used for efficient, large scale unsupervised learning on time series data, mapping the time series data to a latent vector representation. The model is generative, such that …
Bayesian method learns neural network architecture parameters.
problem Estimating optimal neural network architecture parameters.
method Bayesian learning of concrete distributions over layer size and network depth.
result Regular networks with learnt structure generalize better on small datasets, while stochastic networks are more robust to initialisation.
We propose the application of a semi-supervised learning method to improve the performance of acoustic modelling for automatic speech recognition based on deep neural net- works. As opposed to unsupervised initialisation followed by supervised fine tuning, our method takes advantage of both unlabelled and labelled data…
Study combines variational inference and transformers for seasonal climate predictions.
problem Lack of robust seasonal predictions due to limited historical records and computational constraints.
method Combines variational inference with transformer models trained on climate model output.
result Method provides skilful predictions beyond climate change-induced trends in various regions.
Transformers interpreted as probabilistic Laplacian Eigenmaps steps.
problem Improving transformer performance through probabilistic interpretation.
method Probabilistic Laplacian Eigenmaps model derivation and graph diffusion step.
result Subtracting identity from attention matrix improves transformer performance.
Heuristic optimisers which search for an optimal configuration of variables relative to an objective function often get stuck in local optima where the algorithm is unable to find further improvement. The standard approach to circumvent this problem involves periodically restarting the algorithm from random initial con…
We demonstrate the first application of deep reinforcement learning to autonomous driving. From randomly initialised parameters, our model is able to learn a policy for lane following in a handful of training episodes using a single monocular image as input. We provide a general and easy to obtain reward: the distance …
Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then eva…
Hierarchical pretraining with slow-fast ODEs
problem Causal self-attention vs. slow-fast ODEs
method Instantiating fast-slow ODE formalism as a concrete neural network
result Equilibrium manifold x=φ(y) is exactly the master-equation (ME) stationary distribution Machine learning improves chaotic dynamical system simulations with empirical error correction.
problem Improving chaotic dynamical system simulations using machine learning.
method Combining machine learning with physically-derived models to correct timestep errors.
result The approach yields stable models with improved long-term statistics and single time-step tendencies.
A new heuristic LM algorithm improves k-segmentation accuracy with less computation.
problem Efficiently segmenting large video streams into meaningful piecewise-linear segments.
method Inspired by Lloyd's and Lloyd-Max algorithms, LM algorithm iteratively minimizes a cost function.
result LM algorithm achieves competitive accuracy with exact methods at a fraction of the computational cost.
Gaussian processes are typically used for smoothing and interpolation on small datasets. We introduce a new Bayesian nonparametric framework -- GPatt -- enabling automatic pattern extrapolation with Gaussian processes on large multidimensional datasets. GPatt unifies and extends highly expressive kernels and fast exact…
Wide stochastic networks show Gaussian behavior and improve training with PAC-Bayesian methods.
problem Analyzing and training over-parameterised neural networks with large width.
method Establishing Gaussian behavior for a stochastic architecture, applying PAC-Bayesian training.
result PAC-Bayesian training on large but finite-width networks outperforms standard methods.
Bayesian inference for factorial hidden Markov models is challenging due to the exponentially sized latent variable space. Standard Monte Carlo samplers can have difficulties effectively exploring the posterior landscape and are often restricted to exploration around localised regions that depend on initialisation. We …
Optimizes wide low-rank neural networks for reduced parameters and cost.
problem Reducing the number of learnable parameters in wide neural networks.
method Analyzed edge-of-chaos dynamics and derived formulae for optimal weight and bias variances.
result Optimal weight and bias variances for low-rank networks follow from multiplicative scaling.
High-dimensional unimodal distributions can cause MCMC methods to fail.
problem Failure of MCMC methods in high-dimensional unimodal distributions.
method Examples and theoretical analysis of MCMC methods, including Metropolis-Hastings adjusted methods.
result MCMC methods can take an exponential run-time for high-dimensional unimodal distributions.