Study shows proper initialisation of binary weights is crucial for deep neural networks.
problem Training stochastic binary neural networks with continuous surrogates is challenging.
method Developed new surrogates based on Markov chain theory and mean field analysis.
result Critical initialisations are necessary for training deep networks with binary weights.
New weight initialisation for ICNNs accelerates learning and improves generalization.
problem Lack of effective initialisation strategies for ICNNs due to their unique weight and activation properties.
method Derived a principled weight initialisation by generalizing signal propagation theory for ICNNs with non-negative weights.
result Principled initialisation effectively accelerates learning and leads to better generalization in ICNNs.
Unified theory explains two failure modes of deep transformers and provides initialisation guidelines.
problem Two failure modes (rank collapse and entropy collapse) of self-attention layers in deep transformers.
method Analytical theory of signal propagation through deep transformers, using the Random Energy Model analogy.
result Simple algorithm to compute trainability diagrams for correct initialisation hyper-parameters.
Graph classification models are sensitive to initialisation and structure, but simple models perform well.
problem Graph classification models' performance is sensitive to initialisation and structure.
method Examined recent graph coarsening architectures and their performance sensitivity.
result Simple models like MLP, single-layer GCN, and fixed-weight GCN achieve competitive performance.
Study shows critical initialisation not crucial for ReLU networks under dropout limits.
problem Effect of initialisation on training speed and generalisation in ReLU networks.
method Large-scale statistical analysis of over 12,000 trained networks.
result Non-critical initialisations perform similarly to critical initialisations in terms of performance.
Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.
problem Pessimistic initialisation of Q-values in deep RL leads to poor exploration performance.
method Augments pessimistically initialised Q-values with count-based bonuses to ensure optimism.
result OPIQ outperforms non-optimistic DQN variants in hard exploration tasks.
We present a probabilistic framework for both (i) determining the initial settings of kernel adaptive filters (KAFs) and (ii) constructing fully-adaptive KAFs whereby in addition to weights and dictionaries, kernel parameters are learnt sequentially. This is achieved by formulating the estimator as a probabilistic mode…
Critical initialisation strategies are identified for noisy ReLU networks.
problem Understanding signal propagation in noisy rectifier neural networks.
method Developed a new framework for signal propagation in stochastic regularized neural networks, incorporating various noise distributions.
result Critical initialisation strategies for multiplicative noise (e.g. dropout) are identified, but not for additive noise.
Complex-valued neural networks perform similarly to real-valued models for real-valued classification tasks.
problem Comparing real-valued and complex-valued neural networks for real-valued classification tasks.
method Comparison of neural networks with similar capacity sizes, using various activation functions and weight initialisation strategies.
result Complex-valued neural networks perform equal to or slightly worse than real-valued models for real-valued classification tasks.
New method for k-modes algorithm improves clustering performance.
problem Improving initial solution selection for k-modes algorithm.
method Uses Hospital-Resident Assignment Problem to find initial cluster centroids.
result Outperforms other initialisations in most cases, especially for low-density data.
In this paper we make two novel contributions to hierarchical clustering. First, we introduce an anomalous pattern initialisation method for hierarchical clustering algorithms, called A-Ward, capable of substantially reducing the time they take to converge. This method generates an initial partition with a sufficiently…
K-Means clustering improved with sophisticated initialisation techniques.
problem K-Means algorithm's sensitivity to initial centroid positions and local minima.
method Comparison of deterministic and stochastic initialisation techniques for K-Means variations.
result Deterministic methods outperform stochastic methods in clustering quality.
Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then eva…
In this paper we propose a model that combines the strengths of RNNs and SGVB: the Variational Recurrent Auto-Encoder (VRAE). Such a model can be used for efficient, large scale unsupervised learning on time series data, mapping the time series data to a latent vector representation. The model is generative, such that …
The paper extends infinite-width analysis to neural network Jacobians, revealing convergence to Gaussian processes and linear ODEs.
problem Understanding the training dynamics of neural networks in the infinite-width limit.
method Extending infinite-width analysis to Jacobians, characterizing convergence to Gaussian processes and linear ODEs.
result The evolution of MLPs under robust training in the infinite-width limit is described by a linear ODE.
A new method for fitting mixture models using Boltzmann exploration.
problem Challenges in learning mixture models, especially with good initialisation.
method Boltzmann exploration expectation-maximisation (BEEM) algorithm.
result BEEM can escape local optima and is insensitive to parameter initialisation.
Study improves Bayesian optimisation with ensemble transfer learning.
problem Improving sample efficiency in Bayesian optimisation of expensive functions.
method Empirical analysis of ensemble-based transfer learning methods and pipeline components.
result Two components (warm start initialisation and positive weight constraint) improve transfer learning Bayesian optimisation performance.
A new method constrains deep networks during fine-tuning to improve generalization.
problem Improving generalization of fine-tuned deep networks.
method A neural network generalisation bound based on distance from initial weights constrains the hypothesis class to a small sphere.
result Empirical evaluation shows superior generalization performance compared to existing methods.
Early alignment in neural networks leads to sparse representations but hinders convergence.
problem The implicit bias of gradient descent during early training phases.
method Quantitative description of early alignment phase in small initialisation, one hidden layer networks.
result Early alignment induces a sparse representation but also hinders convergence to global minima.
Optimizes wide low-rank neural networks for reduced parameters and cost.
problem Reducing the number of learnable parameters in wide neural networks.
method Analyzed edge-of-chaos dynamics and derived formulae for optimal weight and bias variances.
result Optimal weight and bias variances for low-rank networks follow from multiplicative scaling.
We provide an overview of several non-linear activation functions in a neural network architecture that have proven successful in many machine learning applications. We conduct an empirical analysis on the effectiveness of using these function on the MNIST classification task, with the aim of clarifying which functions…
In this article, we propose the Coopetititve Soft Gating Ensemble or CSGE for general machine learning tasks and interwoven systems. The goal of machine learning is to create models that generalize well for unknown datasets. Often, however, the problems are too complex to be solved with a single model, so several model…
Non-determinism from GPUs dominates ResNet training accuracy variability.
problem Variability in ResNet training accuracy due to GPU non-determinism.
method Analysis of TensorFlow ResNet training on GPUs, comparing fixed seeds vs. different seeds.
result 74% of ResNet model variability is due to GPU non-determinism.
Analyzes deep neural networks training errors with SGD and random init.
problem Lack of rigorous understanding of deep learning algorithms.
method Mathematical analysis of deep learning with SGD and random init.
result First full error analysis for deep learning with SGD and random init.
Paper proposes a new method for better POS tagging adaptation.
problem Difficulty of pre-trained units learning target-specific patterns.
method Augment target-network with normalised, weighted, and randomly initialized units.
result Achieves state-of-the-art performances on POS tagging.
Improved generalization bounds for multi-class CNNs without explicit class dependence.
problem Generalization error bounds for deep learning with multi-class CNNs.
method Adapted Rademacher analysis to incorporate weight sharing, reducing dependence on the number of classes.
result Bounds have no explicit dependence on the number of classes, scaling with the norm of weight matrices.
One-pass optimisation for high-dimensional hyperparameters.
problem Efficient optimisation of hyperparameters in machine learning models.
method Approximate hypergradient-based optimisation for any continuous hyperparameter, requiring only one training episode.
result Competitive performance on various datasets without hyperparameter restarts.
This study explains gradient flow dynamics in neural networks for small initialisation.
problem Understanding the training dynamics of neural networks for small initialisation.
method Analysis of gradient flow dynamics for one-hidden layer ReLU networks with orthogonal inputs.
result Gradient flow converges to zero loss and characterizes implicit bias towards minimum variation norm.
Two algorithms converge to dictionary learning with geometric rate for non-uniform data.
problem Dictionary learning for non-uniform data models.
method Derivation of convergence conditions for MOD and ODL.
result Both algorithms converge to the generating dictionary with geometric rate under certain conditions.
Noise regularisation in deep nets makes them behave like Gaussian processes.
problem Understanding the behavior of noise-regularized deep neural networks as Gaussian processes.
method Analyzing the impact of noise regularisation on neural network Gaussian processes (NNGPs) and relating their behavior to signal propagation theory.
result Best performing NNGPs have kernel parameters corresponding to a specific initialisation scheme.
The study uses machine learning to predict financial market trends.
problem Predicting financial market trends using low-frequency data.
method Modular online machine learning framework using stacked autoencoders and neural networks.
result The approach can predict financial market feature fluctuations effectively.
A new method averages neural network parameters to rank features robustly.
problem Neural networks' sensitivity to random initialization affects feature ranking robustness.
method Parameter averaging of multiple shallow networks trained with different random seeds.
result The averaged model discovers ground-truth feature importance consistently.
New approach connects quantum phases to VQA trainability, enabling better scaling.
problem Scalability issues in VQAs, especially barren plateaus.
method Analog VQA ansätze composed of quenches of a disordered Ising chain, tuning disorder strength.
result Thermalized and MBL phases reach maximal expressivity at large M, but barren plateaus emerge at smaller M in the thermalized phase. We introduce causal pieces to improve spiking neural networks.
problem Improving the expressiveness and trainability of spiking neural networks.
method We decompose the input domain of SNNs into causal regions, proving that the number of these regions is a measure of SNNs' approximation capabilities.
result Parameter initialisations yielding a high number of causal pieces correlate with SNN training success.
The paper analyzes the role of ReLU gates in deep learning networks.
problem Understanding the role of gates in deep learning networks.
method Developed neural path features (NPF) and neural path values (NPV) to characterize the active sub-networks during training.
result The neural path kernel associated with NPFs is a fundamental quantity that characterizes the information stored in the gates of a DNN.
Adversarial adaptation improves reinforcement learning performance.
problem Slow training and exploration in reinforcement learning.
method Adversarial domain adaptation for feature initialization.
result Significant improvement in reinforcement learning performance.
GENESIS-V2 infers unordered object representations without iterative refinement.
problem Unsupervised learning of unordered object representations for complex images.
method Stochastic stick-breaking process for clustering pixel embeddings.
result GENESIS-V2 outperforms recent baselines in unsupervised image segmentation and scene generation.
Neural networks favor Boolean functions with low entropy.
problem Understanding the inductive bias of neural networks.
method Analyzing a single-layer perceptron with random initial weights.
result There is a strong intrinsic bias towards low entropy Boolean functions.
Reinforcement learning improves insurance claims reserving by learning from all claim trajectories.
problem Traditional reserving models learn only from settled claims, missing valuable data from ongoing claims.
method Formulated as a Markov decision process, uses reinforcement learning to update OCL estimates sequentially.
result Soft Actor-Critic implementation achieves competitive claim-level accuracy and strong aggregate performance.
The paper develops a new mathematical framework for group-equivariant operators in machine learning.
problem Developing a robust mathematical framework for group-equivariant operators in machine learning.
method The paper introduces group-equivariant non-expansive operators (GENEOs) and studies their topological and metric properties.
result The space of GENEOs is compact and convex, providing fundamental guarantees for machine learning.
einspace expands NAS search space to include diverse neural architectures.
problem NAS results are often limited to existing structures; new designs are rare.
method einspace uses a probabilistic context-free grammar to create a versatile search space.
result einspace discovers novel and improved architectures, including convolutions and attention.
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
Extends hyperparameter transfer across model sizes and modules, improving training speed.
problem Training stability and performance of large-scale models with optimal hyperparameters.
method Complete(d) Parameterisation, per-module hyperparameter optimisation and transfer. result Hyperparameter transfer holds even in the per-module hyperparameter regime, improving training speed.
Bayesian method learns neural network architecture parameters.
problem Estimating optimal neural network architecture parameters.
method Bayesian learning of concrete distributions over layer size and network depth.
result Regular networks with learnt structure generalize better on small datasets, while stochastic networks are more robust to initialisation.
We propose the application of a semi-supervised learning method to improve the performance of acoustic modelling for automatic speech recognition based on deep neural net- works. As opposed to unsupervised initialisation followed by supervised fine tuning, our method takes advantage of both unlabelled and labelled data…
Study combines variational inference and transformers for seasonal climate predictions.
problem Lack of robust seasonal predictions due to limited historical records and computational constraints.
method Combines variational inference with transformer models trained on climate model output.
result Method provides skilful predictions beyond climate change-induced trends in various regions.
Study finds optimal learning rate schedules for sub-100M quantization-aware training across bit-widths.
problem Optimal learning rate schedules for quantization-aware training depend on bit-width.
method Factorial grid testing over bit-width, warmdown fraction, LR magnitude, model size, and seed.
result INT6 QAT requires a different schedule than higher-precision training, falsifying the primary hypothesis.
Transformers interpreted as probabilistic Laplacian Eigenmaps steps.
problem Improving transformer performance through probabilistic interpretation.
method Probabilistic Laplacian Eigenmaps model derivation and graph diffusion step.
result Subtracting identity from attention matrix improves transformer performance.