Analytical method finds deeper optima in two-layer ReLU networks.
problem Training two-layer ReLU networks with analytical methods.
method Analytically finding critical points of the loss function for one layer while keeping the other fixed.
result Significantly smaller training loss values on real datasets compared to gradient descent methods.
Recent years have witnessed amazing outcomes from "Big Models" trained by "Big Data". Most popular algorithms for model training are iterative. Due to the surging volumes of data, we can usually afford to process only a fraction of the training data in each iteration. Typically, the data are either uniformly sampled or…
New EM algorithm improves deep generative network training.
problem Training deep generative networks with complex posterior and likelihood distributions.
method Derive analytical posterior and marginal distributions using CPA property, derive analytical EM algorithm.
result EM training yields higher likelihood than Variational Autoencoders (VAEs).
GANs analyzed for performance and training issues.
problem Performance and training issues of GANs.
method SDE approximations for training GANs.
result Improved understanding of GANs through analytical perspectives.
Analyzes dynamics of quantum neural networks, predicting exponential decay of training error.
problem Understanding convergence rate of quantum neural networks training.
method Analytic theory for gradient descent dynamics of wide quantum neural networks.
result Simple analytic formula predicts exponential decay of training error.
We present a novel methodology based on a Taylor expansion of the network output for obtaining analytical expressions for the expected value of the network weights and output under stochastic training. Using these analytical expressions the effects of the hyperparameters and the noise variance of the optimization algor…
The combination of the re-parameterization trick with the use of variational auto-encoders has caused a sensation in Bayesian deep learning, allowing the training of realistic generative models of images and has considerably increased our ability to use scalable latent variable models. The re-parameterization trick is …
Paper proposes a new method for learning business process representations.
problem Challenges in capturing all useful information in business process data.
method Combines Gramian Angular Fields and Convolutional Neural Networks for representation learning.
result Demonstrates effectiveness of the approach through visualization and multiple process prediction tasks.
Theory explains creativity in diffusion models generating novel images.
problem Diffusion models generate highly original images far from training data.
method Identified locality and equivariance as inductive biases to prevent optimal score-matching.
result Analytic models predict diffusion model outputs with high accuracy.
Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance. Furthermore, a major hope is that knowledge may transfer across …
We propose a novel Shapley value approach to help address neural networks' interpretability and "vanishing gradient" problems. Our method is based on an accurate analytical approximation to the Shapley value of a neuron with ReLU activation. This analytical approximation admits a linear propagation of relevance across …
Analyzes dropout in neural networks using differential equations.
problem Lack of theoretical understanding of dropout's effectiveness.
method Analytical study of dropout in two-layer neural networks with online stochastic gradient descent.
result Dropout reduces detrimental correlations and optimal probability increases with data noise.
Develops an analytic theory for quantum imaginary time evolution.
problem Lack of a first-principle understanding of quantum imaginary time evolution.
method Interprets QITE as a form of VQA trained with QNGD and connects it to the geometric geodesic distance in the quantum Fisher information metric.
result QITE converges faster than vanilla gradient descent-based VQAs, though the advantage is suppressed by Hilbert space dimensionality.
We address the problem of estimating statistics of hidden units in a neural network using a method of analytic moment propagation. These statistics are useful for approximate whitening of the inputs in front of saturating non-linearities such as a sigmoid function. This is important for initialization of training and f…
Modern predictive analytics underpinned by machine learning techniques has become a key enabler to the automation of data-driven decision making. In the context of business process management, predictive analytics has been applied to making predictions about the future state of an ongoing business process instance, for…
This paper uses SDEs to analyze GANs training and long-run behavior.
problem Understanding the training process and long-run behavior of GANs.
method Established SDE approximations for GANs training and analyzed long-run behavior via invariant measures.
result The long-run behavior of GANs training can be studied via the invariant measures of its SDE approximations.
Tree-based models biased when trained on imbalanced data, requiring new calibration methods.
problem Bias in tree-based models trained on imbalanced datasets.
method Analytical calibration of random forest models, demonstrating bias in decision trees.
result Calibrating tree-based models on imbalanced data negatively impacts predictions, especially for the minority class.
OccamNet finds interpretable symbolic fits to data efficiently.
problem Complex neural models extrapolate poorly and are hard to interpret.
method Samples functions, biases towards better fits, and uses cross-entropy matching.
result Outperforms state-of-the-art symbolic regression methods on real-world datasets.
Study of autocorrelation times in neural MCMC simulations for the 2D Ising model.
problem Estimating autocorrelation times in Neural Markov Chain Monte Carlo simulations.
method Analytical and empirical methods to estimate autocorrelation times, proposing new loss functions and training schemes.
result Proposed new loss functions and training schemes that improve autocorrelation times in neural MCMC simulations.
Neural networks learn modular arithmetic but not all, extending known solutions to generalize.
problem Neural networks struggle with modular arithmetic, especially for polynomials.
method Developed analytical solutions for MLP networks to learn modular addition and multiplication, then combined these solutions to generalize on arbitrary modular polynomials.
result Neural networks can learn and generalize solutions to modular polynomials, supporting the hypothesis that some polynomials are learnable.
A critical decision point when training predictors using multiple studies is whether studies should be combined or treated separately. We compare two multi-study prediction approaches in the presence of potential heterogeneity in predictor-outcome relationships across datasets: 1) merging all of the datasets and traini…
A novel formulation and training procedure for full Boltzmann machines in terms of a mixed binary quadratic feasibility problem is given. As a proof of concept, the theory is analytically and numerically tested on XOR patterns.
Paper presents a method to accurately quantify neural network uncertainty without sampling.
problem Uncertainty quantification in neural networks for reliability and robustness.
method Sample-free moment propagation technique for mean vectors and covariance matrices.
result Analytic solution for covariance of nonlinear activation functions.
Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures. This work demonstrates that dropout introduces two distinct but entangled regularization effects: an explicit effect (also studied in prior work) which occurs since dropout modifies the expected t…
This work considers the problem of binary classification: given training data x1,…,xn from a certain population, together with associated labels y1,…,yn∈{0,1}, determine the best label for an element x not among the training data. More specifically, this work considers a variant o…
Modern machine learning models are typically trained using Stochastic Gradient Descent (SGD) on massively parallel computing resources such as GPUs. Increasing mini-batch size is a simple and direct way to utilize the parallel computing capacity. For small batch an increase in batch size results in the proportional red…
Theory captures feature learning effects in finite CNNs.
problem Feature learning in finite deep neural networks.
method Derive a self-consistent Gaussian Process theory.
result Good agreement with experiments and sharp transition between regimes.
This concept paper highlights a recently opened opportunity for large scale analytical algorithms to be trained directly on edge devices. Such approach is a response to the arising need of processing data generated by natural person (a human being), also known as personal data. Spiking Neural networks are the core meth…
Paper presents a new method for Bayesian deep learning that scales to Atari games.
problem Training neural networks on complex environments like Atari games is challenging.
method Adapted temporal difference Q-learning to work with Bayesian inference.
result TAGI allows for analytical inference of neural network parameters, achieving performance comparable to gradient-based methods.
Self-consistency improves the accuracy of model comparison methods.
problem Improving the accuracy of model comparison methods when simulation models are misspecified.
method Supplement traditional simulation-based training with a self-consistency loss on unlabeled real data.
result Self-consistency training improves model comparison accuracy, especially in open-world scenarios.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Analyzes the generalization and training errors of the random feature model over time.
problem Understanding the temporal behavior of generalization and training errors in deep learning.
method Uses Cauchy complex integral representations and random matrix methods based on linear pencils.
result Analytical solution of the full time-evolution path of generalization and training errors.
Survey on LLMs for time series analytics across various domains.
problem Cross-modality gap between LLMs and time series data.
method Taxonomy of approaches, cross-modality strategies, and experiments on multimodal datasets.
result Effective combinations of textual data and cross-modality strategies enhance time series analytics.
Frequency bias affects neural network training on non-uniform data.
problem Understanding how frequency bias impacts neural networks trained on non-uniformly distributed data.
method Used the Neural Tangent Kernel (NTK) model to explore the effect of variable density on training dynamics.
result Convergence time for learning a pure harmonic function depends on the local density at a point.
Least-squares models such as linear regression and Linear Discriminant Analysis (LDA) are amongst the most popular statistical learning techniques. However, since their computation time increases cubically with the number of features, they are inefficient in high-dimensional neuroimaging datasets. Fortunately, for k-fo…
Quantum neural tangent kernels help understand variational quantum circuits in machine learning.
problem Designing and predicting performance of variational quantum circuits.
method Using quantum neural tangent kernels and dynamical equations for loss functions.
result Analytical solutions for training dynamics in variational quantum circuits.
GAMLA learns manifold structures with auto-encoding for global insights.
problem Limited global insight and lack of interpretable analytical descriptions in manifold learning.
method Two-round auto-encoding process to derive character and complementary representations.
result GAMLA provides global and analytical descriptions of smooth manifolds.
Optimal prototypes found for challenging pathological geometries.
problem Finding optimal prototypes for pathological geometries is challenging.
method Analytical and heuristic algorithms for finding nearly-optimal prototypes.
result Optimal prototypes can be found analytically for challenging geometries.
Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect. Particularly, the properties of critical points and the landscape around them are of importance to determine …
Wide neural networks can learn complex functions like gravitational force law.
problem Learning complex functions like gravitational force law with neural networks.
method Extending theoretical bounds to analytic functions on the sphere using SGD and ReLU networks.
result Wide ReLU networks can learn analytic functions efficiently with proportional number of samples.
Transformers' self-attention mechanism is mapped to a generalized Potts model.
problem Uncertainty in what type of data distribution self-attention can efficiently learn.
method Decouple word positions and embeddings, then show self-attention learns a generalized Potts model.
result Training self-attention is equivalent to solving the inverse Potts problem.
ROI-driven data analytics guides investment in empirical data analysis.
problem Determining the optimal depth and breadth of data analytics.
method Conceptual framework validated through empirical studies focusing on dependency extraction in Mozilla Firefox project.
result ROI-driven data analytics helps avoid over-analyzing empirical data.
Hybrid method improves SABR implied volatility approximation.
problem Improving SABR implied volatility approximation.
method Combining analytical structure with machine learning, using geometric features and residual correction.
result Hybrid model improves accuracy and robustness compared to analytical and neural-network approaches.
This paper explains how batch normalization auto-tunes the regularization parameter based on data statistics.
problem Batch normalization accelerates deep learning training but the exact relationship to regularization is unclear.
method Theoretical analysis and empirical validation of batch normalization's role in auto-tuning the regularization parameter.
result Batch normalization auto-tunes the regularization parameter based on data statistics.
We present analytical exploration of novel activation functions as consequence of integration of several ideas leading to implementation and subsequent use in habitability classification of exoplanets. Neural networks, although a powerful engine in supervised methods, often require expensive tuning efforts for optimize…
Overparameterization aids in model pruning, leading to improved test accuracy.
problem Improving lightweight model performance through pruning.
method Theoretical analysis and high-dimensional asymptotics of model pruning in overparameterized neural networks.
result Even with known informative features, training a large model and then pruning leads to better test accuracy.
Dropout has recently emerged as a powerful and simple method for training neural networks preventing co-adaptation by stochastically omitting neurons. Dropout is currently not grounded in explicit modelling assumptions which so far has precluded its adoption in Bayesian modelling. Using Bayesian entropic reasoning we s…
Wide stochastic networks show Gaussian behavior and improve training with PAC-Bayesian methods.
problem Analyzing and training over-parameterised neural networks with large width.
method Establishing Gaussian behavior for a stochastic architecture, applying PAC-Bayesian training.
result PAC-Bayesian training on large but finite-width networks outperforms standard methods.