GANs improve stochastic parameterization of the Lorenz '96 model.
problem Improving stochastic parameterizations for sub-grid processes.
method Developed a GAN-based stochastic parameterization for the Lorenz '96 model.
result GAN configurations outperform a bespoke parameterization in skillful forecasts and climate simulations.
New parameterization tackles stochasticity in weather models.
problem Uncertainty in small-scale processes in weather models.
method Bayesian neural network with Hamiltonian Monte Carlo for uncertainty quantification and memory.
result Shows skillful forecasts and trustworthy uncertainty quantifications.
Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.
problem Stochastic optimization instability in ReLU networks impedes convergence and generalization.
method Characteristic activation boundaries analysis and Geometric Parameterization (GmP) technique.
result GmP resolves instability, leading to better optimization, convergence, and generalization.
Traditionally, neural networks are parameterized using optimization procedures such as stochastic gradient descent, RMSProp and ADAM. These procedures tend to drive the parameters of the network toward a local minimum. In this article, we employ alternative "sampling" algorithms (referred to here as "thermodynamic para…
New bounds show BBVI's gradient variance matches SGD conditions, improving parameterization efficiency.
problem Understanding and improving the convergence of black-box variational inference (BBVI).
method Showed BBVI satisfies matching gradient variance bounds corresponding to the ABC condition for smooth and quadratically-growing log-likelihoods.
result Proven BBVI's gradient variance matches SGD conditions, with superior dimensional dependence for mean-field parameterization.
New method to recover over-parameterized models corrupted during estimation.
problem Recovering statistical models corrupted after initial estimation.
method Robust estimation using over-parameterized models and redundancy.
result Stochastic gradient descent is well-suited for model repair, but sparsity is generally not repairable.
The sampling efficiency of MCMC methods in Bayesian inference for stochastic volatility (SV) models is known to highly depend on the actual parameter values, and the effectiveness of samplers based on different parameterizations varies significantly. We derive novel algorithms for the centered and the non-centered para…
New research shows deep ReLU networks can be learned with polylogarithmic width.
problem Learning deep ReLU networks with limited over-parameterization.
method Using gradient descent, the study establishes learning guarantees for networks with polylogarithmic width.
result Deep ReLU networks can be learned with a polylogarithmic width condition, not just a high degree polynomial.
New algorithms improve binary neural network configurations.
problem Training binary neural networks efficiently.
method Stochastic message passing algorithms (BP and SP) for discrete inference.
result Stochastic BP and SP find better BNN configurations.
Stochastic gradient methods converge for training wide PINNs.
problem Convergence of stochastic gradient descent in training over-parameterized PINNs.
method Established linear convergence of stochastic gradient descent/flow in training over-parameterized two-layer PINNs.
result Linear convergence with high probability for general activation functions.
Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different k…
Adaptive gradient methods converge faster with over-parameterization and line-search.
problem Training over-parameterized models using adaptive gradient methods.
method Simplified setting of smooth, convex losses with over-parameterized models, proving convergence rates and demonstrating improvements with line-search techniques.
result Adaptive gradient methods, particularly AMSGrad, converge faster with line-search techniques.
The paper analyzes how noise geometry influences the performance of SGD in machine learning.
problem Understanding how noise geometry affects the performance of stochastic gradient descent.
method Developed two metrics to quantify noise alignment strength and analyzed their effects on loss and subspace projection dynamics.
result Noise geometry can be used to guarantee alignment under certain conditions, aiding SGD's ability to escape from sharp minima.
OPT framework improves neural network generalization by learning an orthogonal transformation.
problem Improving neural network generalization.
method Orthogonal over-parameterized training (OPT) framework that minimizes hyperspherical energy.
result OPT framework provably minimizes hyperspherical energy and improves empirical generalization.
We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both…
Proposes a new adaptive learning rate for SGD.
problem Finding an efficient learning rate for SGD.
method Stochastic Polyak step-size (SPS).
result SGD with SPS converges faster for over-parameterized models.
We explore Deep Reinforcement Learning in a parameterized action space. Specifically, we investigate how to achieve sample-efficient end-to-end training in these tasks. We propose a new compact architecture for the tasks where the parameter policy is conditioned on the output of the discrete action policy. We also prop…
Geometric Occam's Razor shapes deep learning solutions.
problem Understanding the regularization in over-parameterized neural networks.
method Analyzing the geometric model complexity and Dirichlet energy in neural networks.
result Over-parameterized neural networks are implicitly regularized by geometric model complexity.
SGD converges to optimal solution in perfect data fitting problem.
problem Noiseless least-squares with perfect data fitting.
method Last iterate of SGD with constant step-size.
result Convergence of SGD final iterate for non-strongly convex problem.
Compact parameterization improves Bayesian neural network performance.
problem Improving performance of Bayesian neural networks using variational methods.
method Restricting variational distribution to a k-tied Normal distribution with low-rank factorization.
result Compact parameterization improves signal-to-noise ratio and convergence speed.
Hierarchical Bayesian networks and neural networks with stochastic hidden units are commonly perceived as two separate types of models. We show that either of these types of models can often be transformed into an instance of the other, by switching between centered and differentiable non-centered parameterizations of …
FP-UCB algorithm achieves bounded regret for finitely parameterized multi-armed bandits.
problem Finitely parameterized multi-armed bandits with unknown but known parameter set.
method FP-UCB algorithm using structural information about the parameter set.
result FP-UCB achieves bounded regret under structural condition, logarithmic otherwise.
A new framework models uncertainty in structured temporal data using SDEs and neural networks.
problem Uncertainty quantification in machine learning applications involving structured and temporal data.
method Integrates stochastic differential equations (SDEs) with deep generative models in a variational autoencoder framework.
result Improves uncertainty quantification in machine learning applications involving structured and temporal data.
SGLBO optimizes quantum circuits with fewer measurements, improving accuracy and noise resilience.
problem Efficiently optimizing parameterized quantum circuits with reduced measurement shots and noise.
method Developed SGLBO combining SGD and BO, with adaptive measurement-shot strategy and suffix averaging.
result Significantly reduces measurement-shot cost while improving accuracy and noise resilience.
New algorithms improve SGD convergence and reduce variance for over-parameterized models.
problem Slower convergence in non-interpolation settings for SGD variants.
method Proposed AdaSPS and AdaSLS with variance reduction for robust convergence.
result Achieves faster convergence rates and robustness in non-interpolation settings.
Over-parameterized neural networks generalize well in practice without any explicit regularization. Although it has not been proven yet, empirical evidence suggests that implicit regularization plays a crucial role in deep learning and prevents the network from overfitting. In this work, we introduce the gradient gap d…
We describe the sample paths of the stochastic field F=Ft(v,x,q) of aggregate utilities parameterized by Pareto weights v and total cash amounts x and stocks' quantities q in an economy. We also describe the sample paths of the stochastic field G=Gt(u,y,q), which is conjugate to F with respect to the …
This paper organizes recent deep learning theory advances.
problem Lack of theoretical foundations in deep learning.
method Literature review and categorization into six groups.
result Organized recent advances in deep learning theory.
This work proposes a mathematical framework for loss landscapes and optimization in deep neural networks.
problem The effectiveness of gradient-based optimization in over-parameterized neural networks.
method A modern view and mathematical framework of loss landscapes and efficient optimization in over-parameterized machine learning models.
result Wide neural networks satisfy the PL∗ condition, explaining (S)GD convergence to a global minimum. Improves conditions for mode connectivity in deep neural networks.
problem Ensuring mode connectivity in deep neural networks under practical conditions.
method Exploiting feature quality and mild over-parameterization conditions.
result Connectivity of solutions found by stochastic gradient descent confirmed under new conditions.
New bounds enable training of probabilistic models for deep networks.
problem Training scalable latent variable models for deep networks.
method Introducing new variational bounds for specific output layers of neural networks.
result Analytical bounds for certain output layers allow training without re-parameterization or Monte Carlo approximations.
Introduces a neural network-based method for efficient state and parameter estimation in complex systems.
problem Efficiently estimating state paths and parameters from noisy measurements in high-dimensional nonlinear systems.
method Bayesian Information Field Theory with neural network parameterization and optimization algorithms.
result Proposes a method to simplify and enrich state path parameterizations using neural networks, improving inference accuracy.
A recent line of research has shown that gradient-based algorithms with random initialization can converge to the global minima of the training loss for over-parameterized (i.e., sufficiently wide) deep neural networks. However, the condition on the width of the neural network to ensure the global convergence is very s…
Neural networks parameterize time-varying Markov dynamics in financial time series.
problem Estimating Markov transition matrices in high-resolution, high-noise financial data.
method Introduces a neural network framework to generate explicit, time-varying Markov transition matrices, constraining neural outputs to formal stochastic operators.
result Learned operators capture regime shifts, with high-volatility regimes homogenizing transition dynamics.
The recently proposed option-critic architecture Bacon et al. provide a stochastic policy gradient approach to hierarchical reinforcement learning. Specifically, they provide a way to estimate the gradient of the expected discounted return with respect to parameters that define a finite number of temporally extended ac…
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
SGD with over-param. makes neural net landscape connected, aiding optimization.
problem Spurious local minima and disconnected landscape in neural networks optimization.
method Stochastic Gradient Descent (SGD) with over-parameterization.
result SGD solutions are connected via a piecewise linear path, making the landscape approximately connected.
Improves understanding of stochastic NGVI convergence rates.
problem Lack of knowledge about non-asymptotic convergence rates in stochastic NGVI.
method Proved non-asymptotic convergence rates for conjugate likelihoods and showed implicit optimization for non-conjugate likelihoods.
result First O(T1) non-asymptotic convergence rate for stochastic NGVI in conjugate likelihoods. Neural Chaos uses neural networks instead of polynomials for stochastic modeling.
problem Challenges in constructing surrogate models with uncertainty quantification for complex or high-dimensional stochastic processes.
method Adopting spectral expansion formalism with neural network basis functions, identifying them data-drivenly without prior assumptions.
result Demonstrates effectiveness of the proposed scheme through numerical examples of varying complexity.
We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, a…
NOVAS uses adaptive stochastic search for non-convex optimization in deep networks.
problem Non-convex optimization challenges in deep neural networks.
method Adaptive stochastic search for non-convex optimization.
result NOVAS outperforms existing alternatives in a structured prediction task.
Study introduces a probabilistic framework for air-sea fluxes using neural networks.
problem Accurately quantifying air-sea fluxes for understanding interactions and improving weather/climate models.
method Gaussian distributions conditioned on input variables, artificial neural networks, eddy-covariance data, minimizing negative log-likelihood loss.
result Trained neural networks provide alternative mean flux estimates and quantify uncertainty.
Neural SDEs reduce variance in stochastic simulations.
problem Efficiency of Monte Carlo simulations in finance.
method Use neural SDEs with control variates parameterized by neural networks.
result Prove optimality conditions for variance reduction in SDEs with infinite activity.
We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i.e., number of hidden nodes per layer) is much larger than the number of training data points. We show that, the expected 0-1 loss of a wide enough ReLU network trained with stochastic…
Stochastic descent methods (of the gradient and mirror varieties) have become increasingly popular in optimization. In fact, it is now widely recognized that the success of deep learning is not only due to the special deep architecture of the models, but also due to the behavior of the stochastic descent methods used, …
This paper is devoted to the application of B-splines to volatility modeling, specifically the calibration of the leverage function in stochastic local volatility models and the parameterization of an arbitrage-free implied volatility surface calibrated to sparse option data. We use an extension of classical B-splines …
Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner t…
Estimates and optimizes UBSR risk in recursive settings.
problem Estimating and optimizing UBSR risk in a recursive setting with one-at-a-time samples.
method Casts UBSR as a root finding problem, uses stochastic approximation and gradient descent.
result Derives non-asymptotic bounds on estimation and optimization errors.