Variational Laplace improves Bayesian neural network performance without sampling.
problem Improving Bayesian neural network performance and calibration.
method Develops a new variational Laplace method for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms standard VI methods in test performance and calibration.
Variational Laplace improves Bayesian neural networks performance.
problem Improving Bayesian neural networks performance.
method Develops variational Laplace for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms other inference methods.
Our study analyzes how neural network initialization affects privacy and utility in overparameterized models.
problem Privacy and utility trade-off in overparameterized neural networks.
method Analytical proof of KL divergence privacy bound, focusing on initialization, width, and depth.
result Privacy bound improvement with increasing depth under certain initializations, degradation under others.
MaxVA improves Adam's step sizes by maximizing gradient variance.
problem Unstable or extreme adaptive learning rates in Adam.
method Maximizing the variance of gradient coordinates in Adam's running mean of squared gradients.
result Faster adaptation and more desirable convergence behaviors than Adam.
Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We…
Stochastic structured prediction under bandit feedback follows a learning protocol where on each of a sequence of iterations, the learner receives an input, predicts an output structure, and receives partial feedback in form of a task loss evaluation of the predicted structure. We present applications of this learning …
We formulate the problem of neural network optimization as Bayesian filtering, where the observations are the backpropagated gradients. While neural network optimization has previously been studied using natural gradient methods which are closely related to Bayesian inference, they were unable to recover standard optim…
GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.
problem Fixed learning rate in AdaGrad leads to step size decrease over time.
method Introduces GradaGrad, which grows or shrinks the learning rate based on a different accumulation in the denominator.
result GradaGrad achieves similar convergence rates as AdaGrad and demonstrates non-monotone adaptation.
Paper introduces a new G⋆ regret measure for online convex optimization with smooth losses.
problem Online convex optimization with smooth losses.
method Introduces a new G⋆ regret measure that depends on the cumulative squared gradient norm. result The G⋆ regret can be arbitrarily sharper than existing measures when losses have vanishing curvature. DoWG optimizer automatically adapts to convex and nonsmooth problems without tuning.
problem Optimizing machine learning models efficiently and adaptively.
method DoWG uses a distance-based weighted version of gradient averaging for optimization.
result DoWG achieves convergence rates similar to optimally tuned gradient descent.
Extends phase retrieval methods to handle sensing vector errors.
problem Phase retrieval with errors in sensing vectors.
method Total Least Squares (TLS) framework applied to gradient descent.
result Gradient descent can efficiently solve TLS phase retrieval.
This paper improves Adam's performance in machine learning tasks.
problem Improving the generalization ability of adaptive gradient methods.
method Develops a control theoretic framework to propose AdamSSM, a new variant of Adam.
result AdamSSM improves generalization accuracy and convergence compared to recent adaptive gradient methods.
This study analyzes AdaGrad's stability and convergence in non-convex optimization.
problem Lack of theoretical analysis for AdaGrad in non-convex optimization.
method Novel stopping time-based techniques from probability theory.
result Established stability and derived convergence rates for AdaGrad.
Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Po…
The paper accelerates ISTA and FISTA algorithms for composite optimization problems.
problem Improving convergence rates of ISTA and FISTA for composite optimization.
method Improved proximal subgradient norm minimization using Lyapunov function.
result Convergence rates of ISTA and FISTA are accelerated.
New methods improve Laplace approximations for deep neural networks by selecting key parameters.
problem Improving uncertainty quantification in deep neural networks using computationally feasible approximations.
method Gradient-Laplace and Greedy-Laplace methods for selecting parameters in sub-network Laplace approximations.
result Gradient-Laplace method outperforms existing heuristic approaches and provides formal optimality guarantees.
CAdam optimizes online learning by adapting to distribution shifts and noise.
problem Challenges in online learning data, including distribution shifts and noise, affect Adam's performance.
method CAdam uses a confidence-based approach to assess the consistency between momentum and gradients before updating parameters.
result CAdam outperforms other optimizers in various settings with distribution shift or noise.
AIR-Net adapts low-rank regularization dynamically for better image completion.
problem Fixed low-rank regularization limits adaptability to different images.
method AIR-Net uses adaptive and implicit regularization parameterized by a dynamic Laplacian matrix.
result AIR-Net enhances implicit regularization and outperforms fixed methods in non-uniform missing data scenarios.
A 6-regular triangulation for hyperbolic plane created.
problem Creating a 6-regular triangulation for hyperbolic plane.
method Constructed a 6-regular geodesic triangulation.
result A 6-regular geodesic triangulation of the hyperbolic plane was successfully created.
Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.
problem How to optimize deep neural networks without explicit regularization.
method Backward error analysis to calculate implicit gradient regularization and demonstrate its effectiveness empirically.
result Implicit gradient regularization biases gradient descent toward flat minima, improving model robustness and test errors.
Choquet regularization improves exploration in RL.
problem Improving exploration in reinforcement learning.
method Introducing Choquet regularizers to measure and manage exploration, reformulating RL problems and deriving explicit solutions.
result Explicit optimal distributions and Choquet regularizers for various exploratory samplers.
The paper explores optimal regularizers for data sources, linking them to star bodies.
problem Understanding optimal regularizers for data sources.
method Investigates optimal regularizers for data distributions using star bodies and dual Brunn-Minkowski theory.
result Identifies optimal regularizers and assesses amenability to convex regularization.
A triangulation of a connected closed surface is called weakly regular if the action of its automorphism group on its vertices is transitive. A triangulation of a connected closed surface is called degree-regular if each of its vertices have the same degree. Clearly, a weakly regular triangulation is degree-regular. In…
In this article, we describe symplectic and complex toric spaces associated to the five regular convex polyhedra. The regular tetrahedron and the cube are rational and simple, the regular octahedron is not simple, the regular dodecahedron is not rational and the regular icosahedron is neither simple nor rational. We re…
The paper studies convergence rates of Tsallis entropic regularization in optimal transport.
problem Optimal transport with regularization.
method Γ-convergence and quantization/shadow arguments.
result Derives convergence rate of Tsallis entropic regularization.
Study uses property elicitation to understand how fairness regularizers affect optimal decisions.
problem Understanding how fairness regularizers change the optimal decision in predictive algorithms.
method Property elicitation to analyze the relationship between loss, regularization, and optimal decision.
result Necessary and sufficient condition for when a property changes with the addition of a regularizer.
Fiedler regularization uses spectral graph theory to improve neural network performance.
problem Improving neural network performance by penalizing weights based on connectivity.
method Uses the Fiedler value of the neural network's graph as a regularization tool, providing theoretical and computational methods.
result Demonstrates Fiedler regularization's effectiveness in improving neural network performance.
Study on convergence rates for optimal transport with regularization.
problem Convergence analysis of divergence-regularized optimal transport.
method Novel methodology using quantization and martingale couplings.
result Sharp rates for various divergences and transport costs.
We establish continuous maximal regularity results for parabolic differential operators acting on sections of tensor bundles on Riemannian manifolds. As an application, we show that solutions to the Yamabe flow instantaneously regularize and become real analytic in space and time. The regularity result is obtained by i…
Improved optimal regularity for harmonic almost complex structures.
problem Establishing optimal regularity for harmonic almost complex structures.
method Quantitative stratification method and rectifiability of singular strata.
result Optimal regularity theory for energy minimizing harmonic almost complex structures.
Dropout is a simple but effective technique for learning in neural networks and other settings. A sound theoretical understanding of dropout is needed to determine when dropout should be applied and how to use it most effectively. In this paper we continue the exploration of dropout as a regularizer pioneered by Wager,…
Regularization plays an important role in generalization of deep neural networks, which are often prone to overfitting with their numerous parameters. L1 and L2 regularizers are common regularization tools in machine learning with their simplicity and effectiveness. However, we observe that imposing strong L1 or L2 reg…
Study on the regularity of p-Gauss curvature flow near flat interfaces.
problem Regularity of p-Gauss curvature flow near flat interfaces. method Analysis of convex hypersurface near the interface.
result Regularity of the convex hypersurface near the interface.
In this paper, we give a new generalization error bound of Multiple Kernel Learning (MKL) for a general class of regularizations, and discuss what kind of regularization gives a favorable predictive accuracy. Our main target in this paper is dense type regularizations including \ellp-MKL. According to the recent numeri…
In this work we study input gradient regularization of deep neural networks, and demonstrate that such regularization leads to generalization proofs and improved adversarial robustness. The proof of generalization does not overcome the curse of dimensionality, but it is independent of the number of layers in the networ…
We study the evolution equations for a regularized version of Dirac-geodesics, which are the one-dimensional version of Dirac-harmonic maps. We show that for the regularization being sufficiently large, the evolution equations subconverge to a regularized Dirac-geodesic. In the end, we discuss the limiting process of r…
Entropy-regularized NPG converges linearly with linear function approximation.
problem Analyzing convergence of entropy-regularized NPG with function approximation.
method Established finite-time convergence analyses with entropy regularization and linear function approximation.
result Entropy-regularized NPG achieves linear convergence up to a function approximation error.
Regularized linear regression improves binary classification performance, especially with ridge and ℓ1 regularization.
problem Improving binary classification accuracy with noisy labels.
method Systematic study of regularization strengths on linear classifiers trained on noisy binary classification data.
result Ridge regression consistently improves classification error, while ℓ1 regularization can induce sparsity and ℓ∞ regularization can concentrate weights to two values. New algorithm adds Hessian regularization to improve neural network robustness.
problem Improving neural network robustness against adversarial attacks.
method Proposes an efficient algorithm to train neural networks with Hessian operator-norm regularization.
result Hessian operator-norm regularization increases neural network robustness over input gradient regularization.
Wasserstein distributionally robust optimization (DRO) has recently achieved empirical success for various applications in operations research and machine learning, owing partly to its regularization effect. Although connection between Wasserstein DRO and regularization has been established in several settings, existin…
Study slice-regular polynomial functions via twistor space group actions.
problem Characterize slice-regular functions and their polynomial subclasses.
method Employ the twistor construction and group actions of PGL(2,H). result Characterize slice-regular functions with planar twistor lifts and normal classes of polynomials.
Selective state-adaptive regularization improves offline RL performance.
problem Extrapolation errors and value overestimation in static dataset RL.
method State-adaptive regularization coefficients trust Bellman-driven results selectively.
result Significant improvement in performance on D4RL benchmark.
New framework assesses regularization norms in ill-posed problems, revealing L2 instability and proposing adaptive fractional RKHS solutions.
problem Comparative analysis of regularization norms in ill-posed problems.
method Small noise analysis framework for Tikhonov and RKHS regularizations.
result Optimal convergence rates achieved with adaptive fractional RKHS, but hyper-parameters decay too fast.
New iterative regularization method tackles non-smooth, non-strongly convex functionals.
problem Tackles non-smooth, non-strongly convex functionals in regularization problems.
method Primal-dual algorithm with convergence and stability analysis.
result First iterative regularization procedure for non-smooth, non-strongly convex functionals.
This study explores star-shaped regularizers learned from critic-based losses.
problem Understanding the structure of regularizers learned from critic-based losses.
method Optimizing critic-based loss functions over star-shaped regularizers.
result Derives exact expressions for optimal regularizers in certain cases.
Regularizers change the geometric properties of loss functions in neural networks.
problem Understanding how different regularizers affect the geometric properties of loss functions in neural networks.
method Examined several regularizers, including weight decay, to determine if the regularized loss function becomes Morse.
result For certain regularizers, the regularized loss function becomes Morse, indicating a change in geometric properties.
We investigate the learning rate of multiple kernel learning (MKL) with ℓ1 and elastic-net regularizations. The elastic-net regularization is a composition of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total …
Introduces self-regularization for analyzing learning algorithms.
problem Analyzing and optimizing learning algorithms without explicit regularization.
method Develops a self-regularization framework for learning algorithms.
result Provides statistical analysis and minmax-optimal rates for self-regularized algorithms.