arXiv research

Optimizing proper loss yields calibrated models under specific conditions.

Hill-ADAM optimizes loss landscapes by exploring state space deterministically.

Optimizes exp-concave losses with a new risk bound.

Local asymptotic minimax risk bounds in a locally asymptotically mixture of normal family of distributions have been investigated under asymmetric loss functions and the asymptotic distribution of the optimal estimator that attains the bound has been obtained.

We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…

New algorithms achieve optimal DP convex optimization with linear time and gradient computations.

Ridge regression CV loss may have multiple local optima.

We analyze the optimization landscape of α-loss in logistic models.

Gradient-based methods find saddle points, not critical points, in neural networks.

We consider deep linear networks with arbitrary convex differentiable loss. We provide a short and elementary proof of the fact that all local minima are global minima if the hidden layers are either 1) at least as wide as the input layer, or 2) at least as wide as the output layer. This result is the strongest possibl…

Paper develops a new local convexity condition for non-isolated minima in non-convex optimization.

We consider distributed convex optimization problems originated from sample average approximation of stochastic optimization, or empirical risk minimization in machine learning. We assume that each machine in the distributed computing system has access to a local empirical loss function, constructed with i.i.d. data sa…

Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network …

Local update methods' performance depends on learning rates, affecting convergence rates and alignment with true loss.

The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses we introduce variational characterizations that naturally suggest a two-step scheme for their optimizatio…

A new method automatically and dynamically sets learning rates in deep learning.

This paper re-visits the spectral method for learning latent variable models defined in terms of observable operators. We give a new perspective on the method, showing that operators can be recovered by minimizing a loss defined on a finite subset of the domain. A non-convex optimization similar to the spectral method …

New insights into convergence and accuracy trade-offs in federated and meta-learning.

Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.

Proposes neuron alignment to optimize mode connectivity in neural networks.

Analytical method finds deeper optima in two-layer ReLU networks.

In this paper, we propose a general model for plane-based clustering. The general model contains many existing plane-based clustering methods, e.g., k-plane clustering (kPC), proximal plane clustering (PPC), twin support vector clustering (TWSVC) and its extensions. Under this general model, one may obtain an appropria…

We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …

The simplicity of gradient descent (GD) made it the default method for training ever-deeper and complex neural networks. Both loss functions and architectures are often explicitly tuned to be amenable to this basic local optimization. In the context of weakly-supervised CNN segmentation, we demonstrate a well-motivated…

While the optimization problem behind deep neural networks is highly non-convex, it is frequently observed in practice that training deep networks seems possible without getting stuck in suboptimal points. It has been argued that this is the case as all local minima are close to being globally optimal. We show that thi…

Proposes a new contrastive loss for semi-supervised medical image segmentation.

Minimizing a convex risk function is the main step in many basic learning algorithms. We study protocols for convex optimization which provably leak very little about the individual data points that constitute the loss function. Specifically, we consider differentially private algorithms that operate in the local model…

LPL optimizes embeddings to align local neighborhoods, improving cross-lingual word alignment.

Distributed optimization often consists of two updating phases: local optimization and inter-node communication. Conventional approaches require working nodes to communicate with the server every one or few iterations to guarantee convergence. In this paper, we establish a completely different conclusion that each node…

Electronic power inverters are capable of quickly delivering reactive power to maintain customer voltages within operating tolerances and to reduce system losses in distribution grids. This paper proposes a systematic and data-driven approach to determine reactive power inverter output as a function of local measuremen…

Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.

Modified K-means ensures local optimality with same complexity.

Adversarial online nonparametric regression achieves optimal rates with locally adaptive learning.

Study problem-dependent rates in statistical learning theory, achieving optimal generalization error bounds.

We introduce a tunable loss function called $α$-loss, parameterized by $α\in (0,\infty]$, which interpolates between the exponential loss ($α= 1/2$), the log-loss ($α= 1$), and the 0-1 loss ($α= \infty$), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between $…

Reviews recent findings on neural network landscapes.

We introduce novel variants of momentum by incorporating the variance of the stochastic loss function. The variance characterizes the confidence or uncertainty of the local features of the averaged loss surface across the i.i.d. subsets of the training data defined by the mini-batches. We show two applications of the g…

New deep learning method solves complex BSDEs efficiently.

Local convergence theory for mildly over-parameterized neural nets.

This paper proposes an alternative to E2E training for deep networks, reducing memory footprint.

A new algorithm improves federated learning by combining knowledge distillation and weighted combination loss.

Adaptive optimization methods bias neural network trajectories towards regions of lower local geometry.

AutoBayes simplifies variational inference by composing models and optimizing them.

In the following paper we present a new type of optimization algorithms adapted for neural network training. These algorithms are based upon sequential operator splitting technique for some associated dynamical systems. Furthermore, we investigate through numerical simulations the empirical rate of convergence of these…

DNNs with $L_{2}$ regularization reveal feature learning dynamics and sparsity.

SGD generalization bounds derived from information theory.

Area under ROC (AUC) is an important metric for binary classification and bipartite ranking problems. However, it is difficult to directly optimizing AUC as a learning objective, so most existing algorithms are based on optimizing a surrogate loss to AUC. One significant drawback of these surrogate losses is that they …

Proposes an exponentially increasing step-size for faster parameter estimation in statistical models.