arXiv research

Efficient algorithm for contextual bandits with first-order guarantees.

This work interprets GELU and related activations via a first-order loss function.

We consider first order expansions of convex penalized estimators in high-dimensional regression problems with random designs. Our setting includes linear regression and logistic regression as special cases. For a given penalty function $h$ and the corresponding penalized estimator $\hatβ$, we construct a quantity $η$,…

Boosting can efficiently optimize any loss function without requiring first-order information.

DEO uses gradient information to escape saddle points in neural networks.

The paper explores how different loss functions impact reinforcement learning algorithms.

In this work, we study data preconditioning, a well-known and long-existing technique, for boosting the convergence of first-order methods for regularized loss minimization. It is well understood that the condition number of the problem, i.e., the ratio of the Lipschitz constant to the strong convexity modulus, has a h…

New insights into convergence and accuracy trade-offs in federated and meta-learning.

First order discretizations of Langevin diffusion can achieve better generalization error with additional smoothness assumptions.

New study shows diversification can increase risk for heavy-tailed losses.

Theory integrates loss aversion into expected utility for monetary returns.

This paper studies distributed estimation and inference for a general statistical problem with a convex loss that could be non-differentiable. For the purpose of efficient computation, we restrict ourselves to stochastic first-order optimization, which enjoys low per-iteration complexity. To motivate the proposed metho…

New solver for MKL-SVM with 0/1 loss function.

Second-order optimizers retain residual information after data deletion, affecting machine unlearning.

Flexible tree ensemble learning framework supports arbitrary loss functions and multi-task learning.

Piecewise polynomial interpolation-based gradient descent reduces oracle complexity for smooth loss functions.

New inequalities help optimize first-order algorithms for statistical risk analysis.

This paper studies an unsupervised deep learning-based numerical approach for solving partial differential equations (PDEs). The approach makes use of the deep neural network to approximate solutions of PDEs through the compositional construction and employs least-squares functionals as loss functions to determine para…

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural n…

Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned optimizers may similarly outperform current hand-designed optimizers, especially for specific problems. However, learned optimizers are notoriously difficult to…

Paper introduces MKL-$L_{0/1}$-SVM for SVM with $(0, 1)$ loss.

Boosting methods are highly popular and effective supervised learning methods which combine weak learners into a single accurate model with good statistical performance. In this paper, we analyze two well-known boosting methods, AdaBoost and Incremental Forward Stagewise Regression (FS$_\varepsilon$), by establishing t…

Graph Convolution Network (GCN) has been recognized as one of the most effective graph models for semi-supervised learning, but it extracts merely the first-order or few-order neighborhood information through information propagation, which suffers performance drop-off for deeper structure. Existing approaches that deal…

Many modern learning tasks involve fitting nonlinear models to data which are trained in an overparameterized regime where the parameters of the model exceed the size of the training dataset. Due to this overparameterization, the training loss may have infinitely many global minima and it is critical to understand the …

Loss minimisation fails to capture epistemic uncertainty in second-order predictors.

We analyze the semi-hard triplet loss using Edgeworth expansion for better understanding of its behavior.

We consider the problem of online combinatorial optimization under semi-bandit feedback, where a learner has to repeatedly pick actions from a combinatorial decision set in order to minimize the total losses associated with its decisions. After making each decision, the learner observes the losses associated with its a…

Paper proposes a method to efficiently cluster stretched mixtures.

Second-order guarantees for federated learning algorithms.

Paper introduces STSL, a second-order Tweedie sampler for efficient posterior sampling in inverse problems.

Heuristic weighting improves denoising score matching without requiring noise distribution assumptions.

We make three contributions to the theory of k-armed adversarial bandits. First, we prove a first-order bound for a modified variant of the INF strategy by Audibert and Bubeck [2009], without sacrificing worst case optimality or modifying the loss estimators. Second, we provide a variance analysis for algorithms based …

We consider empirical risk minimization of linear predictors with convex loss functions. Such problems can be reformulated as convex-concave saddle point problems, and thus are well suitable for primal-dual first-order algorithms. However, primal-dual algorithms often require explicit strongly convex regularization in …

Gaptron algorithm reduces mistakes in online multiclass classification.

New algorithm finds critical points in non-convex optimization with heavy-tailed gradients.

We address online combinatorial optimization when the player has a prior over the adversary's sequence of losses. In this framework, Russo and Van Roy proposed an information-theoretic analysis of Thompson Sampling based on the information ratio, resulting in optimal worst-case regret bounds. In this paper we introduce…

Online Active Learning (OAL) aims to manage unlabeled datastream by selectively querying the label of data. OAL is applicable to many real-world problems, such as anomaly detection in health-care and finance. In these problems, there are two key challenges: the query budget is often limited; the ratio between classes i…

Second-order methods improve differential privacy in convex optimization.

GD with large, adaptive stepsizes achieves optimal risk in logistic regression.

The permutation symmetry of neurons in each layer of a deep neural network gives rise not only to multiple equivalent global minima of the loss function, but also to first-order saddle points located on the path between the global minima. In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, …

Deep ResNets can achieve zero loss with enough layers and weights.

Paper analyzes adversarial dynamics in neural networks.

Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properti…

We investigate the use of regularized Newton methods with adaptive norms for optimizing neural networks. This approach can be seen as a second-order counterpart of adaptive gradient methods, which we here show to be interpretable as first-order trust region methods with ellipsoidal constraints. In particular, we prove …

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

First order optimization algorithms play a major role in large scale machine learning. A new class of methods, called adaptive algorithms, were recently introduced to adjust iteratively the learning rate for each coordinate. Despite great practical success in deep learning, their behavior and performance on more genera…

A novel dictionary-based approach for predicting functions.

In recent years, an increasing number of neural network models have included derivatives with respect to inputs in their loss functions, resulting in so-called double backpropagation for first-order optimization. However, so far no general description of the involved derivatives exists. Here, we cover a wide array of s…