arXiv research

Proposes Deep LTMLE for estimating dynamic treatment effects in longitudinal studies.

Study excess risk in statistical inference with transformations.

The MoN loss fails to accurately represent ground truth probability density functions in probabilistic trajectory prediction.

This paper describes an empirical study of shortfall optimization with Barra Extreme Risk. We compare minimum shortfall to minimum variance portfolios in the US, UK, and Japanese equity markets using Barra Style Factors (Value, Growth, Momentum, etc.). We show that minimizing shortfall generally improves performance ov…

Hill-ADAM optimizes loss landscapes by exploring state space deterministically.

Optimization algorithms help overparameterized neural networks achieve high performance.

In this work, we introduce the {\em average top-$k$} (\atk) loss as a new aggregate loss for supervised learning, which is the average over the $k$ largest individual losses over a training dataset. We show that the \atk loss is a natural generalization of the two widely used aggregate losses, namely the average loss a…

Proposes a method to solve deep neural networks' local minimum problem.

Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect. Particularly, the properties of critical points and the landscape around them are of importance to determine …

A new method optimizes neural sequence models for better task performance.

Optimizes minimum-volume prediction sets for multivariate regression.

Data discretization is an important step in the process of machine learning, since it is easier for classifiers to deal with discrete attributes rather than continuous attributes. Over the years, several methods of performing discretization such as Boolean Reasoning, Equal Frequency Binning, Entropy have been proposed,…

The L1 loss landscape of neural nets near local minima behaves differently, revealing exponential decay and increased vertex density.

Develops a neural network for global minimum variance portfolio optimization.

DNNs with $L_{2}$ regularization reveal feature learning dynamics and sparsity.

AdaLoss optimizes adaptive learning rates for efficient convergence in various models.

Minimum-norm solutions generalize well in over-parametrized neural networks.

DiMS sampler explores neural network loss minima via dissipative dynamics.

This work proposes a mathematical framework for loss landscapes and optimization in deep neural networks.

Large learning rates improve neural network generalization, study shows.

We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …

In this paper, we propose a robust sequential learning strategy for training large-scale Recommender Systems (RS) over implicit feedback mainly in the form of clicks. Our approach relies on the minimization of a pairwise ranking loss over blocks of consecutive items constituted by a sequence of non-clicked items follow…

In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning…

For nonconvex optimization in machine learning, this article proves that every local minimum achieves the globally optimal value of the perturbable gradient basis model at any differentiable point. As a result, nonconvex machine learning is theoretically as supported as convex machine learning with a handcrafted basis …

SGD's escape rate depends on log loss barrier, not linear loss barrier.

Gradient flow in parameters equals linear interpolation in outputs.

Study on the optimization of neural networks with ReLU activation and the degeneracy of their parametrizations.

Study shows deep linear networks can converge to flatter minima at large learning rates.

Paper improves regret bounds for distributed experts problem.

Defines MER for Bayesian learning, a gap between achievable and optimal performance.

This paper proves SGD converges to global minimum for over-parameterized ReLU networks.

SIFT reduces training time by selecting samples with approximate losses.

The paper uses optimal transport to find the minimum loss in adversarial classification.

In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking,…

Over-parameterization makes optimization easier for simple neural networks, even with minor extra neurons.

A new pruning method reduces neural network computation without retraining.

The minimum number of colors is a challenging knot invariant since, by definition, its calculation requires taking the minimum over infinitely many minima. In this article we estimate and in some cases calculate the minimum number of colors for the Turk's head knots on three strands.

Optimizes private learning with differential privacy for LASSO problems.

A new optimizer DDC improves deep learning models by respecting symmetries.

One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection…

SGD noise helps select flat minima by concentrating in sharp directions and being proportional to loss value.

Optimal coupling among random vectors with known statistics and correlation structure found using minimum spanning tree over measure-valued vertices.

New methods evaluate data representations by complexity of low-loss predictor learning.

Even knots with more than 30 crossings are not fertile.

Reconstruction of a function from noisy data is often formulated as a regularized optimization problem over an infinite-dimensional reproducing kernel Hilbert space (RKHS). The solution describes the observed data and has a small RKHS norm. When the data fit is measured using a quadratic loss, this estimator has a know…

We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network (parameterized as $x \mapsto W_N W_{N-1} \cdots W_1 x$) by minimizing the $\ell_2$ loss over whitened data. Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are…

New framework for DNN training guarantees convergence to global minimum.

Two types of 4-manifolds disagree on minimum Euler characteristic.