arXiv research

Improved online learning for hidden-convex losses achieves optimal regret.

New framework identifies hidden risks and optionality in American options.

Gradient Descent Ascent converges to von-Neumann solution in hidden zero-sum games.

Paper reveals hidden convexities in deep learning models using sparse signal processing.

In recent years, spectral clustering has become a standard method for data analysis used in a broad range of applications. In this paper we propose a new class of algorithms for multiway spectral clustering based on optimization of a certain "contrast function" over the unit sphere. These algorithms, partly inspired by…

Optimal hidden-target learning for online inventory optimization on general convex sets.

We analyze deep neural networks using convex duality to reveal hidden layer structures.

The paper explains how simple methods can converge to optimal solutions in complex neural games.

We found hidden convexity in FPCA and developed a faster algorithm.

New algorithm learns HMM parameters on Riemannian manifolds.

New method approximates neural network training for robustness.

Wasserstein GANs are shown to have hidden convexity, enabling exact solutions with convex optimization.

Although deep learning has shown its powerful performance in many applications, the mathematical principles behind neural networks are still mysterious. In this paper, we consider the problem of learning a one-hidden-layer neural network with quadratic activations. We focus on the under-parameterized regime where the n…

New method trains quantized neural networks to global optimality.

A new optimizer for deep learning improves accuracy and reduces training time.

We propose convex relaxations for convolutional neural nets with one hidden layer where the output weights are fixed. For convex activation functions such as rectified linear units, the relaxations are convex second order cone programs which can be solved very efficiently. We prove that the relaxation recovers the glob…

We solve the optimization of two-layer ReLU networks using convex math.

Neural networks can represent complex piecewise functions efficiently.

PredPCA extracts key components for better time series prediction.

DNNs with $L_{2}$ regularization reveal feature learning dynamics and sparsity.

In this paper, we consider regression problems with one-hidden-layer neural networks (1NNs). We distill some properties of activation functions that lead to $\mathit{local~strong~convexity}$ in the neighborhood of the ground-truth parameters for the 1NN squared-loss objective. Most popular nonlinear activation function…

Develops exact convex optimization formulations for neural networks.

Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study a simple minimization method: the unknown measure is discretized into a mixture…

We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned. Our algorithm works for a general class of (potentially overlapping) patches, including commonly used structures for computer vision tasks. Our a…

SNEPPPs use squared neural networks to efficiently model Poisson point processes.

New method shows hidden state can significantly improve differential privacy in SGD.

In this paper we present a model for the hidden Markovian bandit problem with linear rewards. As opposed to current work on Markovian bandits, we do not assume that the state is known to the decision maker before making the decision. Furthermore, we assume structural side information where the decision maker knows in a…

We consider deep linear networks with arbitrary convex differentiable loss. We provide a short and elementary proof of the fact that all local minima are global minima if the hidden layers are either 1) at least as wide as the input layer, or 2) at least as wide as the output layer. This result is the strongest possibl…

CCMM efficiently solves large-scale convex clustering problems.

Generative model initializes 2-layer network weights for small datasets.

Novel analysis of neural networks using geometric algebra and convex optimization.

The paper extends mean field results to three-layer neural networks using SGD.

In this work, we propose an infinite restricted Boltzmann machine~(RBM), whose maximum likelihood estimation~(MLE) corresponds to a constrained convex optimization. We consider the Frank-Wolfe algorithm to solve the program, which provides a sparse solution that can be interpreted as inserting a hidden unit at each ite…

Paper proposes a new clustering model that preserves cluster recovery with fewer dimensions.

Statistical dependencies among wavelet coefficients are commonly represented by graphical models such as hidden Markov trees(HMTs). However, in linear inverse problems such as deconvolution, tomography, and compressed sensing, the presence of a sensing or observation matrix produces a linear mixing of the simple Markov…

Path regularization reveals convex optimization in deep ReLU networks.

Unified framework reveals regularization mechanism in deep ReLU networks via convex optimization.

New lower bound shows bandit convex optimization is harder than linear bandits.

New lower bound shows bandit convex optimization is harder than previously thought.

We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …

This work studies the strong duality of non-convex matrix factorization problems: we show that under certain dual conditions, these problems and its dual have the same optimum. This has been well understood for convex optimization, but little was known for non-convex problems. We propose a novel analytical framework an…

This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.

Convex geometry explains optimal neural network parameters.

Optimization geometrodynamics simplifies adaptive optimizer dynamics.

We study model recovery for data classification, where the training labels are generated from a one-hidden-layer neural network with sigmoid activations, also known as a single-layer feedforward network, and the goal is to recover the weights of the neural network. We consider two network models, the fully-connected ne…

Almost all local minima in neural networks are strongly convex.

Extends HMM to topological spaces for modeling complex data.

Paper develops compact formulations for optimization problems with rank-one convex functions and indicator variables.