Paper develops exact convex optimization for neural networks with polynomial activations.
problem Training two-layer neural networks with nonlinear polynomial activations.
method Exact convex optimization using semidefinite programming.
result Global optimization of neural networks is polynomial-time computable.
Interesting theoretical associations have been established by recent papers between the fields of active learning and stochastic convex optimization due to the common role of feedback in sequential querying mechanisms. In this paper, we continue this thread in two parts by exploiting these relations for the first time …
Convex neural networks enforce convex constraints on weights and activations, improving generalization.
problem Improving generalization and reducing overfitting in neural networks.
method Enforce convex constraints on weights and activations, using non-negative weights and non-decreasing convex activation functions.
result Convex neural networks self-regularize, outperforming base architectures and achieving similar performance to convolutional architectures.
New method trains neural networks with threshold activation functions efficiently.
problem Training neural networks with threshold activation functions is challenging due to zero gradients.
method We study weight decay regularized training problems of deep neural networks with threshold activations, showing they can be formulated as convex optimization problems.
result Regularized deep threshold network training problems can be formulated as standard convex optimization problems, paralleling the LASSO method.
Polynomial-time convex optimization for CNNs with ReLU activations.
problem Training Convolutional Neural Networks (CNNs) with ReLU activations.
method Developed a convex analytic framework using semi-infinite duality to formulate equivalent convex optimization problems for CNN architectures.
result Proved that two-layer CNNs can be globally optimized via an ℓ2 norm regularized convex program. Study efficient active learning for halfspaces with Tsybakov noise using non-convex optimization.
problem Efficiently learn halfspaces with Tsybakov noise under structured unlabeled data.
method Non-convex optimization approach to find approximate first-order stationary points.
result Designs an algorithm with improved label complexity compared to previous methods.
Active learning refers to the learning protocol where the learner is allowed to choose a subset of instances for labeling. Previous studies have shown that, compared with passive learning, active learning is able to reduce the label complexity exponentially if the data are linearly separable or satisfy the Tsybakov noi…
We consider the problem of learning convex aggregation of models, that is as good as the best convex aggregation, for the binary classification problem. Working in the stream based active learning setting, where the active learner has to make a decision on-the-fly, if it wants to query for the label of the point curren…
New algorithms solve large-scale convex regression problems.
problem Large-scale convex regression with subgradient regularization.
method Active set type algorithm on dual QP, approximate optimization, randomized augmentation.
result Solves problems with n=10^5 and d=10 in minutes.
Active-set algorithm improves Cox regression for shape-restricted covariates.
problem Improving Cox regression for shape-restricted covariates.
method Shape-restricted inference using active-set optimization for spline basis expansion.
result Active-set algorithm produces accurate linear covariate effect estimates.
AIF reformulated as convex MDP for adaptive behavior.
problem Adaptive behavior and policy optimization.
method Formulating AIF as convex MDP, deriving mirror descent algorithm.
result EFE minimization in AIF is equivalent to reward maximization in latent MDP, with epistemic component.
We consider the optimization of active extension portfolios. For this purpose, the optimization problem is rewritten as a stochastic programming model and solved using a clever multi-start local search heuristic, which turns out to provide stable solutions. The heuristic solutions are compared to optimization results o…
Proposes active sampling for improving fairness in machine learning.
problem Improving fairness in machine learning models, especially for disadvantaged groups.
method Simple active sampling and reweighting strategies for min-max fairness.
result Proves the rate of convergence to a min-max fair solution for convex problems.
New active learning framework for multiclass classification beyond realizability assumption.
problem Active learning in non-realizable settings with convex model classes.
method Surrogate risk minimization, epoch-based fitting, aggregation of models.
result Achieves label and sample complexity comparable to prior work in non-realizable settings.
Deep neural nets on 1-D data are convex Lasso models with reflection features.
problem Training neural networks on 1-D data.
method Proving equivalence to convex Lasso problems with discrete, explicitly defined dictionary matrices.
result Reflection features in neural networks with certain activations.
This work shows MLPs can approximate monotonic functions without bounded activations.
problem Optimizing MLPs with monotonic constraints and bounded activations.
method Generalized theoretical results showing MLPs with non-negative weights and saturating activations are universal approximators.
result MLPs with non-negative weights and saturating activations are universal approximators for monotonic functions.
Optimizes routing in decentralized exchanges with gas fees.
problem Routing in decentralized exchanges with fixed gas fees.
method General optimization framework with mixed-integer model, incorporating gas fees.
result Explicit Karush-Kuhn-Tucker system linking prices, fees, and activation.
Improved graph-based semi-supervised learning with model change active learning.
problem Identifying which unlabelled data points to label to best improve classifier performance.
method Pairing model change active learning with graph-based semi-supervised learning methods.
result Improved multiclass classification performance over prior methods.
Sharp lower bounds on shallow neural networks' approximation rates are derived.
problem The efficiency of shallow neural networks in approximating functions.
method Lower bounding the L2-metric entropy and Kolmogorov n-widths of the convex hull of neural network basis functions. result Sharp lower bounds on the approximation rates for shallow neural networks are provided.
This paper develops convex surrogates for optimizing the multi-label F-measure.
problem Optimizing the F-measure for multi-label classification is computationally hard.
method Designing convex surrogate losses calibrated for the F-measure.
result The F-measure for multi-label problems has a rank of at most s2+1. Wasserstein GANs are shown to have hidden convexity, enabling exact solutions with convex optimization.
problem Non-convex and non-concave optimization in GANs.
method Convex duality analysis of Wasserstein GANs with two-layer neural network discriminators.
result Wasserstein GANs can be solved exactly with convex optimization under certain conditions.
The paper explores transferability of adversarial examples between convex and 01 loss models, finding non-transferability due to different decision boundaries caused by outliers.
problem Transferability of adversarial examples between convex and 01 loss models.
method Empirical study of transferability between linear 01 loss and convex (hinge) loss models, and between neural networks with different activation functions.
result Adversarial examples are non-transferable between convex and 01 loss models due to different decision boundaries caused by outliers.
This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width wmin(d) so that ReLU nets of width wmin(d) (and arbitrary depth) can approximate any continuous functio…
A new framework for verifying robustness of neural networks.
problem Verifying the robustness of neural networks against adversarial attacks.
method LayerCert framework exploiting the nested hyperplane arrangement structure of ReLU networks.
result LayerCert reduces the number and size of convex programs needed for robustness verification.
Proximal methods avoid local minima in weakly convex problems.
problem Weakly convex optimization problems with strict saddle properties.
method Proximal methods on nonsmooth functions with strict saddle guarantees.
result Proximal methods converge to local minimizers only, when initialized randomly.
We propose convex relaxations for convolutional neural nets with one hidden layer where the output weights are fixed. For convex activation functions such as rectified linear units, the relaxations are convex second order cone programs which can be solved very efficiently. We prove that the relaxation recovers the glob…
SGD avoids critical points on weakly convex functions.
problem Non-convergence of SGD to critical points on specific manifolds.
method Stochastic subgradient descent, Verdier stratification, angle condition.
result SGD converges to local minimizers on weakly convex functions.
Source imaging based on magnetoencephalography (MEG) and electroencephalography (EEG) allows for the non-invasive analysis of brain activity with high temporal and good spatial resolution. As the bioelectromagnetic inverse problem is ill-posed, constraints are required. For the analysis of evoked brain activity, spatia…
TUSLA algorithm solves non-convex optimization problems with ReLU activations.
problem Non-convex stochastic optimization with super-linearly growing and discontinuous gradients.
method Non-asymptotic analysis of TUSLA algorithm for non-convex learning.
result TUSLA provides non-asymptotic error bounds in Wasserstein distances for non-convex learning.
Unified framework for clustering with sparse convex combinations.
problem Challenges in subspace clustering with limited labelled data.
method Spectral-based sparse subspace representation with extensions to constrained and active learning.
result Effective and competitive clustering results on simulated and real data.
We solve ReLU regression with efficient approximations for various distributions.
problem Finding the best fitting ReLU function with square loss from unknown distributions.
method Introduced efficient constant-factor approximation algorithm and polynomial-time approximation scheme.
result First constant-factor approximation algorithm for ReLU regression with weak concentration conditions.
Several recently proposed architectures of neural networks such as ResNeXt, Inception, Xception, SqueezeNet and Wide ResNet are based on the designing idea of having multiple branches and have demonstrated improved performance in many applications. We show that one cause for such success is due to the fact that the mul…
In this paper, we consider regression problems with one-hidden-layer neural networks (1NNs). We distill some properties of activation functions that lead to local strong convexity in the neighborhood of the ground-truth parameters for the 1NN squared-loss objective. Most popular nonlinear activation function…
In this paper, we investigate the geometric structure of activation spaces of fully connected layers in neural networks and then show applications of this study. We propose an efficient approximation algorithm to characterize the convex hull of massive points in high dimensional space. Based on this new algorithm, four…
AGGLIO optimizes non-convex functions with local convexity guarantees.
problem Optimizing non-convex functions with local convexity.
method Stage-wise, graduated optimization technique for locally convex functions.
result Global convergence to the global optimum for non-convex and locally convex objectives.
We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…
The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes o…
Stochastic subgradient descent avoids critical points in definable functions.
problem Finding local minima in definable functions.
method Stochastic subgradient descent with density-like perturbation.
result SGD converges to a local minimum in definable functions.
New binary AA methods improve on existing techniques.
problem Binary data limitations in AA methods.
method Proposed two optimization frameworks for binary AA.
result Superior performance on synthetic and real binary data.
We present a novel factor analysis method that can be applied to the discovery of common factors shared among trajectories in multivariate time series data. These factors satisfy a precedence-ordering property: certain factors are recruited only after some other factors are activated. Precedence-ordering arise in appli…
Information Cascades Model captures dynamical properties of user activity in a social network. In this work, we develop a novel framework for activity shaping under the Continuous-Time Information Cascades Model which allows the administrator for local control actions by allocating targeted resources that can alter the…
We develop a variant of multiclass logistic regression that is significantly more robust to noise. The algorithm has one weight vector per class and the surrogate loss is a function of the linear activations (one per class). The surrogate loss of an example with linear activation vector a and class c has t…
We describe a novel family of models of multi- layer feedforward neural networks in which the activation functions are encoded via penalties in the training problem. Our approach is based on representing a non-decreasing activation function as the argmin of an appropriate convex optimiza- tion problem. The new framewor…
Paper reveals hidden convexities in deep learning models using sparse signal processing.
problem Non-convex loss functions in deep learning models complicate optimization and theoretical understanding.
method Developed convex equivalences of ReLU NNs and their connections to sparse signal processing models.
result Recent research has uncovered hidden convexities in certain NN architectures, notably two-layer ReLU networks and other architectures.
New algorithms recover clusters with minimal queries, connecting margins to recoverability.
problem Active cluster recovery with oracle queries for minimal cost.
method Introducing margin-based clustering, designing algorithms for various spaces.
result Achieve O(logn) queries for general pseudometric spaces and convex clusters. Bounds on chemical reaction network relaxation rates using convex analysis.
problem Understanding relaxation dynamics in chemical reaction networks.
method Convex analysis, generalized gradient flows, singular values of stoichiometric matrix.
result Bounds on Kullback-Leibler divergence to equilibrium for CRNs.
Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.
problem Understanding learnability in overparameterized quadratic neural networks.
method Mapping ERM to convex matrix sensing with nuclear norm penalization.
result Characterization of global minima and precise generalization thresholds.
Based on a new atomic norm, we propose a new convex formulation for sparse matrix factorization problems in which the number of nonzero elements of the factors is assumed fixed and known. The formulation counts sparse PCA with multiple factors, subspace clustering and low-rank sparse bilinear regression as potential ap…