arXiv research

The study analyzes convergence rates for sparse pivotal estimators in high-dimensional regression.

New method tackles non-smooth tensor data for better recovery.

New SPS variant improves non-smooth optimization without small gradients.

In this paper, we develop a novel {\bf ho}moto{\bf p}y {\bf s}moothing (HOPS) algorithm for solving a family of non-smooth problems that is composed of a non-smooth term with an explicit max-structure and a smooth term or a simple non-smooth term whose proximal mapping is easy to compute. The best known iteration compl…

In this paper we consider the problem of minimizing composite objective functions consisting of a convex differentiable loss function plus a non-smooth regularization term, such as $L_1$ norm or nuclear norm, under Rényi differential privacy (RDP). To solve the problem, we propose two stochastic alternating direction m…

New method for efficient proximal mapping of 1-path-norm in shallow networks.

New bounds for online portfolio selection without smoothness assumptions.

As one of the most popular linear subspace learning methods, the Linear Discriminant Analysis (LDA) method has been widely studied in machine learning community and applied to many scientific applications. Traditional LDA minimizes the ratio of squared L2-norms, which is sensitive to outliers. In recent research, many …

The Schatten-$p$ norm ($0<p<1$) has been widely used to replace the nuclear norm for better approximating the rank function. However, existing methods are either 1) not scalable for large scale problems due to relying on singular value decomposition (SVD) in every iteration, or 2) specific to some $p$ values, e.g., $1/…

We study decreasing rearrangements of functions defined on (possibly non-smooth) metric measure spaces with Ricci curvature bounded below by $K>0$ and dimension bounded above by $N\in (1,\infty)$ in a synthetic sense, the so called $CD(K,N)$ spaces. We first establish a Polya-Szego type inequality stating that the $W^{…

Algorithm approximates regularization path for deep neural networks efficiently.

The heavy-tailed distributions of corrupted outliers and singular values of all channels in low-level vision have proven effective priors for many applications such as background modeling, photometric stereo and image alignment. And they can be well modeled by a hyper-Laplacian. However, the use of such distributions g…

Many practical applications such as gene expression analysis, multi-task learning, image recognition, signal processing, and medical data analysis pursue a sparse solution for the feature selection purpose and particularly favor the nonzeros \emph{evenly} distributed in different groups. The exclusive sparsity norm has…

Proximal operators are of particular interest in optimization problems dealing with non-smooth objectives because in many practical cases they lead to optimization algorithms whose updates can be computed in closed form or very efficiently. A well-known example is the proximal operator of the vector $\ell_1$ norm, whic…

We consider a class of sparse learning problems in high dimensional feature space regularized by a structured sparsity-inducing norm which incorporates prior knowledge of the group structure of the features. Such problems often pose a considerable challenge to optimization algorithms due to the non-smoothness and non-s…

Many popular statistical models, such as factor and random effects models, give arise a certain type of covariance structures that is a summation of low rank and sparse matrices. This paper introduces a penalized approximation framework to recover such model structures from large covariance matrix estimation. We propos…

This work presents a general framework for solving the low rank and/or sparse matrix minimization problems, which may involve multiple non-smooth terms. The Iteratively Reweighted Least Squares (IRLS) method is a fast solver, which smooths the objective function and minimizes it by alternately updating the variables an…

We present an algorithm for L1-norm kernel PCA and provide a convergence analysis for it. While an optimal solution of L2-norm kernel PCA can be obtained through matrix decomposition, finding that of L1-norm kernel PCA is not trivial due to its non-convexity and non-smoothness. We provide a novel reformulation through …

The problem of joint feature selection across a group of related tasks has applications in many areas including biomedical informatics and computer vision. We consider the l2,1-norm regularized regression model for joint feature selection from multiple tasks, which can be derived in the probabilistic framework by assum…

Gradient descent is a simple and widely used optimization method for machine learning. For homogeneous linear classifiers applied to separable data, gradient descent has been shown to converge to the maximal margin (or equivalently, the minimal norm) solution for various smooth loss functions. The previous theory does …

In this paper, we endow the space of continuous translation invariant valuation on convex sets generated by mixed volumes coupled with a suitable Radon measure on tuples of convex bodies with two appropriate norms. This enables us to construct a continuous extension of the convolution operator on smooth valuations to n…

MARINA-P improves non-smooth federated optimization with adaptive stepsizes.

Extends curve theory to non-smooth data with finite curvature and torsion.

New algorithms delete user data from machine learning models efficiently.

AsylADMM improves gossip-based learning for non-smooth objectives.

New bounds on adaptivity cost in stochastic optimization.

Survey on preserving curvature bounds for non-smooth Ricci flow.

Recent research has shown that performance in signal processing tasks can often be significantly improved by using signal models based on sparse representations, where a signal is approximated using a small number of elements from a fixed dictionary. Unfortunately, inference in this model involves solving non-smooth op…

We introduce non-smooth symplectic forms on manifolds and describe corresponding Poisson structures on the algebra of Colombeau generalized functions. This is achieved by establishing an extension of the classical map of smooth functions to Hamiltonian vector fields to the setting of non-smooth geometry. For mildly sin…

The paper explores various stationarity concepts in non-smooth optimization.

New bounds explain deterministic non-smooth deep nets without large Lipschitz constants.

Automl is the key technology for machine learning problem. Current state of art hyperparameter optimization methods are based on traditional black-box optimization methods like SMBO (SMAC, TPE). The objective function of black-box optimization is non-smooth, or time-consuming to evaluate, or in some way noisy. Recent y…

In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Fo…

Smooth activations enable optimal error rates in neural networks for Sobolev function classes.

We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…

Smoothness analysis of adversarial training reveals $L_\infty$ constraints cause more non-smoothness.

We show that deep networks are better than shallow networks at approximating functions that can be expressed as a composition of functions described by a directed acyclic graph, because the deep networks can be designed to have the same compositional structure, while a shallow network cannot exploit this knowledge. Thu…

In the framework of Lorentzian warped products, we study the Friedmann-Robertson-Walker cosmological model to investigate non-smooth curvatures associated with multiple discontinuities involved in the evolution of the universe. In particular we analyze non-smooth features of the spatially flat Friedmann-Robertson-Walke…

This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.

Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast…

New algorithms optimize non-smooth, non-convex objectives with improved complexity.

The Positive Mass Theorem implies that any smooth, complete, asymptotically flat 3-manifold with non-negative scalar curvature which has zero total mass is isometric to (R^3, delta_{ij}). In this paper, we quantify this statement using spinors and prove that if a complete, asymptotically flat manifold with non-negative…

Positive mass theorem for non-smooth metrics on flat manifolds with corners.

Paper tackles private optimization for non-smooth objectives efficiently.

Deep neural networks perform well on real world data but are prone to adversarial perturbations: small changes in the input easily lead to misclassification. In this work, we propose an attack methodology not only for cases where the perturbations are measured by $\ell_p$ norms, but in fact any adversarial dissimilarit…

The curvature-dimension condition fails in sub-Finsler geometry, extending previous results in sub-Riemannian geometry.

We investigate a generalization of the so-called metric splitting of globally hyperbolic space-times to non-smooth Lorentzian manifolds and show the existence of this metric splitting for a class of wave-type space-times. Our approach is based on smooth approximations of non-smooth space-times by families (or sequences…

Advances smooth over-parameterization for solving non-smooth optimization problems.