arXiv research

Paper develops methods for non-quadratic loss low-rank matrix recovery.

Through the direct study of the analysis estimator we derive oracle inequalities with fast and slow rates by adapting the arguments involving projections by Dalalyan, Hebiri and Lederer (2017). We then extend the theory to the square root analysis estimator. Finally, we focus on (square root) total variation regularize…

Paper proposes estimators for sparse PCA with oracle property.

Prior knowledge on properties of a target model often come as discrete or combinatorial descriptions. This work provides a unified computational framework for defining norms that promote such structures. More specifically, we develop associated tools for optimization involving such norms given only the orthogonal proje…

New algorithm reduces adaptive regret without projections.

Paper tackles online learning on curved spaces without projections.

We propose in this paper a general framework for deriving loss functions for structured prediction. In our framework, the user chooses a convex set including the output space and provides an oracle for projecting onto that set. Given that oracle, our framework automatically generates a corresponding convex and smooth l…

Develops accelerated methods for optimization using low-dimensional projected-gradient information.

New RLHF framework handles general preference oracles without reward functions.

Improved method reduces projection calls for nonsmooth convex optimization.

Nyström subsampling with Tikhonov regularization for covariate shift adaptation under misspecified case

New algorithms optimize actions under time-varying constraints without projecting.

Riemannian Proximal Sampler improves sampling on manifold data.

New algorithms sample convex bodies using Markov chains and restricted Gaussian oracles.

Two important goals of high-dimensional modeling are prediction and variable selection. In this article, we consider regularization with combined $L_1$ and concave penalties, and study the sampling properties of the global optimum of the suggested method in ultra-high dimensional settings. The $L_1$-penalty provides th…

Develops new oracle inequalities for Gaussian ranking estimators.

Paper addresses online alignment of large language models under uncertain preference feedback.

New algorithm solves complex optimization problems without needing projections.

We revisit the challenge of designing online algorithms for the bandit convex optimization problem (BCO) which are also scalable to high dimensional problems. Hence, we consider algorithms that are \textit{projection-free}, i.e., based on the conditional gradient method whose only access to the feasible decision set, i…

Statistical analysis of regularization in continual learning tasks.

Oracle inequality for sparse neural nets adapts to unknown structure.

Study efficient algorithms for nonconvex optimization with state-dependent Markov data.

High-dimensional data analysis has motivated a spectrum of regularization methods for variable selection and sparse modeling, with two popular classes of convex ones and concave ones. A long debate has been on whether one class dominates the other, an important question both in theory and to practitioners. In this pape…

High throughput genetic sequencing arrays with thousands of measurements per sample and a great amount of related censored clinical data have increased demanding need for better measurement specific model selection. In this paper we establish strong oracle properties of nonconcave penalized methods for nonpolynomial (N…

Proposes RDIV for IV estimation avoiding limitations of existing methods.

The paper tackles deep learning from dependent data, achieving optimal performance.

A desirable property of interpretable models is small size, so that they are easily understandable by humans. This leads to the following challenges: (a) small sizes typically imply diminished accuracy, and (b) bespoke levers provided by model families to restrict size, e.g., L1 regularization, might be insufficient to…

New algorithm predicts piecewise regular functions online.

A new test method improves goodness-of-fit tests for copulas.

New GIC improves model selection for structured sparse models.

The paper improves count data regression models for overdispersed data.

SFC aims to protect the Amazon with a digital currency and smart contracts.

New analysis reveals optimal regularization for ESNs, avoiding double descent.

We give nearly matching upper and lower bounds on the oracle complexity of finding $ε$-stationary points ($\| \nabla F(x) \| \leqε$) in stochastic convex optimization. We jointly analyze the oracle complexity in both the local stochastic oracle model and the global oracle (or, statistical learning) model. This allows u…

New algorithms optimize convex functions with high-order derivatives.

This paper investigates tradeoffs among optimization errors, statistical rates of convergence and the effect of heavy-tailed errors for high-dimensional robust regression with nonconvex regularization. When the additive errors in linear models have only bounded second moment, we show that iteratively reweighted $\ell_1…

PCA-based dimensionality reduction improves robustness in overparameterized linear models.

ADML combines debiased learning with data-driven model selection for efficient inference.

Constrained second-order convex optimization algorithms are the method of choice when a high accuracy solution to a problem is needed, due to their local quadratic convergence. These algorithms require the solution of a constrained quadratic subproblem at every iteration. We present the \emph{Second-Order Conditional G…

Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse data. This line of work shows that $\ell_1$-regularized least squares regression can accurately estimate a sparse linear model from $n$ nois…

High-dimensional sparse modeling via regularization provides a powerful tool for analyzing large-scale data sets and obtaining meaningful, interpretable models. The use of nonconvex penalty functions shows advantage in selecting important features in high dimensions, but the global optimality of such methods still dema…

SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization. However, SPIDER uses an accuracy-dependent stepsize that slows down the convergence in practice, and cannot handle objec…

Study mini-batch SGD noise and its limits, proving complexity guarantees.

Paper proposes a new method to optimize deep neural networks with sparse regularization.

In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate $O\left(\frac{1}{\varepsilon^2}\right)$ improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…

New algorithms ensure reproducibility and optimal convergence in convex optimization.

Federated learning is the centralized training of statistical models from decentralized data on mobile devices while preserving the privacy of each device. We present a robust aggregation approach to make federated learning robust to settings when a fraction of the devices may be sending corrupted updates to the server…

Estimates CATEs for structured treatments using a new decomposition method.