arXiv research

We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance meas…

A new method for estimating large-scale linear models with improved precision.

New algorithm improves online binary classification with constant time complexity.

Support vector machines (SVMs) are an important tool in modern data analysis. Traditionally, support vector machines have been fitted via quadratic programming, either using purpose-built or off-the-shelf algorithms. We present an alternative approach to SVM fitting via the majorization--minimization (MM) paradigm. Alg…

Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.

The paper tackles robust reinforcement learning with performance guarantees.

New algorithm reduces bias and variance in weighted least-squares solutions.

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as…

We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves…

Developed an efficient iterative algorithm for SVI model.

A fast sketching algorithm solves regularized least squares problems efficiently.

Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottlene…

Paper improves worst-case regret bounds for RLSVI in reinforcement learning.

Improved RL algorithm reduces regret in large state spaces.

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…

New algorithm reduces sketching dimension to effective problem size.

Unified analysis of reweighted least-squares algorithms for linear models.

Principal component analysis (PCA) is often used to reduce the dimension of data by selecting a few orthonormal vectors that explain most of the variance structure of the data. L1 PCA uses the L1 norm to measure error, whereas the conventional PCA uses the L2 norm. For the L1 PCA problem minimizing the fitting error of…

We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions. We explain why versions of least-squares value iteration that use Boltzmann or epsilon-greedy exploration can be highly ineffic…

This work provides a simplified proof of the statistical minimax optimality of (iterate averaged) stochastic gradient descent (SGD), for the special case of least squares. This result is obtained by analyzing SGD as a stochastic process and by sharply characterizing the stationary covariance matrix of this process. The…

Efficiently estimates rotations with corrupted data.

Study on adversarial training dynamics in high dimensions using SGD.

In this paper we study the performance of the Projected Gradient Descent(PGD) algorithm for $\ell_{p}$-constrained least squares problems that arise in the framework of Compressed Sensing. Relying on the Restricted Isometry Property, we provide convergence guarantees for this algorithm for the entire range of $0\leq p\…

SGD converges to optimal solution in perfect data fitting problem.

This paper analyzes multi-pass SGD for least squares, improving generalization bounds.

We propose a version of least-mean-square (LMS) algorithm for sparse system identification. Our algorithm called online linearized Bregman iteration (OLBI) is derived from minimizing the cumulative prediction error squared along with an l1-l2 norm regularizer. By systematically treating the non-differentiable regulariz…

Improved neural network training with ADMM for hardware compatibility.

This work proves convergence of adaptive resampling for random Fourier features.

Improves Bayesian optimisation for engineering design problems with many variables.

Bayesian method improves online NARMAX model identification.

Federated learning is protected against adversarial attacks with residual-based reweighting.

Given a linear regression setting, Iterative Least Trimmed Squares (ILTS) involves alternating between (a) selecting the subset of samples with lowest current loss, and (b) re-fitting the linear model only on that subset. Both steps are very fast and simple. In this paper we analyze ILTS in the setting of mixed linear …

Improved SGD for non-strongly-convex regression with faster convergence.

New method for optimistic planning in MDPs using regularization.

Paper proves IRLS converges to subspace from any start, with practical benefits.

Improved privacy-preserving linear regression via iterative Hessian mixing.

Sublinear LSVI via LSH reduces runtime to sublinear in actions.

New method speeds up distributed linear regression.

New algorithm reduces rank constrained optimization problems.

Simplifies neural regression by combining two sub-networks for predictions and uncertainties.

This work presents a general framework for solving the low rank and/or sparse matrix minimization problems, which may involve multiple non-smooth terms. The Iteratively Reweighted Least Squares (IRLS) method is a fast solver, which smooths the objective function and minimizes it by alternately updating the variables an…

We propose a novel algorithm for greedy forward feature selection for regularized least-squares (RLS) regression and classification, also known as the least-squares support vector machine or ridge regression. The algorithm, which we call greedy RLS, starts from the empty feature set, and on each iteration adds the feat…

Improved matching for multiple objects using a novel reweighting method.

Exact LAD line fitting via PALB with linear scaling and speed.

New algorithm extracts shared latent space for cortico-muscular interactions.

We consider solving the $\ell_1$-regularized least-squares ($\ell_1$-LS) problem in the context of sparse recovery, for applications such as compressed sensing. The standard proximal gradient method, also known as iterative soft-thresholding when applied to this problem, has low computational cost per iteration but a r…

In this paper we investigate panel regression models with interactive fixed effects. We propose two new estimation methods that are based on minimizing convex objective functions. The first method minimizes the sum of squared residuals with a nuclear (trace) norm regularization. The second method minimizes the nuclear …

Derives scaling limits and fluctuations for SGD in high dimensions.