New algorithm improves online binary classification with constant time complexity.
problem Online binary classification with rebalancing.
method Non-iteratively reweighted recursive least-squares.
result Exacts converges to batch formulation and outperforms existing algorithms.
Unified analysis of reweighted least-squares algorithms for linear models.
problem Recovering unknown signals from linear measurements using reweighted least squares.
method Unified asymptotic analysis of IRLS, lin-RFM, and alternating minimization algorithms.
result The algorithms can achieve favorable performance in a few iterations with appropriate reweighting.
New algorithms improve L1 PCA performance.
problem Optimizing PCA with L1 norm for better data reduction.
method Iteratively reweighted least squares algorithms.
result Proposed algorithms outperform existing methods.
New IRLS algorithms for SVM fitting via MM approach.
problem Fitting support vector machines (SVMs) via quadratic programming.
method Majorization--Minimization (MM) paradigm for iteratively-reweighted least-squares (IRLS) algorithms.
result IRLS algorithms for SVM risk minimization problems with various losses and penalties.
Federated learning is protected against adversarial attacks with residual-based reweighting.
problem Adversarial attacks on federated learning's aggregation process.
method Residual-based reweighting combined with iteratively reweighted least squares.
result Our aggregation algorithm outperforms other methods in label-flipping and backdoor attacks.
Improved robustness in kernel-based regression via novel loss function and IRLS.
problem Noise sensitivity in kernel-based regression methods.
method Proposed ℓ s \ell_s ℓ s -loss function and iteratively reweighted least squares (IRLS) optimization. result Improved noise robustness in kernel-based regression methods.
Method improves SINDy for noisy nonlinear systems.
problem Recover nonlinear dynamical systems from noisy data.
method Reweighted ℓ 1 \ell_1 ℓ 1 -regularized least squares. result Improved accuracy and robustness in noisy conditions.
Improved matching for multiple objects using a novel reweighting method.
problem Current multi-object matching methods have limitations and are not robust.
method Proposes a novel iterative reweighting strategy using the graph connection Laplacian.
result Demonstrates superior performance over state-of-the-art methods.
Kernel regression predicts graph signals in noisy environments.
problem Predicting smooth graph signals in the presence of sparse noise.
method Kernel regression with ℓ 1 \ell_1 ℓ 1 -norm and ℓ 2 \ell_2 ℓ 2 -norm optimization using IRLS. result Efficacy demonstrated on real-world temperature data.
Efficiently estimates rotations with corrupted data.
problem Rotation synchronization under high corruption and noise.
method Message passing algorithm with reweighted least squares.
result Superior performance over state-of-the-art methods.
Paper proves IRLS converges to subspace from any start, with practical benefits.
problem Robust subspace estimation in machine learning.
method Iteratively Reweighted Least Squares (IRLS) with dynamic smoothing regularization.
result IRLS converges linearly to the underlying subspace from any initialization under deterministic conditions.
Simplifies neural regression by combining two sub-networks for predictions and uncertainties.
problem Neural networks underestimate uncertainty, leading to overly confident predictions.
method Extends IRLS to a two-sub-network approach with shared representations and complementary loss functions.
result Proposed network is simpler to implement and more robust to uncertainty variations.
This work presents a general framework for solving the low rank and/or sparse matrix minimization problems, which may involve multiple non-smooth terms. The Iteratively Reweighted Least Squares (IRLS) method is a fast solver, which smooths the objective function and minimizes it by alternately updating the variables an…
Global convergence for robust regression problems via IRLS with enhancements.
problem Global convergence for robust regression problems.
method Augmentations to IRLS to ensure global recovery and improved robustness.
result Global recovery guarantees for robust regression problems, outperforming state-of-the-art algorithms.
The paper improves sample reweighting methods for adapting to covariate shifts.
problem Improving accuracy in reproducing kernel Hilbert spaces when data distributions differ.
method Combining known error bounds for reweighted kernel regression in RKHS to show reduced sample size needed for accuracy.
result Under weak smoothness conditions, fewer samples are needed for the same accuracy as standard supervised learning.
Paper accelerates an optimization algorithm using extrapolation techniques.
problem Optimizing problems with a sum of a differentiable loss and a nonconvex sparsity regularizer.
method Incorporates extrapolation techniques into iteratively reweighted ℓ 1 \ell_1 ℓ 1 algorithms. result Sequence generated clusters at stationary points of the optimization problem.
A novel AIRLS algorithm for multiaffine variable relations in high-dimensional problems.
problem Challenges in Maximum Likelihood Estimation in high-dimensional settings with complex variable relations.
method Proposes an Alternating and Iteratively-Reweighted Least Squares (AIRLS) algorithm for multiaffine variable relations.
result Proves convergence for problems with Generalized Normal Distributions and shows empirically super-linear convergence rate.
RFSVM with random features achieves faster learning rates.
problem Improving the learning rate of SVM with random features.
method Support Vector Machine with N ≪ m N\ll m N ≪ m random features, optimized feature map, and reweighted feature selection. result RFSVM achieves faster learning rates than O ( 1 / m ) O(1/\sqrt{m}) O ( 1/ m ) under low noise assumptions. New robust MPCA method handles casewise and cellwise outliers in tensor data.
problem Outliers, especially casewise and cellwise, affect the performance of standard MPCA.
method Uses a single loss function to reduce the influence of both types of outliers and missing values.
result The new method improves robustness and performance in tensor data analysis.
We propose a new sparse regression method called the component lasso, based on a simple idea. The method uses the connected-components structure of the sample covariance matrix to split the problem into smaller ones. It then solves the subproblems separately, obtaining a coefficient vector for each one. Then, it uses n…
A new approach for signal parametrization, which consists of a specific regression model incorporating a discrete hidden logistic process, is proposed. The model parameters are estimated by the maximum likelihood method performed by a dedicated Expectation Maximization (EM) algorithm. The parameters of the hidden logis…
We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning ( L / μ ) 2 (L/μ)^2 ( L / μ ) 2 (where L L L is a bound on the smoothness and μ μ μ on the strong convexity) to a linear dependence on L / μ L/μ L / μ . …
Develops a privacy-preserving IRLS algorithm for L1 minimization.
problem Privacy concerns in iteratively reweighted least squares for L1 minimization.
method Separates sensitivity analysis, uses concentrated differential privacy.
result Outputs privatised and accurate IRLS solutions.
Exact LAD line fitting via PALB with linear scaling and speed.
problem Robust line fitting for data with outliers.
method Piecewise Affine Lower-Bounding (PALB) method using supporting lines and subdivision scheme.
result Empirical log-linear scaling and significantly faster than LP and IRLS methods.
Robust method estimates state, input, and parameters of linear systems online.
problem Joint estimation of state, input, and parameters in noisy or outlier-prone measurements.
method Combines recursive, alternating, and iteratively-reweighted least squares into a single algorithm.
result Good performance in presence of outliers and compared to state-of-the-art methods.
A new approach for feature extraction from time series is proposed in this paper. This approach consists of a specific regression model incorporating a discrete hidden logistic process. The model parameters are estimated by the maximum likelihood method performed by a dedicated Expectation Maximization (EM) algorithm. …
Convex program recovers mixture components in well-separated data.
problem Mixed linear regression with well-separated classes.
method Second-order cone program based on L1 minimization.
result The convex program exactly recovers mixture components under well-separation assumptions.
This paper addresses the problem of blind and fully constrained unmixing of hyperspectral images. Unmixing is performed without the use of any dictionary, and assumes that the number of constituent materials in the scene and their spectral signatures are unknown. The estimated abundances satisfy the desired sum-to-one …
Using the ℓ 1 \ell_1 ℓ 1 -norm to regularize the estimation of the parameter vector of a linear model leads to an unstable estimator when covariates are highly correlated. In this paper, we introduce a new penalty function which takes into account the correlation of the design matrix to stabilize the estimation. This norm, ca…
A new framework detects changepoints in complex data.
problem Detecting structural changes in data with various patterns and trends.
method Iteratively Reweighted Fused Lasso (IRFL) for L0 model selection.
result IRFL achieves accurate changepoint detection across various challenging scenarios.
Optimal Biweight kernel and computationally efficient Epanechnikov kernel for modal linear regression.
problem Finding the best kernel for modal linear regression.
method Refined analysis of asymptotic statistical behavior and IRLS algorithm convergence.
result Biweight kernel minimizes asymptotic mean squared error, Epanechnikov kernel guarantees IRLS convergence.
In this paper, a new descriptor selection method for selecting an optimal combination of important descriptors of sulfonamide derivatives data, named self tuned reweighted sampling (STRS), is developed. descriptors are defined as the descriptors with large absolute coefficients in a multivariate linear regression model…
CD converges linearly for MCP/SCAD penalized least squares.
problem Recovering sparse signals from data.
method Coordinate descent for MCP/SCAD penalized least squares.
result CD converges linearly to solutions of MCP/SCAD penalized least squares.
RFM reduces feature space for linear models, improving sparse recovery.
problem Sparse linear regression and low-rank matrix recovery.
method Recursive Feature Machines (RFM) that alternates between reweighting feature vectors by AGOP and learning prediction function.
result RFM generalizes IRLS and outperforms deep linear networks.
New method corrects least-squares temporal difference for better lambda-return estimation.
problem Improving lambda-return estimation in reinforcement learning.
method Uncorrected least-squares temporal difference with a correction method.
result Enhanced accuracy in temporal difference learning.
Cross validation residuals extended to GLS models.
problem Validating models with correlated data.
method Leave-M-out cross validation for GLS models, demonstrating relationship with Cook's distance.
result No need to refit model for reduced datasets.
Illustrates interleaved learning with Kalman Filter for linear least squares.
problem Improving machine learning algorithms through interleaved learning.
method Simple statistical and optimization framework based on Kalman Filter.
result Demonstrates the effectiveness of interleaved learning.
We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance meas…
Connection found between signal processing and slime mold dynamics.
problem Convergence of IRLS algorithm remains an open problem.
method Connection between IRLS and Physarum dynamics, proving convergence of a damped version.
result Convergence and complexity bounds for a damped version of the IRLS algorithm.
Jointly learns neural networks across datasets to improve network quality.
problem Jointly learning neural networks across diverse datasets to extract correlated information.
method Formulates joint learning as sharing network weights across multiple networks, solves an optimization problem to determine shared layers.
result Our approach outperforms baselines in image classification, auto-encoders, and image generation tasks.
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
New method improves accuracy of compressed least-squares solutions.
problem Error and noise introduced by compression in least-squares problems.
method Proposes robust partially-compressed least-squares models and algorithms.
result Robust solutions more accurate than classical compressed variants.
Simplified proof shows SGD optimality for least squares.
problem Optimizing SGD for least squares efficiency.
method Analyzing SGD as a stochastic process, characterizing stationary covariance matrix.
result Statistical minimax optimality of SGD for least squares.
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
Reduced-rank method improves least-squares regression under output regularity.
problem Least-squares regression with infinite dimensional outputs.
method Reduced-rank method for solving least-squares problems with output regularity assumptions.
result Learning bounds and improved statistical performance compared to full-rank method.
Estimates Heston model parameters from continuous data.
problem Estimating parameters of the Heston model from continuous data.
method Least squares estimation for subcritical Heston model.
result Strong consistency and asymptotic normality of estimators proved.
A new convex loss function optimizes set predictions with balanced size and coverage.
problem Optimizing set predictions with balanced size and coverage.
method Proposes a convex loss function using Choquet integrals for nondecreasing subset-valued functions.
result Optimal trade-offs between conditional probabilistic coverage and set size.
Paper uses deep learning to solve PDEs without supervision.
problem Solving elliptic PDEs without labeled data.
method Uses deep neural networks and least-squares functionals.
result Demonstrates effectiveness on 1D second-order elliptic PDEs.