A novel AIRLS algorithm for multiaffine variable relations in high-dimensional problems.
problem Challenges in Maximum Likelihood Estimation in high-dimensional settings with complex variable relations.
method Proposes an Alternating and Iteratively-Reweighted Least Squares (AIRLS) algorithm for multiaffine variable relations.
result Proves convergence for problems with Generalized Normal Distributions and shows empirically super-linear convergence rate.
Support vector machines (SVMs) are an important tool in modern data analysis. Traditionally, support vector machines have been fitted via quadratic programming, either using purpose-built or off-the-shelf algorithms. We present an alternative approach to SVM fitting via the majorization--minimization (MM) paradigm. Alg…
Federated learning is protected against adversarial attacks with residual-based reweighting.
problem Adversarial attacks on federated learning's aggregation process.
method Residual-based reweighting combined with iteratively reweighted least squares.
result Our aggregation algorithm outperforms other methods in label-flipping and backdoor attacks.
New algorithm improves online binary classification with constant time complexity.
problem Online binary classification with rebalancing.
method Non-iteratively reweighted recursive least-squares.
result Exacts converges to batch formulation and outperforms existing algorithms.
This work presents a general framework for solving the low rank and/or sparse matrix minimization problems, which may involve multiple non-smooth terms. The Iteratively Reweighted Least Squares (IRLS) method is a fast solver, which smooths the objective function and minimizes it by alternately updating the variables an…
Principal component analysis (PCA) is often used to reduce the dimension of data by selecting a few orthonormal vectors that explain most of the variance structure of the data. L1 PCA uses the L1 norm to measure error, whereas the conventional PCA uses the L2 norm. For the L1 PCA problem minimizing the fitting error of…
Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottlene…
Improved matching for multiple objects using a novel reweighting method.
problem Current multi-object matching methods have limitations and are not robust.
method Proposes a novel iterative reweighting strategy using the graph connection Laplacian.
result Demonstrates superior performance over state-of-the-art methods.
Paper proves IRLS converges to subspace from any start, with practical benefits.
problem Robust subspace estimation in machine learning.
method Iteratively Reweighted Least Squares (IRLS) with dynamic smoothing regularization.
result IRLS converges linearly to the underlying subspace from any initialization under deterministic conditions.
Unified analysis of reweighted least-squares algorithms for linear models.
problem Recovering unknown signals from linear measurements using reweighted least squares.
method Unified asymptotic analysis of IRLS, lin-RFM, and alternating minimization algorithms.
result The algorithms can achieve favorable performance in a few iterations with appropriate reweighting.
Simplifies neural regression by combining two sub-networks for predictions and uncertainties.
problem Neural networks underestimate uncertainty, leading to overly confident predictions.
method Extends IRLS to a two-sub-network approach with shared representations and complementary loss functions.
result Proposed network is simpler to implement and more robust to uncertainty variations.
Robust method estimates state, input, and parameters of linear systems online.
problem Joint estimation of state, input, and parameters in noisy or outlier-prone measurements.
method Combines recursive, alternating, and iteratively-reweighted least squares into a single algorithm.
result Good performance in presence of outliers and compared to state-of-the-art methods.
Global convergence for robust regression problems via IRLS with enhancements.
problem Global convergence for robust regression problems.
method Augmentations to IRLS to ensure global recovery and improved robustness.
result Global recovery guarantees for robust regression problems, outperforming state-of-the-art algorithms.
Iteratively reweighted ℓ1 algorithm is a popular algorithm for solving a large class of optimization problems whose objective is the sum of a Lipschitz differentiable loss function and a possibly nonconvex sparsity inducing regularizer. In this paper, motivated by the success of extrapolation techniques in accele…
New robust MPCA method handles casewise and cellwise outliers in tensor data.
problem Outliers, especially casewise and cellwise, affect the performance of standard MPCA.
method Uses a single loss function to reduce the influence of both types of outliers and missing values.
result The new method improves robustness and performance in tensor data analysis.
Iteratively reweighted least squares (IRLS) is a widely-used method in machine learning to estimate the parameters in the generalised linear models. In particular, IRLS for L1 minimisation under the linear model provides a closed-form solution in each step, which is a simple multiplication between the inverse of the we…
A new approach for signal parametrization, which consists of a specific regression model incorporating a discrete hidden logistic process, is proposed. The model parameters are estimated by the maximum likelihood method performed by a dedicated Expectation Maximization (EM) algorithm. The parameters of the hidden logis…
Proposes a partitioned least squares model for feature grouping.
problem Modeling with feature groups to assess variable importance.
method Two methods: alternating least squares and exact reformulation.
result Exact method provides better results in less time.
We introduce a convex approach for mixed linear regression over d features. This approach is a second-order cone program, based on L1 minimization, which assigns an estimate regression coefficient in Rd for each data point. These estimates can then be clustered using, for example, k-means. For problem…
Paper improves AIRL by enhancing policy imitation and addressing reward recovery issues.
problem Inadequate policy imitation and limited transferable reward recovery in AIRL.
method Substituted built-in algorithm with SAC for policy updating and proposed PPO-AIRL + SAC hybrid framework.
result SAC improves policy imitation but hinders reward recovery; PPO-AIRL + SAC achieves satisfactory transfer effect.
A new approach for feature extraction from time series is proposed in this paper. This approach consists of a specific regression model incorporating a discrete hidden logistic process. The model parameters are estimated by the maximum likelihood method performed by a dedicated Expectation Maximization (EM) algorithm. …
GPCDL uses Gaussian Processes to learn smooth templates from data.
problem Lack of smoothness in learned templates leads to overfitting and poor predictive performance.
method GPCDL incorporates Gaussian Process priors to enforce smoothness in the learned templates.
result GPCDL outperforms unregularized CDL in accuracy and predictive performance across various SNRs and applications.
Exact LAD line fitting via PALB with linear scaling and speed.
problem Robust line fitting for data with outliers.
method Piecewise Affine Lower-Bounding (PALB) method using supporting lines and subdivision scheme.
result Empirical log-linear scaling and significantly faster than LP and IRLS methods.
In presence of sparse noise we propose kernel regression for predicting output vectors which are smooth over a given graph. Sparse noise models the training outputs being corrupted either with missing samples or large perturbations. The presence of sparse noise is handled using appropriate use of ℓ1-norm along-wi…
This paper explores alternative regression techniques in pricing American put options and compares to the least-squares method (LSM) in Monte Carlo implemented by Longstaff-Schwartz, 2001 which uses least squares to estimate the conditional expected payoff to the option holder from continuation. The pricing is done und…
Paper analyzes AIRL in high-dimensional spaces using random matrix theory.
problem AIRL's performance challenges in high-dimensional environments.
method Examined the rank of the matrix derived from transition matrix, applied random matrix theory.
result High-dimensional scenarios reveal transfer limitations not inherent to AIRL framework.
Time series are used in many domains including finance, engineering, economics and bioinformatics generally to represent the change of a measurement over time. Modeling techniques may then be used to give a synthetic representation of such data. A new approach for time series modeling is proposed in this paper. It cons…
Optimal Biweight kernel and computationally efficient Epanechnikov kernel for modal linear regression.
problem Finding the best kernel for modal linear regression.
method Refined analysis of asymptotic statistical behavior and IRLS algorithm convergence.
result Biweight kernel minimizes asymptotic mean squared error, Epanechnikov kernel guarantees IRLS convergence.
Private ALS method improves matrix completion with tighter rates and better privacy.
problem Differential privacy in matrix completion for user-level privacy.
method Joint differentially private ALS method with tighter sample complexity and privacy trade-offs.
result Achieves nearly optimal sample complexity and best privacy/utility trade-off.
In this paper, we introduce the problem of jointly learning feed-forward neural networks across a set of relevant but diverse datasets. Compared to learning a separate network from each dataset in isolation, joint learning enables us to extract correlated information across multiple datasets to significantly improve th…
In this paper we present a connection between two dynamical systems arising in entirely different contexts: one in signal processing and the other in biology. The first is the famous Iteratively Reweighted Least Squares (IRLS) algorithm used in compressed sensing and sparse recovery while the second is the dynamics of …
A new method speeds up ALS for recommender systems by subsampling key elements.
problem High computational cost of ALS for large-scale datasets.
method Core-elements subsampling method for efficient ALS approximation.
result Achieves similar accuracy with significantly reduced computational time.
In this paper, we propose a general framework to accelerate significantly the algorithms for nonnegative matrix factorization (NMF). This framework is inspired from the extrapolation scheme used to accelerate gradient methods in convex optimization and from the method of parallel tangents. However, the use of extrapola…
Randomized matrix compression techniques, such as the Johnson-Lindenstrauss transform, have emerged as an effective and practical way for solving large-scale problems efficiently. With a focus on computational efficiency, however, forsaking solutions quality and accuracy becomes the trade-off. In this paper, we investi…
A new method solves sparse regularization problems efficiently and robustly.
problem Sparse regularization in machine learning problems.
method Iteratively reweighted least square (IRLS) with bilevel resolution and BFGS solver.
result Achieves top performances on various sparsity and regularization problems.
Estimates network structure and interaction rules from multiple agent trajectories.
problem Modeling multi-agent systems on networks from data.
method Jointly infers network topology and interaction kernels using non-convex optimization.
result ORALS estimator is consistent and asymptotically normal under coercivity conditions.
The ratio of two probability densities can be used for solving various machine learning tasks such as covariate shift adaptation (importance sampling), outlier detection (likelihood-ratio test), and feature selection (mutual information). Recently, several methods of directly estimating the density ratio have been deve…
Deep reinforcement learning achieves superhuman performance in a range of video game environments, but requires that a designer manually specify a reward function. It is often easier to provide demonstrations of a target behavior than to design a reward function describing that behavior. Inverse reinforcement learning …
OLS predictions are shown to be similar to attention mechanisms in models.
problem OLS in traditional statistics and econometrics.
method Rewriting OLS as an attention mechanism in a transformed space.
result OLS can be understood as minimizing squared prediction errors via optimal embedding and decoding.
RFM reduces feature space for linear models, improving sparse recovery.
problem Sparse linear regression and low-rank matrix recovery.
method Recursive Feature Machines (RFM) that alternates between reweighting feature vectors by AGOP and learning prediction function.
result RFM generalizes IRLS and outperforms deep linear networks.
Matrix factorization is a popular approach to solving matrix estimation problems based on partial observations. Existing matrix factorization is based on least squares and aims to yield a low-rank matrix to interpret the conditional sample means given the observations. However, in many real applications with skewed and…
Motivated by the reconstruction and the prediction of electricity consumption, we extend Nonnegative Matrix Factorization~(NMF) to take into account side information (column or row features). We consider general linear measurement settings, and propose a framework which models non-linear relationships between features …
Scalable method completes ill-conditioned matrices from few samples.
problem Matrix completion from few samples for ill-conditioned matrices.
method Iterative algorithm combining IRLS, smoothing Newton, and proximal gradient methods.
result Local quadratic convergence rate and well-conditioned linear systems.
Functional PLS improves prediction and inference for scalar responses from functional predictors.
problem Estimating scalar responses from functional predictors in an ill-posed inverse problem.
method Functional partial least squares (PLS) estimator with adaptive early stopping and new tests.
result PLS attains nearly minimax-optimal convergence rates and detects local alternatives.
Paper presents novel online MTL methods using WRLS and OSLSSVR.
problem Online Multi-Task Learning (MTL) Regression Problems
method Develops recursive versions of WRLS and OSLSSVR for MTL.
result Achieves exact and approximate recursions with quadratic cost.
We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our proposed scheme is equivalent to running regular temporal difference learning with linear function approximation, albeit with samples picked unifo…
Improved neural network training with ADMM for hardware compatibility.
problem Training scalable and parallelizable neural networks.
method Alternating Direction Method of Multipliers (ADMM) combined with iterative least-square methods.
result Achieved significant accuracy improvements over SGD and Adam.
Reinforcement learning agents are prone to undesired behaviors due to reward mis-specification. Finding a set of reward functions to properly guide agent behaviors is particularly challenging in multi-agent scenarios. Inverse reinforcement learning provides a framework to automatically acquire suitable reward functions…