Improved robustness in kernel-based regression via novel loss function and IRLS.
problem Noise sensitivity in kernel-based regression methods.
method Proposed ℓs-loss function and iteratively reweighted least squares (IRLS) optimization. result Improved noise robustness in kernel-based regression methods.
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
We find a convex model for traditional nonlinear regression under L2 loss.
problem Nonlinear regression under L2 loss with non-convex optimization.
method Showed a convex nonlinear regression model for least squares problem.
result Existence of a convex model simplifies training complex systems.
Paper develops a least-squares framework for learning discrete losses.
problem Learning strategies for discrete losses (e.g., multilabeling, ranking).
method Least-squares framework to systematically design learning algorithms for discrete losses.
result Improved results with explicit dependence on the number of labels and faster learning rates.
New algorithms estimate Jacobian matrices for large-scale machine learning.
problem Efficiently computing search directions for large nonlinear least squares.
method Exploit low-rank structure in Hessian to estimate Jacobian matrices.
result Two algorithms perform well compared to state-of-the-art methods.
We introduce the implicitly constrained least squares (ICLS) classifier, a novel semi-supervised version of the least squares classifier. This classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-s…
LSGANs improve GANs by using least squares loss, leading to better image quality and stability.
problem Vanishing gradients in GANs during training.
method Introducing LSGANs with least squares loss for both discriminator and generator.
result LSGANs generate higher quality images and are more stable during training.
3D GAN improves MRI image quality from low-res scans.
problem Improving MRI image quality from low-resolution scans.
method Adversarial learning using 3D convolutions and least squares loss.
result 3D GAN generates high-quality 3D MRI images from low-resolution scans.
Optimal weight windows are symmetric rectangles centered at peak.
problem Finding the best weight windows for weighted least squares.
method Investigated symmetric and tapered rectangle window weights, showing the best rectangle window is optimal.
result The best rectangle window is optimal for all tapered rectangle window definitions.
Paper uses deep learning to solve PDEs without supervision.
problem Solving elliptic PDEs without labeled data.
method Uses deep neural networks and least-squares functionals.
result Demonstrates effectiveness on 1D second-order elliptic PDEs.
We introduce a novel semi-supervised version of the least squares classifier. This implicitly constrained least squares (ICLS) classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-supervised method…
Algorithm solves robust linear regression with block Lewis weights.
problem Group distributionally robust least squares problem.
method Algorithm based on geometric construction and block Lewis weights, using accelerated proximal methods.
result Improves over known methods for moderate accuracy regimes and matches state-of-the-art guarantees.
Optimal multiscale learning of linear operators
problem Statistical and computational limits of learning bounded linear operators between Sobolev spaces
method Reformulate as an infinite-dimensional matrix regression problem with heterogeneous multiscale structure
result Establish minimax rates and construct a finite-resolution blockwise least-squares estimator attaining these rates
The paper explores machine learning methods for proxy modeling in life insurance solvency capital requirements.
problem Life insurance companies need to estimate solvency capital requirements from full loss distributions, but computational limitations restrict full simulations.
method The paper presents various adaptive machine learning approaches to approximate the risk-dependent proxy function using least-squares Monte Carlo.
result The machine learning methods significantly improve the accuracy and efficiency of proxy modeling compared to traditional regression techniques.
New algorithm improves regression error bounds and accelerates performance for low noise.
problem Nonparametric least square regression in RKHS with optimal error bounds.
method Kernel Truncated Randomized Ridge Regression (KTRRR) with optimal generalization error bounds.
result Faster finite-time and asymptotic rates on low noise problems.
Support vector machines (SVMs) are an important tool in modern data analysis. Traditionally, support vector machines have been fitted via quadratic programming, either using purpose-built or off-the-shelf algorithms. We present an alternative approach to SVM fitting via the majorization--minimization (MM) paradigm. Alg…
Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.
problem Optimal risk in bounded distributions for constrained least squares.
method Comparison of least squares and non-linear predictors.
result Non-linear predictors can achieve optimal risk O(d/n) in bounded distributions. This work studies applications and generalizations of a simple estimation technique that provides exponential concentration under heavy-tailed distributions, assuming only bounded low-order moments. We show that the technique can be used for approximate minimization of smooth and strongly convex losses, and specificall…
Principal Component Analysis (PCA) is a very successful dimensionality reduction technique, widely used in predictive modeling. A key factor in its widespread use in this domain is the fact that the projection of a dataset onto its first K principal components minimizes the sum of squared errors between the original …
We consider the problem of predicting as well as the best linear combination of d given functions in least squares regression, and variants of this problem including constraints on the parameters of the linear combination. When the input distribution is known, there already exists an algorithm having an expected excess…
Paper improves RLS for sparse outlier detection in linear models.
problem Outliers contaminate linear regression models infrequently.
method Hierarchical-optimization recursive least squares with sparsity-inducing regularization.
result The method robustly estimates linear filters/systems with outliers.
Least Squares Estimators are suboptimal for 5D convex functions.
problem Suboptimality of Least Squares Estimators in estimating multidimensional convex functions.
method Analysis of natural subclasses of convex functions in random and fixed design settings.
result Risk of LSE is n−2/d while minimax risk is n−4/(d+4) for d≥5. In recent years Variation Autoencoders have become one of the most popular unsupervised learning of complicated distributions.Variational Autoencoder (VAE) provides more efficient reconstructive performance over a traditional autoencoder. Variational auto enocders make better approximaiton than MCMC. The VAE defines a …
R2T hybrid model improves robust regression for asymmetric noise.
problem Least-squares regression fails with asymmetric structured noise.
method Transformer encoder, compression NN, fixed symbolic equation.
result Median regression MSE of 6e-6 to 3.5e-5 on synthetic data.
New method quantifies life insurance risk using Monte Carlo simulations.
problem Quantifying long-term risks in insurance portfolios over a one-year horizon.
method Least-squares Monte Carlo methods to quantify impact of new experience.
result Model can be used as an internal model under Solvency II or SST.
Improved robust regression for heavy-tailed and contaminated data.
problem Linear regression with heavy-tailed and adversarially contaminated covariates and responses.
method Applying a filtering algorithm to covariates and then using Huber regression, least trimmed squares, or least absolute deviation estimators on the remaining data.
result Near-optimal error rates achieved for the Huber regression estimator.
We provide a pointwise confidence bound for non-linear least-squares with fixed design.
problem Confidence estimation in non-linear ℓ2-regularized least squares. method Pointwise confidence bound for local minimizers, using weighted norm involving inverse-Hessian.
result The proposed confidence bound scales with the test input's similarity to the training data.
A structured prediction method for ranking labels.
problem Solving label ranking problems as structured output regression.
method Two-step approach: regression in feature space followed by pre-image solving.
result Efficiency on real-world datasets for partial and complete rankings.
New algorithms improve scalar quantization by optimizing sparse least squares.
problem Improving efficiency and accuracy of scalar quantization for neural networks.
method Sparse least square optimization, iterative and clustering-based methods.
result Proposed algorithms outperform existing methods, especially in bit-width reduction scenarios.
Unified framework for non-Euclidean CPD under scalable stochastic mirror descent.
problem Handling non-Euclidean losses in tensor decomposition.
method Tensor fiber sampling strategy-based stochastic mirror descent.
result Global convergence to a stationary point under reasonable conditions.
Develops a privacy-preserving algorithm for sparse robust regression.
problem Privacy-preserving machine learning for sparse robust regression.
method Develops FRAPPE algorithm for non-smooth loss under differential privacy.
result Achieves better privacy and statistical accuracy trade-off.
RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.
problem Improving performance of deep residual neural networks (RFNNs) while preserving convex optimization benefits.
method Random Feature Representation Boosting (RFRBoost) using boosting theory and random features at each layer.
result RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in small- to medium-scale tabular datasets.
The discovery of non-linear causal relationship under additive non-Gaussian noise models has attracted considerable attention recently because of their high flexibility. In this paper, we propose a novel causal inference algorithm called least-squares independence regression (LSIR). LSIR learns the additive noise model…
This paper extends the standard chaining technique to prove excess risk upper bounds for empirical risk minimization with random design settings even if the magnitude of the noise and the estimates is unbounded. The bound applies to many loss functions besides the squared loss, and scales only with the sub-Gaussian or …
Optimal weight windows are found by projecting the origin onto a convex polytope.
problem Finding the best weight windows for a weighted moving average smoother.
method Formulated as a quadratic program and projection onto a convex polytope.
result Optimal weight windows are symmetrical and decrease in weight away from the center.
Study optimal rates for spectral algorithms in Hilbert spaces.
problem Regression problems over separable Hilbert spaces with square loss.
method Investigate spectral/regularized algorithms including ridge, principal component, and gradient methods.
result Prove optimal, high-probability convergence results in terms of norms.
Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.
problem Limited theoretical analysis for distributed ERM with general loss functions and hypothesis spaces.
method Derive tight risk bounds under assumptions on hypothesis space and loss function.
result Developed more general risk bound for distributed ERM without strong convexity restriction.
We develop and analyze stochastic optimization algorithms for problems in which the expected loss is strongly convex, and the optimum is (approximately) sparse. Previous approaches are able to exploit only one of these two structures, yielding an $\order(\pdim/T)$ convergence rate for strongly convex objectives in $\pd…
Classification and regression tasks in overparameterized models show different generalization properties.
problem Comparing classification and regression in overparameterized models.
method Comparison of least-squares minimum-norm interpolation and hard-margin SVM using different loss functions.
result Interpolating solutions generalize well with 0-1 loss but not with square loss.
Conditional expectiles are becoming an increasingly important tool in finance as well as in other areas of applications. We analyse a support vector machine type approach for estimating conditional expectiles and establish learning rates that are minimax optimal modulo a logarithmic factor if Gaussian RBF kernels are u…
CD converges linearly for MCP/SCAD penalized least squares.
problem Recovering sparse signals from data.
method Coordinate descent for MCP/SCAD penalized least squares.
result CD converges linearly to solutions of MCP/SCAD penalized least squares.
Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.
problem Understanding Neural Collapse in MSE-trained deep nets.
method Developed a new MSE loss decomposition and introduced the central path concept.
result Exact dynamics of Neural Collapse along the central path can be predicted.
We demonstrate that the primal-dual witness proof method may be used to establish variable selection consistency and ℓ∞-bounds for sparse regression problems, even when the loss function and/or regularizer are nonconvex. Using this method, we derive two theorems concerning support recovery and ℓ∞-…
The paper provides bounds for regression schemes using nonstationary training samples.
problem Developing confidence intervals for nonparametric regression with nonstationary data.
method The approach involves Rademacher and Vapnik-Chervonenkis theories to analyze the cost and optimality of regression schemes.
result The paper establishes nonasymptotic bounds for regression schemes and optimality in L2-distance. Study fast learning rates for square loss in dependent data with hypercontractivity condition.
problem Learning from dependent data with fast rates matching independent data.
method Martingale difference noise, trajectory hypercontractivity condition, least-squares estimator.
result Excess risk bound matches iid rate after burn-in time, independent of mixing-time.
The paper analyzes error bounds and KL properties for noisy matrix recovery problems.
problem Noisy low-rank matrix recovery problems.
method Squared F-norm regularization, accelerated alternating minimization method.
result Established error bounds and KL properties for critical points and global minimizers.
The goal of temporal alignment is to establish time correspondence between two sequences, which has many applications in a variety of areas such as speech processing, bioinformatics, computer vision, and computer graphics. In this paper, we propose a novel temporal alignment method called least-squares dynamic time war…
Least symmetry breaking principle explains SGD's local minima in shallow ReLU networks.
problem Understanding the structure of local minima in two-layer ReLU networks.
method Analyzing the squared loss optimization problem for ReLU networks with Gaussian inputs and applying the principle of least symmetry breaking.
result The principle of least symmetry breaking explains the structure of spurious local minima detected by SGD.