Improves numerical solution of ill-conditioned linear systems for machine learning.
problem Wastefulness and instability in solving ill-conditioned linear systems.
method autonugget combines Richardson extrapolation to determine the solution of the ill-conditioned system, improving accuracy over a single nugget.
result Improves accuracy of numerical solution of ill-conditioned linear systems.
Scalable method completes ill-conditioned matrices from few samples.
problem Matrix completion from few samples for ill-conditioned matrices.
method Iterative algorithm combining IRLS, smoothing Newton, and proximal gradient methods.
result Local quadratic convergence rate and well-conditioned linear systems.
Kaczmarz++ accelerates convergence for ill-conditioned systems.
problem Solving ill-conditioned linear systems efficiently.
method Adaptive momentum acceleration, Tikhonov-regularized projections, and memoization.
result Kaczmarz++ converges faster than Krylov methods on ill-conditioned systems.
AI-driven framework optimizes MCMC-based preconditioners for faster linear system solving.
problem Slow convergence of Krylov subspace solvers for ill-conditioned matrices.
method Graph neural surrogate and Bayesian optimization for AI-tuned MCMC parameters.
result 50% reduction in iterations to convergence on unseen system.
The paper addresses ill-conditioning in large spatial data, proposing solutions for prediction and likelihood estimation.
problem Ill-conditioning of the kernel matrix in large spatial data sets.
method Introduction of various optimality criteria and solutions for managing large spatial data.
result Solutions for managing large spatial data, addressing ill-conditioning and improving prediction and likelihood estimation.
New framework uses score-based priors to solve ill-conditioned polynomial equations, improving signal recovery from noisy data.
problem Recovering signals from low-order moments in inverse problems, especially ill-conditioned polynomial equations.
method Integrates score-based diffusion priors with moment-based estimators to regularize and solve nonlinear inverse problems.
result Diffusion priors improve recovery from third-order moments and make super-resolution MTD feasible.
Paper explores challenges in training PINNs and loss landscape effects.
problem Challenges in training Physics-Informed Neural Networks (PINNs) due to loss landscape issues.
method Examined gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, and introduced NysNewton-CG (NNCG).
result Adam+L-BFGS outperforms other optimizers, and NysNewton-CG significantly improves PINN performance.
SAPPHIRE tackles ill-conditioned rERM problems with faster convergence.
problem Ill-conditioned objectives and non-smooth regularizers in rERM.
method Sketch-based preconditioning and scaled proximal mapping.
result Achieves condition-number-free linear convergence.
This paper analyzes adaptive gradient algorithms for better performance in ill-conditioned problems.
problem Poor performance of standard stochastic gradient algorithms in ill-conditioned problems.
method Non-asymptotic analysis of adaptive gradient algorithms (Adagrad and Stochastic Newton) for strongly convex objectives.
result Theoretical analysis and adaptation to practical applications like linear regression and regularized GLM.
Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD(λ) - preconditioned gradient descent method. result ScaledGD(λ) converges at a constant linear rate after a logarithmic number of iterations. Scaled gradient descent improves matrix recovery for ill-conditioned matrices with optimal sampling complexity.
problem Recovering low-rank matrices from limited measurements efficiently and accurately.
method Scaled gradient descent (ScaledGD) with optimal sample complexity and improved iteration complexity.
result ScaledGD achieves optimal sample complexity and improved iteration complexity for ill-conditioned matrices.
This paper accelerates distributed convex optimization by mitigating ill-conditioning issues.
problem Distributed convex optimization with ill-conditioned aggregate cost functions.
method Iterative pre-conditioning technique to improve convergence rate and stability.
result The proposed algorithm converges linearly with improved convergence rate and superlinearly under certain conditions.
ScaledGD accelerates ill-conditioned low-rank estimation.
problem Slow convergence of gradient descent in ill-conditioned problems.
method Scaled gradient descent (ScaledGD) with preconditioning.
result Linear convergence rate independent of condition number.
New method speeds up Gaussian process training and inference for large datasets.
problem Training and inference in Gaussian processes are computationally expensive for large datasets.
method Iterative alternating projection method that accesses subblocks of the kernel matrix, reducing time and space complexity.
result Empirically, the method accelerates GP training and inference by up to 72x compared to conjugate gradients.
Preconditioned SGD accelerates convergence for ill-conditioned huge-scale matrix completion.
problem Recovering a low-rank matrix from incomplete data with high condition number.
method Preconditioned Stochastic Gradient Descent (SGD) for huge-scale online optimization.
result Preconditioned SGD converges to ε-accuracy in O(log(1/ε)) iterations, compared to O(κlog(1/ε)) for unpreconditioned SGD.
Proposes an efficient shrinkage path for ridge regression.
problem Ill-conditioned data in linear models.
method A new generalized ridge regression shrinkage path that minimizes MSE risk.
result The path is as short as possible while maintaining optimal trade-off.
Recently, Neural networks have seen a huge surge in its adoption due to their ability to provide high accuracy on various tasks. On the other hand, the existence of adversarial examples have raised suspicions regarding the generalization capabilities of neural networks. In this work, we focus on the weight matrix learn…
Bayesian regularization tackles collinearity in large-scale systems with correlated inputs.
problem Collinearity in large-scale linear systems identification due to correlated inputs.
method Bayesian regularization with stable spline covariance and Markov chain Monte Carlo scheme.
result Efficient reconstruction of impulse responses with high correlation among inputs.
We propose a fast, simple and robust algorithm for computing shortest paths and distances on Riemannian manifolds learned from data. This amounts to solving a system of ordinary differential equations (ODEs) subject to boundary conditions. Here standard solvers perform poorly because they require well-behaved Jacobians…
Regularized LAEs learn principal components efficiently.
problem Learning optimal linear representations with LAEs.
method Proper regularization schemes (non-uniform ℓ2 and nested dropout).
result Convergence to optimal representation is slow due to ill-conditioning.
Bayesian approach tackles collinearity in large-scale linear system identification.
problem Collinearity in large-scale linear system identification.
method Bayesian regularization framework with Gaussian process and stable spline kernel. Novel Markov chain Monte Carlo scheme.
result Efficiently reconstructs impulse responses posterior by dealing with collinearity.
In this paper, we consider the use of deep neural networks in the context of Multiple-Input-Multiple-Output (MIMO) detection. We give a brief introduction to deep learning and propose a modern neural network architecture suitable for this detection task. First, we consider the case in which the MIMO channel is constant…
Improved subgradient method tackles ill-conditioned composite optimization problems.
problem Slow convergence of subgradient method for composite optimization problems.
method Preconditioned subgradient method with Levenberg-Marquardt approach.
result Linear convergence rate for composite optimization problems under mild conditions.
The Dynamical Gaussian Process Latent Variable Models provide an elegant non-parametric framework for learning the low dimensional representations of the high-dimensional time-series. Real world observational studies, however, are often ill-conditioned: the observations can be noisy, not assuming the luxury of relative…
Stochastic gradient descent approximates Gaussian process posteriors efficiently.
problem Efficiently sampling from Gaussian process posteriors with limited computational resources.
method Developed stochastic gradient optimization objectives for sampling from Gaussian process posteriors.
result Stochastic gradient descent produces accurate predictive distributions, even in non-convergent cases.
We orthogonalize the NSS model to condition and diagnose its ill-conditioned parameters.
problem The ill-conditioning of the NSS model's design matrix.
method Exact orthogonal reparametrization via QR decomposition.
result Orthogonalization isolates the conditioning structure and maintains fit uncertainty.
Dual-space sampling tackles ill-conditioned inverse problems with Bayesian methods.
problem Bayesian inference in constrained inverse problems with ill-conditioned solutions.
method Dual-space posterior sampling using ADMM and SVGD.
result Well-calibrated uncertainty estimates and posterior contraction with increasing data.
Paper shows linear convergence of ISTA and FISTA for ill-conditioned images.
problem Solving linear inverse problems with sparse representation in signal and image processing.
method Revisits iterative shrinkage-thresholding algorithms (ISTA) and improves their convergence properties.
result Linear convergence of ISTA and FISTA for strongly convex smooth parts, even in ill-conditioned cases.
A new optimization method reduces memory and compute requirements for deep learning.
problem Memory and compute constraints in second-order stochastic optimizers for deep learning.
method Proposes KrAD, a novel factorization to approximate inverse Fisher matrix without inversion, leading to KrADagrad.
result Improves performance over Shampoo for 32-bit precision and comparable/generalization on real datasets.
New algorithms solve nonconvex-nonconcave minimax optimization problems.
problem Solving minimax optimization problems in machine learning.
method Two novel Newton-type algorithms for nonconvex-nonconcave minimax optimization.
result Proved local convergence at strict local minimax points.
ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.
problem Efficiently solving ill-conditioned low-rank matrix estimation problems.
method Scaled Gradient Descent (ScaledGD) with adaptive pre-conditioners.
result Linear convergence rate independent of condition number, low per-iteration cost.
RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.
problem Large linear systems in portfolio optimization and option pricing lead to slow convergence.
method Reinforcement Learning (RL) dynamically adjusts block-preconditioner sizes to accelerate convergence.
result RL-driven solver significantly reduces computational cost and accelerates convergence.
The total variation (TV) penalty, as many other analysis-sparsity problems, does not lead to separable factors or a proximal operatorwith a closed-form expression, such as soft thresholding for the ℓ_1 penalty. As a result, in a variational formulation of an inverse problem or statisticallearning estimation, it l…
Paper proposes a pre-conditioning method to speed up gradient descent in multi-agent optimization.
problem Speed up convergence of gradient descent in multi-agent optimization problems.
method Iterative pre-conditioning approach to mitigate the effect of problem conditioning.
result Significant improvement in convergence speed of gradient descent method.
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
New algorithm reduces sample complexity for sparse linear regression.
problem Sparse linear regression with correlated covariates and approximate dependencies.
method Polynomial-time algorithm that adapts the Lasso to tolerate approximate dependencies.
result Achieves near-optimal sample complexity for constant sparsity and ill-conditioned covariates.
In this paper, we study large-scale convex optimization algorithms based on the Newton method applied to regularized generalized self-concordant losses, which include logistic regression and softmax regression. We first prove that our new simple scheme based on a sequence of problems with decreasing regularization para…
A new method for robust matrix completion overcomes limitations of existing approaches.
problem Robust matrix completion from corrupted entries, especially under overparameterization and ill-conditioning.
method Factorization-based iterative algorithm combining Gauss-Newton linearization and outlier removal.
result Theoretical guarantees of exact recovery for suitable assumptions.
We present a machine learning approach to the inversion of Fredholm integrals of the first kind. The approach provides a natural regularization in cases where the inverse of the Fredholm kernel is ill-conditioned. It also provides an efficient and stable treatment of constraints. The key observation is that the stabili…
Gaussian Processes (GPs) are a popular approach to predict the output of a parameterized experiment. They have many applications in the field of Computer Experiments, in particular to perform sensitivity analysis, adaptive design of experiments and global optimization. Nearly all of the applications of GPs require the …
New insights into convergence of optimization methods for DAG structure learning.
problem Unclear convergence properties of optimization methods for structure learning.
method Examined the convergence of augmented Lagrangian method (ALM) and quadratic penalty method (QPM) for structure learning.
result Standard convergence result of ALM does not hold in various cases, and QPM is prone to ill-conditioning.
Developed shrinkage methods for Poisson regression models with experts to handle multicollinearity.
problem Multicollinearity in Poisson regression models with experts.
method Ridge and Liu-type shrinkage methods.
result Shrinkage methods offer more reliable estimates for coefficients in multicollinearity.
New algorithm improves heteroskedastic PCA performance.
problem Estimating low-rank matrix subspace from noisy data.
method Deflated-HeteroPCA algorithm, dividing spectrum into subblocks.
result Near-optimal and condition-number-free statistical guarantees.
New insights into adversarial vulnerability linked to manifold separability issues.
problem Adversarial vulnerability in machine learning models.
method Characterized data distribution as a low-dimensional manifold, focusing on on/off manifold directions and the impact of first-order vs. second-order optimization methods.
result First-order optimization methods lead to poor convergence in the off-manifold direction, causing adversarial vulnerability in inseparable datasets.
Consider Least Squares Monte Carlo (LSM) algorithm, which is proposed by Longstaff and Schwartz (2001) for pricing American style securities. This algorithm is based on the projection of the value of continuation onto a certain set of basis functions via the least squares problem. We analyze the stability of the algori…
Developed MF-PINNs to solve coupled Stokes-Darcy equations more accurately.
problem Solving coupled Stokes-Darcy equations with varying physical constants.
method Combining VP and SV forms with adjusted weights in MF-PINNs.
result Improved accuracy of streamline and pressure fields in numerical experiments.
Method interpolates option prices and volatilities without arbitrage.
problem Interpolating option prices and volatilities without arbitrage.
method Sparse modeling approach based on integral equations and SVD.
result Flexible and efficient framework for arbitrage-free interpolation.
Several methods have been recently proposed for estimating sparse Gaussian graphical models using ℓ1 regularization on the inverse covariance matrix. Despite recent advances, contemporary applications require methods that are even faster in order to handle ill-conditioned high dimensional modern day datasets. I…