Bayesian optimisation framework considers only variable orderings to handle ill-conditioned objectives.
problem Bayesian optimisation fails with ill-conditioned or discontinuous objectives.
method Proposes a new framework that considers only the ordering of variables in input and output spaces, fitting a Gaussian process in a latent space.
result Proves optimal performance under the measure of regret for an optimistic strategy in the latent space.
SAPPHIRE tackles ill-conditioned rERM problems with faster convergence.
problem Ill-conditioned objectives and non-smooth regularizers in rERM.
method Sketch-based preconditioning and scaled proximal mapping.
result Achieves condition-number-free linear convergence.
ScaledGD accelerates ill-conditioned low-rank estimation.
problem Slow convergence of gradient descent in ill-conditioned problems.
method Scaled gradient descent (ScaledGD) with preconditioning.
result Linear convergence rate independent of condition number.
This paper analyzes adaptive gradient algorithms for better performance in ill-conditioned problems.
problem Poor performance of standard stochastic gradient algorithms in ill-conditioned problems.
method Non-asymptotic analysis of adaptive gradient algorithms (Adagrad and Stochastic Newton) for strongly convex objectives.
result Theoretical analysis and adaptation to practical applications like linear regression and regularized GLM.
The paper addresses ill-conditioning in large spatial data, proposing solutions for prediction and likelihood estimation.
problem Ill-conditioning of the kernel matrix in large spatial data sets.
method Introduction of various optimality criteria and solutions for managing large spatial data.
result Solutions for managing large spatial data, addressing ill-conditioning and improving prediction and likelihood estimation.
Paper shows linear convergence of ISTA and FISTA for ill-conditioned images.
problem Solving linear inverse problems with sparse representation in signal and image processing.
method Revisits iterative shrinkage-thresholding algorithms (ISTA) and improves their convergence properties.
result Linear convergence of ISTA and FISTA for strongly convex smooth parts, even in ill-conditioned cases.
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
Scalable method completes ill-conditioned matrices from few samples.
problem Matrix completion from few samples for ill-conditioned matrices.
method Iterative algorithm combining IRLS, smoothing Newton, and proximal gradient methods.
result Local quadratic convergence rate and well-conditioned linear systems.
Improved subgradient method tackles ill-conditioned composite optimization problems.
problem Slow convergence of subgradient method for composite optimization problems.
method Preconditioned subgradient method with Levenberg-Marquardt approach.
result Linear convergence rate for composite optimization problems under mild conditions.
Improves numerical solution of ill-conditioned linear systems for machine learning.
problem Wastefulness and instability in solving ill-conditioned linear systems.
method autonugget combines Richardson extrapolation to determine the solution of the ill-conditioned system, improving accuracy over a single nugget.
result Improves accuracy of numerical solution of ill-conditioned linear systems.
Paper explores challenges in training PINNs and loss landscape effects.
problem Challenges in training Physics-Informed Neural Networks (PINNs) due to loss landscape issues.
method Examined gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, and introduced NysNewton-CG (NNCG).
result Adam+L-BFGS outperforms other optimizers, and NysNewton-CG significantly improves PINN performance.
Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD(λ) - preconditioned gradient descent method. result ScaledGD(λ) converges at a constant linear rate after a logarithmic number of iterations. This paper investigates the control of an ML component within the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) devoted to black-box optimization. The known CMA-ES weakness is its sample complexity, the number of evaluations of the objective function needed to approximate the global optimum. This weakness is…
Scaled gradient descent improves matrix recovery for ill-conditioned matrices with optimal sampling complexity.
problem Recovering low-rank matrices from limited measurements efficiently and accurately.
method Scaled gradient descent (ScaledGD) with optimal sample complexity and improved iteration complexity.
result ScaledGD achieves optimal sample complexity and improved iteration complexity for ill-conditioned matrices.
We provide a new theoretical analysis framework to investigate online gradient descent in the dynamic environment. Comparing with the previous work, the new framework recovers the state-of-the-art dynamic regret, but does not require extra gradient queries for every iteration. Specifically, when functions are α stron…
New framework uses score-based priors to solve ill-conditioned polynomial equations, improving signal recovery from noisy data.
problem Recovering signals from low-order moments in inverse problems, especially ill-conditioned polynomial equations.
method Integrates score-based diffusion priors with moment-based estimators to regularize and solve nonlinear inverse problems.
result Diffusion priors improve recovery from third-order moments and make super-resolution MTD feasible.
Preconditioned SGD accelerates convergence for ill-conditioned huge-scale matrix completion.
problem Recovering a low-rank matrix from incomplete data with high condition number.
method Preconditioned Stochastic Gradient Descent (SGD) for huge-scale online optimization.
result Preconditioned SGD converges to ε-accuracy in O(log(1/ε)) iterations, compared to O(κlog(1/ε)) for unpreconditioned SGD.
RQMC improves optimization in variational Bayes problems.
problem Optimizing variational Bayes problems with noisy objective functions.
method Use of randomized quasi-Monte Carlo (RQMC) sampling with stochastic L-BFGS.
result RQMC can significantly speed up optimization and find better parameter values.
Proposes an efficient shrinkage path for ridge regression.
problem Ill-conditioned data in linear models.
method A new generalized ridge regression shrinkage path that minimizes MSE risk.
result The path is as short as possible while maintaining optimal trade-off.
Recently, Neural networks have seen a huge surge in its adoption due to their ability to provide high accuracy on various tasks. On the other hand, the existence of adversarial examples have raised suspicions regarding the generalization capabilities of neural networks. In this work, we focus on the weight matrix learn…
Kaczmarz++ accelerates convergence for ill-conditioned systems.
problem Solving ill-conditioned linear systems efficiently.
method Adaptive momentum acceleration, Tikhonov-regularized projections, and memoization.
result Kaczmarz++ converges faster than Krylov methods on ill-conditioned systems.
We introduce a generic scheme for accelerating gradient-based optimization methods in the sense of Nesterov. The approach, called Catalyst, builds upon the inexact accelerated proximal point algorithm for minimizing a convex objective function, and consists of approximately solving a sequence of well-chosen auxiliary p…
Regularized LAEs learn principal components efficiently.
problem Learning optimal linear representations with LAEs.
method Proper regularization schemes (non-uniform ℓ2 and nested dropout).
result Convergence to optimal representation is slow due to ill-conditioning.
Yau's Affine Normal Descent optimizes smooth unconstrained problems with geometrically adapted directions.
problem Optimizing smooth unconstrained problems with geometrically adapted directions.
method Yau's Affine Normal Descent (YAND) uses the equi-affine normal of level-set hypersurfaces as search directions.
result YAND converges globally under standard smoothness assumptions and locally quadratically near nondegenerate minimizers.
A large body of animation research focuses on optimization of movement control, either as action sequences or policy parameters. However, as closed-form expressions of the objective functions are often not available, our understanding of the optimization problems is limited. Building on recent work on analyzing neural …
We orthogonalize the NSS model to condition and diagnose its ill-conditioned parameters.
problem The ill-conditioning of the NSS model's design matrix.
method Exact orthogonal reparametrization via QR decomposition.
result Orthogonalization isolates the conditioning structure and maintains fit uncertainty.
Dual-space sampling tackles ill-conditioned inverse problems with Bayesian methods.
problem Bayesian inference in constrained inverse problems with ill-conditioned solutions.
method Dual-space posterior sampling using ADMM and SVGD.
result Well-calibrated uncertainty estimates and posterior contraction with increasing data.
This paper accelerates distributed convex optimization by mitigating ill-conditioning issues.
problem Distributed convex optimization with ill-conditioned aggregate cost functions.
method Iterative pre-conditioning technique to improve convergence rate and stability.
result The proposed algorithm converges linearly with improved convergence rate and superlinearly under certain conditions.
A new optimization method reduces memory and compute requirements for deep learning.
problem Memory and compute constraints in second-order stochastic optimizers for deep learning.
method Proposes KrAD, a novel factorization to approximate inverse Fisher matrix without inversion, leading to KrADagrad.
result Improves performance over Shampoo for 32-bit precision and comparable/generalization on real datasets.
AI-driven framework optimizes MCMC-based preconditioners for faster linear system solving.
problem Slow convergence of Krylov subspace solvers for ill-conditioned matrices.
method Graph neural surrogate and Bayesian optimization for AI-tuned MCMC parameters.
result 50% reduction in iterations to convergence on unseen system.
New algorithms solve nonconvex-nonconcave minimax optimization problems.
problem Solving minimax optimization problems in machine learning.
method Two novel Newton-type algorithms for nonconvex-nonconcave minimax optimization.
result Proved local convergence at strict local minimax points.
ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.
problem Efficiently solving ill-conditioned low-rank matrix estimation problems.
method Scaled Gradient Descent (ScaledGD) with adaptive pre-conditioners.
result Linear convergence rate independent of condition number, low per-iteration cost.
The total variation (TV) penalty, as many other analysis-sparsity problems, does not lead to separable factors or a proximal operatorwith a closed-form expression, such as soft thresholding for the ℓ_1 penalty. As a result, in a variational formulation of an inverse problem or statisticallearning estimation, it l…
New algorithm reduces sample complexity for sparse linear regression.
problem Sparse linear regression with correlated covariates and approximate dependencies.
method Polynomial-time algorithm that adapts the Lasso to tolerate approximate dependencies.
result Achieves near-optimal sample complexity for constant sparsity and ill-conditioned covariates.
This study improves numeric data generation using constrained WGAN structures.
problem Overfitting and ill-conditioning in numeric data generation with GANs.
method Designs and evaluates constrained network structures (isomorphic, mirror, self-symmetric) in WGANs for numeric data generation.
result Constrained structures significantly improve numeric data generation in 17/20 experiments.
In this paper, we study large-scale convex optimization algorithms based on the Newton method applied to regularized generalized self-concordant losses, which include logistic regression and softmax regression. We first prove that our new simple scheme based on a sequence of problems with decreasing regularization para…
A new method for robust matrix completion overcomes limitations of existing approaches.
problem Robust matrix completion from corrupted entries, especially under overparameterization and ill-conditioning.
method Factorization-based iterative algorithm combining Gauss-Newton linearization and outlier removal.
result Theoretical guarantees of exact recovery for suitable assumptions.
New method speeds up Gaussian process training and inference for large datasets.
problem Training and inference in Gaussian processes are computationally expensive for large datasets.
method Iterative alternating projection method that accesses subblocks of the kernel matrix, reducing time and space complexity.
result Empirically, the method accelerates GP training and inference by up to 72x compared to conjugate gradients.
We present a machine learning approach to the inversion of Fredholm integrals of the first kind. The approach provides a natural regularization in cases where the inverse of the Fredholm kernel is ill-conditioned. It also provides an efficient and stable treatment of constraints. The key observation is that the stabili…
Gaussian Processes (GPs) are a popular approach to predict the output of a parameterized experiment. They have many applications in the field of Computer Experiments, in particular to perform sensitivity analysis, adaptive design of experiments and global optimization. Nearly all of the applications of GPs require the …
Stochastic gradient descent approximates Gaussian process posteriors efficiently.
problem Efficiently sampling from Gaussian process posteriors with limited computational resources.
method Developed stochastic gradient optimization objectives for sampling from Gaussian process posteriors.
result Stochastic gradient descent produces accurate predictive distributions, even in non-convergent cases.
New insights into convergence of optimization methods for DAG structure learning.
problem Unclear convergence properties of optimization methods for structure learning.
method Examined the convergence of augmented Lagrangian method (ALM) and quadratic penalty method (QPM) for structure learning.
result Standard convergence result of ALM does not hold in various cases, and QPM is prone to ill-conditioning.
Developed shrinkage methods for Poisson regression models with experts to handle multicollinearity.
problem Multicollinearity in Poisson regression models with experts.
method Ridge and Liu-type shrinkage methods.
result Shrinkage methods offer more reliable estimates for coefficients in multicollinearity.
New algorithm improves heteroskedastic PCA performance.
problem Estimating low-rank matrix subspace from noisy data.
method Deflated-HeteroPCA algorithm, dividing spectrum into subblocks.
result Near-optimal and condition-number-free statistical guarantees.
New insights into adversarial vulnerability linked to manifold separability issues.
problem Adversarial vulnerability in machine learning models.
method Characterized data distribution as a low-dimensional manifold, focusing on on/off manifold directions and the impact of first-order vs. second-order optimization methods.
result First-order optimization methods lead to poor convergence in the off-manifold direction, causing adversarial vulnerability in inseparable datasets.
New gradient methods solve multiscale optimization problems efficiently.
problem Minimizing functions with multiple non-interacting smooth, strongly convex components.
method Big-Step-Little-Step interleaving of standard methods.
result Complexity bound scales as product of square-roots of condition numbers of components, improving on accelerated gradient methods.
Consider Least Squares Monte Carlo (LSM) algorithm, which is proposed by Longstaff and Schwartz (2001) for pricing American style securities. This algorithm is based on the projection of the value of continuation onto a certain set of basis functions via the least squares problem. We analyze the stability of the algori…
New priors for deep neural networks converge to Gaussian processes.
problem Improving the performance and stability of deep neural networks.
method Extending prior distributions to include non-zero means and partially exchangeable priors, leading to a new Gaussian process model.
result The new Gaussian process model avoids pathologies and improves performance on regression problems.