New methods find eigenvectors faster than Lanczos's method.
problem Finding the leading eigenvector efficiently.
method Coordinate-wise methods combining shift-and-invert with linear regression.
result Global convergence with runtime guarantees better than Lanczos's method.
Proposes blockwise adaptive stepsize for faster training and better generalization in deep learning.
problem Widespread use of coordinate-wise adaptive methods like RMSprop and Adam leads to worse generalization than SGD.
method Splits network parameters into blocks and uses a blockwise adaptive stepsize, balancing adaptivity and generalization.
result Blockwise adaptive gradient descent converges faster and has lower generalization error than coordinate-wise adaptive methods.
Proposes a new Armijo's condition for coordinate-wise functions.
problem Finding optimal step sizes in coordinate-wise optimization.
method Introduces a new Armijo's condition for functions defined on product spaces.
result Shows the advantage of the new condition over the standard Armijo's condition.
A new framework improves solving mixed-integer convex problems with binary indicators.
problem Optimizing mixed-integer convex problems with binary indicators controlling continuous variables.
method Coordinate Optimality Reformulation (CORe) framework, incorporating coordinate-wise optimality information.
result CORe reformulations improve branch-and-bound performance, especially in sparse and structured settings.
AdaCliP reduces noise in private SGD training.
problem Privacy preserving machine learning over user data.
method Adaptive clipping of gradients to reduce noise in private SGD.
result AdaCliP adds less noise and improves model accuracy.
Proposes a new Armijo's condition for general functions and provides an algorithm for its application.
problem Finding suitable step sizes for general functions in optimization.
method Introduces a coordinate-wise Armijo's condition and provides an algorithm to find suitable step sizes.
result Proves convergent results for various functions using the proposed algorithm.
New algorithm resists up to half of Byzantine workers in distributed learning.
problem Resilience of distributed SGD in the presence of Byzantine attackers.
method Lipschitz-inspired coordinate-wise median approach (LICM-SGD).
result LICM-SGD can resist up to half of Byzantine workers in non-convex settings.
Scale-invariant algorithms for unconstrained online learning.
problem Designing online algorithms invariant to arbitrary linear transformations of input vectors.
method Exploiting scale invariance symmetry, developing algorithms for coordinate-wise and general invariance.
result Achieved optimal regret bound for coordinate-wise invariance, and almost achieved it for general invariance with logarithmic overhead.
New DP-CD method outperforms DP-SGD in solving composite DP-ERM problems.
problem Privacy-preserving machine learning with differential privacy.
method Differentially Private proximal Coordinate Descent (DP-CD) for composite Empirical Risk Minimization (ERM).
result DP-CD outperforms DP-SGD due to larger step sizes and better gradient exploitation.
Optimal algorithms for continuous non-monotone submodular and DR-submodular maximization.
problem Maximizing continuous non-monotone submodular and DR-submodular functions.
method Developed novel algorithms for both continuous submodular and DR-submodular maximization problems.
result First $ rac{1}{2}$-approximation algorithm for continuous submodular maximization.
Sparse high dimensional graphical model selection is a popular topic in contemporary machine learning. To this end, various useful approaches have been proposed in the context of ℓ1-penalized estimation in the Gaussian framework. Though many of these inverse covariance estimation approaches are demonstrably scala…
Advanced deep learning model improves speech enhancement by estimating phase accurately.
problem Difficulty in estimating the phase of clean speech in speech enhancement.
method Proposes Deep Complex U-Net, polar coordinate-wise complex-valued masking, and wSDR loss function.
result Achieves state-of-the-art performance in all metrics, outperforming previous approaches.
A new algorithm improves posterior sampling for linear inverse problems.
problem Efficiently sampling from posterior distributions in noisy linear inverse problems.
method Proposes \pddim, a DDIM-type sampler that separately samples along singular directions of the measurement operator.
result The method converges to the Bayesian posterior conditioned on the measurements.
Trimmed Lasso offers sparse modeling with robustness control.
problem Sparse modeling in linear regression with robustness.
method Trimmed Lasso penalty function and its analysis.
result Trimmed Lasso offers exact sparsity control and robustness.
In this short report, we discuss how coordinate-wise descent algorithms can be used to solve minimum variance portfolio (MVP) problems in which the portfolio weights are constrained by lq norms, where 1≤q≤2. A portfolio which weights are regularised by such norms is called a sparse portfolio (Brodie et …
PROBE algorithm efficiently solves sparse high-dimensional linear regression.
problem Sparse high-dimensional linear regression models with complex parameter spaces.
method Partitioned empirical Bayes ECM algorithm for computationally efficient MAP estimation.
result PROBE algorithm provides robust and efficient coordinate-wise optimization.
New attacks break robust aggregation methods for SGD in Byzantine-tolerant systems.
problem Breaking Byzantine-tolerant techniques in distributed machine learning.
method Inner product manipulation to break robust aggregation methods (median and Krum).
result Coordinate-wise median and Krum can be broken using new attack strategies.
In this paper we generalize the framework of the feasible descent method (FDM) to a randomized (R-FDM) and a coordinate-wise random feasible descent method (RC-FDM) framework. We show that the famous SDCA algorithm for optimizing the SVM dual problem, or the stochastic coordinate descent method for the LASSO problem, f…
Paper tackles Byzantine attacks in Federated Learning by clustering and robustifying.
problem Adversarial attacks from Byzantine machines in Federated Learning.
method Iterative Federated Clustering Algorithm (IFCA) with trimmed mean and median aggregation.
result Improved convergence rate for strongly convex loss functions in Byzantine-Robust IFCA.
New bound on Rademacher complexity for vector functions.
problem Bounding Rademacher complexity for vector-valued functions.
method Bounding Rademacher complexity by coordinate-wise complexity with a factor of sqrt(K).
result Rademacher complexity is bounded by the maximum coordinate-wise complexity times sqrt(K).
We present an approach for penalized tensor decomposition (PTD) that estimates smoothly varying latent factors in multi-way data. This generalizes existing work on sparse tensor decomposition and penalized matrix decompositions, in a manner parallel to the generalized lasso for regression and smoothing problems. Our ap…
Note on subgaussian bounds for sign-quantized linear maps.
problem Understanding subgaussian behavior of sign-quantized linear maps.
method Developed a dimension-independent subgaussian concentration bound for Gaussian vectors under nonlinear mappings.
result Answered a question about sign-quantized linear maps using a new subgaussian bound.
AdaGrad outperforms SGD in non-convex optimization problems by a factor of d.
problem Finding near-stationary points in stochastic non-convex optimization.
method Refined assumptions on smoothness and gradient noise variance, l1-norm stationarity measure. result AdaGrad achieves a convergence rate favorable over SGD in certain non-convex settings.
We design a randomised parallel version of Adaboost based on previous studies on parallel coordinate descent. The algorithm uses the fact that the logarithm of the exponential loss is a function with coordinate-wise Lipschitz continuous gradient, in order to define the step lengths. We provide the proof of convergence …
A method for constructing tight prediction intervals for multiple numerical outputs.
problem Constructing tight prediction intervals for multiple related numerical outputs.
method A novel coordinate-wise standardization procedure that makes residuals comparable across output dimensions, estimating suitable scaling parameters using calibration data.
result The method produces tighter prediction intervals than existing baselines while maintaining valid simultaneous coverage.
Differentially private random block coordinate descent improves utility in machine learning.
problem Lack of privacy in classical CD methods when handling sensitive information.
method Proposes a differentially private random block coordinate descent method using sketch matrices and importance sampling.
result Demonstrates improved convergence rates and utility guarantees compared to non-private methods.
New framework improves EM algorithm convergence under log-Sobolev inequality.
problem Improving convergence of the EM algorithm.
method Extending gradient flow techniques to EM algorithm, using free energy representation.
result Exponential convergence of EM algorithm under log-Sobolev inequality.
The paper introduces a new method for feature selection without explicit sparsification.
problem Feature selection with implicit sparsity-inducing mechanisms.
method Optimization over a family of kernels with gradient descent.
result The method achieves exact sparsity without explicit penalization techniques.
FedDuA adapts global learning rate for federated learning.
problem Slow convergence in federated learning due to dataset and parameter space heterogeneity.
method FedDuA uses mirror descent to adaptively select global learning rate based on inter-client and coordinate-wise heterogeneity.
result FedDuA achieves minimax optimal convergence for convex objectives and outperforms baselines in various settings.
Paper tackles distributed linear regression with compositional covariates.
problem Solving distributed statistical methodology and computing for massive compositional data.
method Proposes two distributed optimization techniques based on ADMM and CDMM for solving constrained convex optimization problems.
result Established convergence theories for the proposed algorithms under regularity conditions.
Paper explores differential privacy in high-dimensional federated learning, tackling server trustworthiness and estimation.
problem Maintaining privacy in distributed environments with high-dimensional data.
method Investigates scenarios with untrusted and trusted central servers, introduces novel federated estimation algorithms for linear regression models.
result Tight minimax rates depend on high-dimensionality even with sparsity assumptions, and novel algorithms handle slight variations among distributed models.
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
A new method for streaming PCA provides confidence intervals for eigenvector entries.
problem Uncertainty quantification for individual entries in streaming PCA.
method Oja's algorithm, Bernstein-type concentration bound, Central Limit Theorem, subsampling algorithm.
result Sharp concentration bound and Central Limit Theorem for streaming PCA entries.
Extends PD-NJ-ODE to noisy observations and dependent observation times.
problem Predicting continuous-time stochastic processes with irregular and noisy observations.
method Extends PD-NJ-ODE to handle conditional independence and noisy observations.
result Theoretical guarantees and empirical examples for handling noisy observations and dependent observation times.
This paper introduces a new method to compare collections of distributions on manifolds and graphs.
problem Comparing collections of probability distributions over diverse domains.
method Intrinsic slicing construction for Wasserstein distances, Hilbert embedding, resampling, p-value combination.
result Powerful and well-calibrated p-values for comparing distributions on manifolds and graphs.
High throughput genetic sequencing arrays with thousands of measurements per sample and a great amount of related censored clinical data have increased demanding need for better measurement specific model selection. In this paper we establish strong oracle properties of nonconcave penalized methods for nonpolynomial (N…
For the problem of high-dimensional sparse linear regression, it is known that an ℓ0-based estimator can achieve a 1/n "fast" rate on the prediction error without any conditions on the design matrix, whereas in absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarante…
Decentralized learning for GLMs with feature distribution and network connectivity.
problem Optimizing generalized linear models in a decentralized network with feature partitioning.
method Chambolle--Pock primal--dual algorithm applied to an equivalent saddle-point formulation.
result Convergence rates for empirical risk minimization under Lipschitz and square root Lipschitz assumptions.
A simple log-transform fixes heavy-tailed data for generative models.
problem Standard generative models struggle with heavy-tailed data.
method Apply the soft-log transform to data before training and exponentiate samples after generation.
result Log-FM outperforms specialized baselines on multivariate benchmarks.
New method predicts state evolution for non-first-order algorithms on nonconvex problems.
problem Analyzing nonconvex optimization problems with random data.
method Developed a state evolution for a broader class of algorithms including first-order and saddle point updates.
result Established rigorous state evolution predictions and finite-sample guarantees for non-first-order methods.
In this paper, we consider the problem of recovering a sparse signal based on penalized least squares formulations. We develop a novel algorithm of primal-dual active set type for a class of nonconvex sparsity-promoting penalties, including ℓ0, bridge, smoothly clipped absolute deviation, capped ℓ1 and mini…
The paper analyzes high-dimensional linear regression using parametric empirical Bayes methods.
problem Estimation of i.i.d. priors in high-dimensional Bayesian linear regression with random design.
method Parametric empirical Bayes estimation, variational lower bound maximization, phase transition analysis.
result The vEB estimator is information theoretically optimal up to p=o(n2/3) but sub-optimal in higher dimensions. New algorithm detects and discards faulty updates in federated learning.
problem Byzantine failures, biased local datasets, and poisoning attacks in federated learning.
method Adaptive Federated Averaging with Hidden Markov Model for quality update detection.
result Significantly more robust to faulty, noisy, and malicious participants.
New framework controls FDR for grouped features in sequential models.
problem FDR control for grouped features in sequential models.
method Grouped-feature FDR control framework for sequential and grouped models using mirror statistics and Permutation SHAP.
result FDR control for low- and high-dimensional grouped linear models and improved power under correlated signals.
Study on Metropolis-within-Gibbs schemes for high-dimensional Bayesian models.
problem Improving the scalability of MCMC methods for complex Bayesian models.
method Relating convergence properties to conditional conductance for non-conjugate hierarchical models.
result Established dimension-free convergence results for Metropolis-within-Gibbs schemes.
New algorithm optimizes Bayesian network learning from Gaussian data.
problem Learning Bayesian networks from Gaussian observational data.
method Proposes a coordinate descent algorithm for ℓ0-penalized maximum likelihood estimation. result The algorithm converges to a coordinate-wise minimum and achieves optimal objective value as sample size increases.
New algorithm improves privacy in high-dimensional machine learning models.
problem Privacy issues in learning large machine learning models.
method Differentially private greedy coordinate descent (DP-GCD) algorithm.
result Achieves logarithmic dependence on dimension for quasi-sparse solutions.
AdaDPIGU improves privacy in deep learning by adaptively clipping and pruning gradients.
problem Privacy in deep learning models, especially in high-dimensional settings.
method Importance-based gradient updates, adaptive clipping, differentially private SGD.
result AdaDPIGU achieves high accuracy while maintaining privacy, outperforming non-private models.