Riemannian Proximal Sampler improves sampling on manifold data.
problem Sampling from densities on Riemannian manifolds.
method Uses MBI and RHK oracles for high-accuracy sampling.
result Sampling with ε-accuracy requires O(log(1/ε)) iterations in KL divergence.
New study shows Gaussian samplers struggle with heavy-tailed targets, while stable samplers excel.
problem The difficulty of sampling from heavy-tailed distributions using Gaussian versus stable oracles.
method Comparison of Gaussian and stable oracles for proximal samplers.
result Gaussian samplers have a fundamental barrier for high-accuracy guarantees in heavy-tailed sampling, while stable samplers excel.
New algorithms solve large-scale low-rank and nonsmooth optimization problems efficiently.
problem Solving large-scale composite convex optimization problems with nonsmooth and low-rank terms.
method Stochastic optimization algorithms combining variance reduction and weak proximal oracle.
result First algorithm with nearly optimal sample complexity, single low-rank SVD per iteration, and log 1 / ε \log{1/ε} log 1/ ε thin-SVD computations. Paper establishes lower bounds for optimization of convex functions.
problem Lower bounds for optimization of convex functions with gradient and proximal oracle access.
method Developed a novel construction to prove lower bounds for strongly-convex case.
result Lower bound matches upper bound of existing algorithm Point-SAGA.
New research shows label refinement and weak training have limitations for aligning LLMs.
problem Limitations of refinement methods for aligning large language models.
method Analyzed probabilistic assumptions and alternative approaches to label refinement and weak training.
result Label refinement and weak training suffer from irreducible error, leaving a performance gap.
SpiderBoost improves SPIDER's efficiency and applicability in optimization.
problem Optimization of smooth nonconvex functions and handling nonsmooth regularizers.
method SpiderBoost uses a larger constant-level stepsize and proximal mapping for composite optimization, achieving improved oracle complexity.
result SpiderBoost achieves an oracle complexity of O ( min { n 1 / 2 ε − 2 , ε − 3 } ) \mathcal{O}(\min\{n^{1/2}ε^{-2},ε^{-3}\}) O ( min { n 1/2 ε − 2 , ε − 3 }) in composite nonconvex optimization. Study on tradeoffs between mistakes and ERM oracle calls in online and transductive learning.
problem Analyzing online and transductive learning with limited ERM and weak consistency oracle access.
method Proves lower bounds and upper bounds on mistakes and oracle calls, considering realizable and agnostic cases.
result Achieves optimal mistake bounds with weak consistency queries for certain concept classes.
New sampling methods for constrained and composite distributions.
problem Sampling from log-concave distributions with constraints and composite structures.
method Proximal sampler applied to lifted convex sets with separation and subgradient oracles.
result Practical and unbiased samplers for constrained and composite distributions.
Semi-supervised active clustering (SSAC) utilizes the knowledge of a domain expert to cluster data points by interactively making pairwise "same-cluster" queries. However, it is impractical to ask human oracles to answer every pairwise query. In this paper, we study the influence of allowing "not-sure" answers from a w…
Pairwise "same-cluster" queries are one of the most widely used forms of supervision in semi-supervised clustering. However, it is impractical to ask human oracles to answer every query correctly. In this paper, we study the influence of allowing "not-sure" answers from a weak oracle and propose an effective algorithm …
An algorithm learns from multiple models to match an oracle's risk.
problem Learning from multiple noisy models to estimate a target parameter.
method Elimination rounds algorithm for adaptive learning.
result Risk of weak-oracle learner matches that of an oracle in multiple source case.
We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly non-differentiable but convex component. We propose a proximal stochastic gradie…
New algorithm tackles complex optimization problems with inexact and stochastic methods.
problem Solving complex optimization problems with inexact and stochastic methods.
method Developed ICGALP algorithm for composite minimization problems with inexact computations.
result Convergence of Lagrangian to an optimum and asymptotic feasibility of the affine constraint.
The paper addresses nonconvex penalized LAD estimation in partial linear models using DNNs.
problem Challenges in nonconvex penalized LAD estimation with DNNs in partial linear models.
method Parameterizes nonparametric term with DNNs, formulates penalized LAD problem, introduces proximal subgradient method.
result Establishes consistency, convergence rate, and asymptotic normality of the estimator.
Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a functional variable. This paper proposes to build upon the proximal point algorithm, wh…
There is increasing interest in learning algorithms that involve interaction between human and machine. Comparison-based queries are among the most natural ways to get feedback from humans. A challenge in designing comparison-based interactive learning algorithms is coping with noisy answers. The most common fix is to …
Paper establishes lower bounds for finite-sum optimization problems using novel construction methods.
problem Lower complexity bounds for finite-sum optimization problems with various component functions.
method Developed novel approach to construct hard instances and analyzed PIFO algorithms.
result Established lower complexity bounds for convex-concave and nonconvex-strongly-concave objectives.
New algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with noisy stochastic data.
method Stochastic proximal point algorithm with weak linear regularity condition.
result Achieves $\mathcal{O}\left(\frac{1}{k}
ight)$ convergence rate for SPP.
New algorithms sample convex bodies using Markov chains and restricted Gaussian oracles.
problem Sampling uniformly from convex bodies efficiently.
method Markov chain Monte Carlo with proximal sampler and restricted Gaussian oracle.
result Efficient implementation of RGO for uniform sampling on convex bodies.
An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses…
New algorithms optimize convex functions with high-order derivatives.
problem Optimizing convex functions with high-order derivatives under various norms.
method Developed a non-Euclidean inexact accelerated proximal point method using an inexact uniformly convex regularizer.
result Showed nearly optimal algorithms for high dimensions in the black-box oracle model for ℓ p \ell_p ℓ p -settings and all q ≥ 1 q \geq 1 q ≥ 1 . Algorithm samples from composite log-concave distributions using gradient evaluations and restricted Gaussian oracles.
problem Sampling from composite log-concave distributions with limited gradient evaluations.
method Proximal gradient algorithm with RGO for g g g and strong/strongly convex conditions for f f f . result Achieves ε ε ε error in total variation distance in O ~ ( κ d log 4 ( 1 / ε ) ) \widetilde{\mathcal O}(κ\sqrt d \log^4(1/ε)) O ( κ d log 4 ( 1/ ε )) iterations. Sparse-penalized deep neural networks improve performance in weakly dependent processes.
problem Nonparametric regression and classification under weak dependence.
method Sparse-penalized deep neural networks with oracle inequalities and convergence rates established.
result The proposed estimators outperform non-penalized ones in simulations.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
Study shows tractable generalization in RL is impossible but possible with Strong Proximity.
problem RL agents struggle to generalize to new environments.
method Introduced Weak and Strong Proximity conditions to capture similarity between environments.
result Proved Strong Proximity is sufficient for efficient generalization.
Extracts fairness truth from classifiers using an oracle.
problem Fairness in classification algorithms without context.
method Uses fairness oracle to learn underlying fairness truth.
result Extracts metric fairness from classifiers.
Estimates CATEs for structured treatments using a new decomposition method.
problem Estimating conditional average treatment effects for complex data types.
method Generalized Robinson decomposition, isolating causal estimand, arbitrary model plugging, quasi-oracle convergence guarantee.
result Demonstrates superior performance in CATE estimation compared to prior work.
Efficient algorithm solves sparse nonconvex regression problems.
problem Sparse nonconvex square-root-loss regression problems.
method Proximal majorization-minimization (PMM) algorithm with sparse semismooth Newton method.
result Converges to a d-stationary point with Kurdyka-Łojasiewicz property.
Algorithm samples composite logconcave densities efficiently.
problem Sampling from composite logconcave densities efficiently.
method Uses a restricted Gaussian oracle and gradient queries.
result Achieves strong total variation distance guarantees.
In this paper, we develop an approach to recursively estimate the quadratic risk for matrix recovery problems regularized with spectral functions. Toward this end, in the spirit of the SURE theory, a key step is to compute the (weak) derivative and divergence of a solution with respect to the observations. As such a so…
Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namely the accelerated stochastic compositional proximal gradient (ASC-PG) method, which updates based on queries to the sampling oracle using tw…
Online algorithm identifies PDEs from noisy data snapshots.
problem Identifying PDEs from sequential solution snapshots.
method Combines weak-form discretization with online proximal gradient descent.
result Efficiently identifies and tracks systems with time-varying coefficients.
Paper proposes a new combined regression strategy for conditional survival prediction.
problem Improving survival prediction accuracy using conditional survival function.
method Uses regression-based weak learners with area-norm proximity measure to create an ensemble technique.
result The proposed model outperforms Random Survival Forest and selects important variables effectively.
Improved sampling algorithm with state-of-the-art complexity bounds.
problem Efficient sampling from various probability distributions.
method Proximal sampler with inexact restricted Gaussian oracle.
result State-of-the-art complexity bounds in almost all settings.
New algorithms sample structured logconcave families with improved efficiency.
problem Sampling structured logconcave families to high accuracy.
method Reduction framework inspired by proximal point methods, combined with restricted Gaussian oracles.
result Improved bounds for sampling structured distributions, matching or surpassing state-of-the-art results.
Paper improves a method for fast global and local convergence in optimization.
problem Slow global convergence in optimization methods with noisy Hessian estimates.
method Stochastic Newton Proximal Extragradient method using HPE framework.
result Faster global linear rate and superlinear convergence in fewer iterations.
Paper proposes a new method for supervised manifold learning using random forest proximities.
problem Existing supervised manifold learning methods fail to uncover meaningful embeddings due to using class-conditional distances.
method Proposes a data-geometry-preserving variant of random forest proximities as an initialization for manifold learning methods.
result Local and global structure preservation is near universal across manifold learning approaches using diffusion-based algorithms.
Minimizing a function over an intersection of convex sets is an important task in optimization that is often much more challenging than minimizing it over each individual constraint set. While traditional methods such as Frank-Wolfe (FW) or proximal gradient descent assume access to a linear or quadratic oracle on the …
Two new methods solve nonsmooth optimization on Riemannian Stiefel manifold.
problem Optimization over nonsmooth, non-differentiable functions on Riemannian manifolds.
method R-ProxSGD and R-ProxSPB, generalizing proximal SGD and SpiderBoost.
result R-ProxSPB finds ε-stationary points with IFO complexity of Ø(ε^(-3)) in online and Ø(n + √nε^(-2)) in finite-sum cases.
MAMBA learns policies competitive with multiple conflicting oracles.
problem Learning policies from multiple conflicting oracles in reinforcement learning.
method MAMBA uses a gradient estimator in the style of GAE to optimize policies, leveraging demonstrations from multiple weak oracles.
result MAMBA outperforms the state-of-the-art in learning policies competitive with multiple conflicting oracles.
Develops methods for solving convex optimization problems with improved accuracy and convergence.
problem Solving stochastic convex optimization problems with improved accuracy and robustness.
method Approximate-proximal point (aProx) family, including stochastic subgradient, proximal point, and bundle methods.
result Improved models converge with probability 1 and enjoy optimal asymptotic normality results under weak assumptions.
New algorithm speeds up sampling from complex distributions.
problem Efficiently sampling from non-log-concave distributions.
method Stochastic Proximal Samplers (SPS) based on SGLD and MALA.
result SPS-SGLD and SPS-MALA achieve faster sampling with reduced gradient complexity.
ASVRG accelerates stochastic variance reduction methods with simplicity and efficiency.
problem Efficiently solving convex and non-convex optimization problems.
method Accelerated proximal stochastic variance reduced gradient (ASVRG) method with momentum acceleration.
result ASVRG achieves best known oracle complexities for strongly and non-strongly convex objectives.
New algorithms solve stochastic variational inequalities without bounded variance assumption.
problem Solving stochastic variational inequalities without bounded variance assumption.
method Developed algorithms for two classes of problems: monotone and structured nonmonotone VIs.
result Oracle complexity of O(ε^-4) for solving VIs with unbounded domains and possibly unbounded variance.
Paper develops robust regression method for heavy-tailed errors.
problem High-dimensional robust regression with heavy-tailed errors.
method Iteratively reweighted ℓ 1 \ell_1 ℓ 1 -penalized adaptive Huber regression. result Oracle convergence rate and variable selection consistency achieved.
A new hybrid algorithm reduces stochastic gradient evaluations for nonconvex optimization.
problem Solving stochastic composite nonconvex optimization problems efficiently.
method Proposes a new hybrid variance-reduced proximal gradient method with a stochastic gradient estimator.
result Achieves optimal stochastic oracle complexity bound with one less gradient evaluation.
New method controls gradient error for sparse MRFs.
problem Efficient learning for sparse discrete MRFs with NP-hard inference.
method Stochastic proximal gradient (SPG) with controlled gradient approximation error.
result Novel bounds control gradient approximation quality.
The paper introduces vortex nerve complexes and new Betti numbers in CW spaces.
problem Understanding the structure and properties of CW complexes and their nerves.
method Introducing vortex nerve complexes and defining new Betti numbers for CW complexes.
result New Betti numbers (vortex B v t e x \mathcal{B}_{vtex} B v t e x , vortex nerve B v N r v \mathcal{B}_{vNrv} B v N r v , shape B s h \mathcal{B}_{sh} B s h ) are introduced and studied.