A dissertation on scalable projection-free optimization methods.
problem Efficient optimization algorithms for large-scale machine learning problems.
method Study of Frank-Wolfe variants and their extensions to distributed and derivative-free settings.
result Development of 1-SFW and QFW, achieving state-of-the-art complexity and efficiency.
New algorithm reduces adaptive regret without projections.
problem Computational expense of projections in online convex optimization.
method Lazy gradient-based algorithm with set-membership computations.
result Near-optimal adaptive regret bounds for general convex functions.
Efficient algorithm for bandit convex optimization with sublinear regret.
problem Optimizing in unknown convex functions without projection.
method Projection-free algorithm achieving O ( n T 4 / 5 ) O(nT^{4/5}) O ( n T 4/5 ) sublinear regret. result Achieves O ( n T 4 / 5 ) O(nT^{4/5}) O ( n T 4/5 ) sublinear regret for bounded convex functions. Improved regret bounds for scalable bandit convex optimization.
problem Designing online algorithms for high-dimensional bandit convex optimization.
method Projection-free algorithms using a linear optimization oracle.
result First algorithm with O ( T 3 / 4 ) O(T^{3/4}) O ( T 3/4 ) expected regret in O ( T ) O(T) O ( T ) calls. New algorithm reduces online learning iterations by a factor of T^2/3.
problem Efficiency in online learning with smooth cost functions.
method Follow-the-Perturbed-Leader method using online primal-dual framework.
result Guaranteed T^2/3 regret for general online convex optimization.
Efficient boosting method for regression with limited feedback.
problem Online boosting for regression tasks with noisy multi-point bandit feedback.
method Efficient regret minimization method with online boosting algorithm and projection-free online convex optimization.
result Improved state-of-the-art guarantees in efficiency.
Paper extends FW algorithms for large-scale, high-dimensional statistical estimation.
problem High-dimensional statistical estimation with non-strongly convex objectives.
method Extends FW type algorithms to handle restricted strong convexity.
result Achieves log(1/ε) gradient complexity for large-scale estimation problems.
A new method reduces rank in Frank-Wolfe steps for nuclear norm problems.
problem High rank intermediate iterates in Frank-Wolfe algorithm for nuclear norm problems.
method Rank-drop steps to ensure rank decreases and feasibility.
result Reduced rank of solutions compared to Frank-Wolfe and variants.
Study efficient algorithms for nonconvex optimization with state-dependent Markov data.
problem Stochastic optimization with Markovian data and state-dependent transition kernels.
method Projection-based and projection-free algorithms for constrained nonconvex problems.
result The number of oracle calls to achieve an ε ε ε -stationary point is O ( 1 / ε 2.5 ) \mathcal{O}(1/ε^{2.5}) O ( 1/ ε 2.5 ) . Two new methods reduce OCO problem complexity without projections.
problem Efficiently solving smooth Online Convex Optimization problems without projections.
method ORGFW and MORGFW methods using recursive gradient estimation.
result Achieve optimal regret bounds with low computational costs.
Optimized algorithms for online learning with linear constraints improve performance and provide worst-case analysis.
problem Improving online learning algorithms for constrained optimization problems.
method Developed an optimized variant of an online Frank-Wolfe algorithm and used semidefinite programming for numerical analysis.
result No pure online Frank-Wolfe algorithm can have a better regret guarantee than O(T^3/4) without additional assumptions.
Paper tackles online learning on curved spaces without projections.
problem Online learning on Riemannian manifolds with computational constraints.
method Develops projection-free algorithms for geodesically convex optimization.
result Achieves sub-linear regret guarantees in online geodesically convex optimization.
New algorithm solves complex optimization problems without needing projections.
problem Optimizing nested functions under convex constraints with noisy evaluations.
method Projection-free conditional gradient-type algorithm for smooth stochastic multi-level composition optimization.
result The algorithm achieves ε ε ε -stationary solutions with complexity bounds independent of ε ε ε and T T T . New algorithms optimize actions under time-varying constraints without projecting.
problem Optimizing actions under time-varying constraints without projecting.
method Projection-free algorithms using linear optimization oracle.
result Guaranteed i l d e O ( T 3 / 4 ) ilde{O}(T^{3/4}) i l d e O ( T 3/4 ) regret and O ( T 7 / 8 ) O(T^{7/8}) O ( T 7/8 ) constraints violation. New neural network approach for projection-free optimization.
problem Feasibility constraints in optimization problems.
method Designing projection-free convex optimization algorithms as Frank-Wolfe Networks.
result LSTM-learned optimizers outperform hand-designed and unconstrained optimizers.
New method learns MDP policies without projections, achieving near optimal results.
problem Learning policies for large MDPs with high sample complexity.
method Projection-free stochastic primal-dual method for approximate linear programming.
result PAC sample complexity analysis and improved efficiency compared to existing methods.
Quantized Frank-Wolfe reduces communication costs in distributed optimization.
problem Efficiently reducing communication overhead in distributed machine learning optimization.
method Quantized Frank-Wolfe (QFW), a projection-free algorithm for constrained optimization.
result Strong theoretical guarantees on convergence rate, efficient compression of gradients.
New framework for decentralized optimization of upper-linearizable functions with improved regret and complexity.
problem Decentralized optimization of upper-linearizable functions with general constraints.
method Decentralized projection-free optimization with upper-linearizable function framework.
result Regret of O ( T 1 − θ / 2 ) O(T^{1-θ/2}) O ( T 1 − θ /2 ) with communication complexity of O ( T θ ) O(T^θ) O ( T θ ) and linear optimization calls of O ( T 2 θ ) O(T^{2θ}) O ( T 2 θ ) . Algorithm identifies best arm in combinatorial bandits with semi-bandit feedback.
problem Identifying the best arm in combinatorial bandits with semi-bandit feedback.
method Interpreted as a sequential zero-sum game, developed a CombGame meta-algorithm with finite time guarantees.
result First computationally efficient algorithm that is asymptotically optimal and has competitive empirical performance.
Improved SGD algorithm with faster convergence.
problem Optimization of machine learning models.
method Conditional accelerated lazy stochastic gradient descent.
result Convergence rate of $O\left(\frac{1}{\varepsilon^2}
ight)$ , faster than previous methods.
New theorem connects diverse machine learning algorithms using Bregman divergences.
problem Design and analysis of machine learning algorithms.
method Scaled Bregman theorem involving Bregman divergences.
result Allows re-writing certain distortions as scaled Bregman divergences.
Improved algorithm reduces communication rounds for distributed online learning.
problem Complicated constraints in distributed online learning with locally light computations.
method Proposed D-BOCG algorithm with delayed update mechanism and redefined surrogate loss function.
result Achieved O ( T 3 / 4 ) O(T^{3/4}) O ( T 3/4 ) regret bound with O ( T ) O(\sqrt{T}) O ( T ) communication rounds for convex losses. Two new online optimization algorithms tackle convex and submodular problems without projections or exact gradients.
problem Efficiently optimize non-convex functions like submodular functions under computational constraints.
method Meta-Frank-Wolfe and One-Shot Frank-Wolfe algorithms using stochastic gradient estimates.
result Achieve optimal adversarial regret bounds for convex and continuous submodular optimization.
Improved online learning with time-varying constraints for complex domains.
problem Constrained online convex optimization with time-varying constraints.
method Constructing a composite surrogate loss and using the online Frank-Wolfe method.
result Novel regret and cumulative constraint violation bounds for strongly convex losses.
Improved TD learning with neural nets reduces sample complexity and overparameterization.
problem Temporal difference learning with neural networks in large state spaces.
method Projection-free and max-norm regularized Neural TD learning, with Lyapunov drift analysis.
result Max-norm regularization significantly improves TD learning's sample complexity and overparameterization.
A new algorithm for solving constrained convex optimization problems efficiently.
problem Constrained convex optimization problems requiring high accuracy solutions.
method Second-Order Conditional Gradient Sliding (SOCGS) algorithm, using projection-free methods to solve quadratic subproblems inexactly.
result Converges quadratically in primal gap after a finite number of linearly convergent iterations.
A distributed algorithm for online multi-task learning reduces communication and runtime costs.
problem Heavy communication and high runtime complexity in online multi-task learning.
method Adaptive primal-dual algorithm that synchronizes data across geographically distributed tasks.
result The proposed algorithm achieves optimal regret and is effective on real-world datasets.
Lloyd's K-means is shown to be a Frank-Wolfe algorithm variant.
problem Optimizing the sum of squared errors in clustering.
method Established a connection between Lloyd's K-means and Frank-Wolfe algorithm, derived convergence rates, and developed FW variants for empty clusters.
result Lloyd's K-means is a special case of the Frank-Wolfe algorithm with a non-asymptotic convergence rate of O(1/t).
Local LMO optimizes constrained problems using local linear minimization.
problem Constrained optimization problems with complex feasible sets.
method Designs a new projection-free gradient method using local linear minimization.
result Transfers convergence rates of Projected Gradient Descent to the projection-free world.
Two new Frank-Wolfe algorithms improve convergence for constrained optimization.
problem Solving optimization problems with structured constraints in machine learning.
method Two new variants of the Frank-Wolfe (FW) method for stochastic finite-sum minimization.
result Best convergence guarantees for convex and non-convex objective functions.
PF-LaCG removes the need for knowing smoothness and strong convexity parameters for locally accelerated CG.
problem Locally accelerated CG requires knowledge of smoothness and strong convexity parameters.
method Parameter-Free Locally Accelerated CG (PF-LaCG) algorithm.
result PF-LaCG achieves local acceleration without requiring knowledge of smoothness and strong convexity parameters.
New algorithm for online optimization over symmetric cones, unifying previous methods.
problem Online convex optimization over symmetric cones.
method Symmetric-Cone Multiplicative Weights Update (SCMWU) algorithm.
result SCMWU is a no-regret algorithm.
New nonconvex Frank-Wolfe methods for faster optimization.
problem Nonconvex optimization problems in machine learning.
method Stochastic Frank-Wolfe methods for nonconvex optimization.
result Improved convergence rates for nonconvex optimization.
Locally Accelerated Conditional Gradients improve convergence rates for smooth convex optimization problems.
problem Achieving optimal convergence rates for smooth convex optimization problems over polytopes.
method Locally Accelerated Conditional Gradients, coupling accelerated steps with conditional gradient steps.
result Achieves optimal accelerated local convergence for smooth strongly convex problems.
We produce a new proof and extend results by Harrell and Stubbe for the discrete spectrum of a self-adjoint operator. An abstract approach--based on commutator algebra, the Rayleigh-Ritz principle, and an ``optimal'' usage of the Cauchy-Schwarz inequality--is used to produce ``parameter-free'', ``projection-free'' vers…
Stochastic Frank-Wolfe method solves large-scale Lasso problems efficiently.
problem Sparse model optimization for large-scale Lasso regression.
method Randomized Stochastic Frank-Wolfe algorithm with convergence guarantees.
result Algorithm outperforms state-of-the-art methods on large datasets.
A new decentralized Frank-Wolfe algorithm tackles high-dimensional constrained optimization problems.
problem High-dimensional constrained optimization problems in decentralized settings.
method Projection-free optimization approach using Frank-Wolfe algorithm.
result The DeFW algorithm converges with rates for convex, strongly convex, and non-convex objectives.
Paper develops zeroth and first order stochastic Frank-Wolfe algorithms for constrained optimization.
problem Optimization problems with difficult-to-project deterministic constraints and efficient projection constraints.
method Stochastic Frank-Wolfe algorithms with momentum and trimmed variants.
result Guaranteed fast convergence rates comparable to unconstrained problems.
New method tackles online DR-submodular maximization with improved regret guarantees.
problem Online maximization of non-monotone DR-submodular functions over down-closed convex sets.
method 1/e-linearization through exponential reparametrization, surrogate potential, and reduction to online linear optimization.
result Achieves O ( T 1 / 2 ) O(T^{1/2}) O ( T 1/2 ) static regret with single gradient query per round, improving state of the art. Bipartite networks are a common type of network data in which there are two types of vertices, and only vertices of different types can be connected. While bipartite networks exhibit community structure like their unipartite counterparts, existing approaches to bipartite community detection have drawbacks, including im…
Algorithm identifies correct hypothesis from alternatives in bandit problems.
problem Efficiently identifying the correct hypothesis from a finite set of alternatives in structured stochastic multi-armed bandits.
method Frank-Wolfe Self-Play (FWSP) reformulates the game as a saddle-point problem, using a differential-inclusion argument to prove convergence.
result Convergence of the game value for best-arm identification in linear bandits, with uniform global convergence to the optimal value.
This work tackles regression on non-Euclidean spaces, specifically positive-definite matrices with the Bures-Wasserstein metric.
problem Regression on non-Euclidean spaces, specifically positive-definite matrices with the Bures-Wasserstein metric.
method Developed a sufficient condition for the existence of a minimizer of the conditional barycenter problem, characterized the optimization landscape, and developed a projection-free algorithm for approximate computation of first-order stationary points.
result The objective is free of local maxima under the sufficient condition, and the algorithm enables the use of stochastic Riemannian optimization methods for large-scale setups.
Robust CG methods avoid data corruption and solve structured statistical estimation problems.
problem Data corruption and heavy-tailed data in structured statistical estimation.
method Robustification of Conditional Gradient (CG) type methods using Huber's corruption model and robust mean gradient estimation.
result Robust CG methods converge linearly with correct sample complexity, even for high-dimensional problems.
FedSGM tackles constrained federated learning with unified framework.
problem Functional constraints, communication bottlenecks, local updates, and partial client participation in federated learning.
method Unified framework based on switching gradient method, incorporating bi-directional error feedback, and soft switching for stability.
result Achieves O ( 1 / T ) \boldsymbol{\mathcal{O}}(1/\sqrt{T}) O ( 1/ T ) convergence rate with high-probability bounds decoupling from sampling noise. A fast metric learning framework using Gershgorin disc alignment.
problem Learning effective metrics for graph-based data.
method Fast projection-free metric learning via Gershgorin disc alignment.
result Efficiently computed graph metric matrices outperform competing methods.
New methods optimize complex optimization problems with improved efficiency.
problem Optimizing complex problems with a convex lower-level objective.
method Uses stochastic cutting planes and conditional gradient updates.
result Improves complexity for both convex and non-convex upper-level functions.
A new metric learning framework for signed graphs using Gershgorin disc alignment.
problem Learning Mahalanobis metrics from signed graphs efficiently.
method Proposes a fast metric learning framework using Gershgorin disc perfect alignment (GDPA) to circumvent full eigen-decomposition.
result Proves that Gershgorin disc left-ends of similarity transform are perfectly aligned at the smallest eigenvalue, enabling efficient optimization.
This paper shows hedging algorithms improve performance in repeated matrix games.
problem Improving multi-agent learning algorithms in repeated matrix games.
method Develops and experiments with hedging algorithms combining a top-level and a set of basic algorithms.
result Well-selected hedging algorithms outperform previous MAL algorithms on repeated matrix games.