Dual optimization connects ERM-fDR to normalization function.
problem Empirical risk minimization with f-divergence regularization.
method Dual formulation, Legendre-Fenchel transform, implicit function theorem, nonlinear ODE.
result Computational method to calculate normalization function efficiently.
Accelerates machine learning algorithms for sparse data.
problem Efficiently solving composite convex minimization problems.
method Accelerated dual-averaging primal-dual method for composite convex minimization.
result Demonstrates advantages in handling sparse data both theoretically and empirically.
We consider a generic convex optimization problem associated with regularized empirical risk minimization of linear predictors. The problem structure allows us to reformulate it as a convex-concave saddle point problem. We propose a stochastic primal-dual coordinate (SPDC) method, which alternates between maximizing ov…
We consider empirical risk minimization of linear predictors with convex loss functions. Such problems can be reformulated as convex-concave saddle point problems, and thus are well suitable for primal-dual first-order algorithms. However, primal-dual algorithms often require explicit strongly convex regularization in …
We propose a doubly stochastic primal-dual coordinate optimization algorithm for empirical risk minimization, which can be formulated as a bilinear saddle-point problem. In each iteration, our method randomly samples a block of coordinates of the primal and dual solutions to update. The linear convergence of our method…
The paper explores dual learning, a technique that improves machine translation and image transformation.
problem Understanding and improving dual learning's effectiveness and conditions.
method Theoretical analysis and algorithmic extension of dual learning.
result Multi-step dual learning boosts performance under mild conditions.
Simplified screening tests for data points in optimization.
problem Discarding irrelevant data points in empirical risk minimization.
method Designing loss functions and regularizing convex losses to induce sparsity, using ellipsoidal approximations.
result Automatic discarding of data samples without losing optimization guarantees.
New dual formulation reduces generalization error for ERM-fDR.
problem Generalization error in constrained optimization problems.
method Introduces a dual formulation of ERM-fDR using Legendre-Fenchel transform and implicit function theorem.
result Explicit characterizations of generalization error for algorithms under mild conditions.
In this paper, we study randomized reduction methods, which reduce high-dimensional features into low-dimensional space by randomized methods (e.g., random projection, random hashing), for large-scale high-dimensional classification. Previous theoretical results on randomized reduction methods hinge on strong assumptio…
This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSD…
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…
Unified view connects CoCoA and ADMM for distributed ERM.
problem Connection between CoCoA and ADMM for distributed ERM.
method Unified primal-dual perspective reformulation.
result Unified ADMM variants perform at least as good as CoCoA in ridge-regularized ERM.
Previous studies on stochastic primal-dual algorithms for solving min-max problems with faster convergence heavily rely on the bilinear structure of the problem, which restricts their applicability to a narrowed range of problems. The main contribution of this paper is the design and analysis of new stochastic primal-d…
New algorithm for federated learning with non-smooth regularizers.
problem Federated Learning with non-smooth composite optimization problems.
method Proposed Federated Dual Averaging (FedDualAvg) algorithm to overcome convergence issues.
result FedDualAvg outperforms other algorithms in federated composite optimization.
DualIV simplifies non-linear IV regression via dual formulation.
problem Non-linear instrumental variable regression with potential first-stage regression bottleneck.
method Dual formulation of non-linear IV regression as a convex-concave saddle-point problem, leading to a kernel-based algorithm with analytic solution.
result Empirical results show competitive performance compared to existing algorithms.
A new distributed optimization method for ERM problems.
problem Efficiently solving ERM problems with nonsmooth regularization in a distributed setting.
method Second-order distributed optimization using successive quadratic approximations and Hessian approximation.
result Global linear convergence for a broad range of non-strongly convex problems.
Paper proposes a method to recover point configurations from noisy distance data.
problem Recovering point configurations from noisy distance data.
method Robust Euclidean Distance Geometry via Dual Basis (RoDEoDB) algorithm.
result Exact recovery guarantees for point configuration and Gram matrix under mild conditions.
In this work we develop a new algorithm for regularized empirical risk minimization. Our method extends recent techniques of Shalev-Shwartz [02/2015], which enable a dual-free analysis of SDCA, to arbitrary mini-batching schemes. Moreover, our method is able to better utilize the information in the data defining the ER…
Paper develops efficient algorithms for robust optimization across multiple groups.
problem Minimizing maximal empirical risk across distinct groups in robust optimization.
method Develops ALEG and ALEM algorithms for two-level finite-sum convex-concave minimax optimization.
result Achieves ε-accuracy with complexity O(m√(nlnm/ε)) and outperforms state-of-the-art methods.
Paper introduces SGA for barycenter optimization in optimal transport.
problem Optimizing Wasserstein barycenter for discrete distributions.
method Sobolev gradient ascent algorithm tailored to Wasserstein geometry.
result SGA achieves convergence rate similar to subgradient descent.
We propose a variant of the Frank-Wolfe algorithm for solving a class of sparse/low-rank optimization problems. Our formulation includes Elastic Net, regularized SVMs and phase retrieval as special cases. The proposed Primal-Dual Block Frank-Wolfe algorithm reduces the per-iteration cost while maintaining linear conver…
In modern large-scale machine learning applications, the training data are often partitioned and stored on multiple machines. It is customary to employ the "data parallelism" approach, where the aggregated training loss is minimized without moving data across machines. In this paper, we introduce a novel distributed du…
We study dual volume sampling, a method for selecting k columns from an n x m short and wide matrix (n <= k <= m) such that the probability of selection is proportional to the volume spanned by the rows of the induced submatrix. This method was proposed by Avron and Boutsidis (2013), who showed it to be a promising met…
Random extrapolation speeds up coordinate descent for sparse and dense data.
problem Efficiently solving primal-dual coordinate descent for sparse and dense data.
method Adapts to sparsity and uses large step sizes for dense data, proving linear convergence under metric subregularity.
result Linear convergence under metric subregularity and optimal sublinear convergence rates in general convex-concave problems.
LEAD algorithm speeds up decentralized optimization with compression.
problem Slow convergence and stability issues in decentralized optimization with compression.
method Proposes the first linearly convergent decentralized algorithm with compression.
result First consensus error bound for coupled dynamics of primal and dual updates.
Decentralized learning for GLMs with feature distribution and network connectivity.
problem Optimizing generalized linear models in a decentralized network with feature partitioning.
method Chambolle--Pock primal--dual algorithm applied to an equivalent saddle-point formulation.
result Convergence rates for empirical risk minimization under Lipschitz and square root Lipschitz assumptions.
Ensembles improve classifier performance by reducing bias, not variance.
problem Improving classifier performance through ensemble methods.
method Extended bias-variance decomposition for classification tasks, introducing dual reparameterization.
result Ensembling reduces bias in classifiers, contrary to the traditional view.
Study efficient convergence of RL algorithm with function approximation.
problem Convergence of actor-critic algorithm with nonlinear function approximation.
method Stochastic gradient descent ascent with adaptive proximal term, Polyak-Łojasiewicz condition.
result First efficient convergence result with rate of O(sqrt{ln(N d G^2) / N}).
Paper tackles constrained learning with non-convex losses, overcoming challenges with new approach.
problem Challenges in learning with non-convex losses and statistical constraints.
method Learning in the empirical dual domain, bounding empirical duality gap.
result Established a constrained counterpart to classical learning theory.
We propose and analyze a method for semi-supervised learning from partially-labeled network-structured data. Our approach is based on a graph signal recovery interpretation under a clustering hypothesis that labels of data points belonging to the same well-connected subset (cluster) are similar valued. This lends natur…
Dual-T method improves transition matrix estimation in noisy label learning.
problem Large estimation error in noisy class posterior leads to poor transition matrix estimation.
method Introducing an intermediate class to avoid direct estimation of noisy class posterior, factorizing the transition matrix into two easier-to-estimate matrices.
result The dual-T estimator leads to better classification performances.
We present an efficient algorithm for maximum likelihood estimation (MLE) of exponential family models, with a general parametrization of the energy function that includes neural networks. We exploit the primal-dual view of the MLE with a kinetics augmented model to obtain an estimate associated with an adversarial dua…
Proposes an online method for solving non-convex DRO with KL regularization.
problem Solving distributionally robust optimization with non-convex objectives.
method Practical online stochastic methods for DRO with KL regularization, avoiding high-dimensional dual variables and online learning issues.
result Empirical studies show significant speedup and efficiency in training deep learning models.
DanSmp predicts stock movement using a hybrid-relational MKG and dual attention networks.
problem Predicting stock price trends in volatile financial markets.
method Constructs a bi-typed MKG with hybrid-relations and uses DanSmp, a dual attention network, to learn momentum spillover signals.
result DanSmp improves stock prediction accuracy using the MKG.
FeDualEx tackles saddle point optimization in federated learning with composite objectives.
problem Saddle point optimization with constraints and non-smooth regularization in federated learning.
method Federated Dual Extrapolation (FeDualEx) algorithm for saddle point optimization and composite objectives.
result FeDualEx effectively solves saddle point optimization problems with composite objectives in federated learning.
Dual learning algorithm addresses delayed conversions in CVR prediction.
problem Challenges in predicting conversion rate due to delayed feedback.
method Proposes two unbiased estimators and a dual learning algorithm.
result Demonstrates practical value of the proposed approach through empirical evaluations.
PDA method optimizes neural networks with global convergence rate analysis.
problem Quantitative convergence rate for neural network optimization in mean field regime.
method Particle dual averaging (PDA) method, combining Langevin algorithm and outer loop optimization.
result Established quantitative global convergence for two-layer mean field neural networks.
The paper explores how AI systems use information geometry to encode semantic structure.
problem How AI systems encode semantic structure into geometric representation spaces.
method Focuses on softmax distributions and develops dual steering method for robust concept manipulation.
result Dual steering optimally modifies target concepts while minimizing off-target changes.
We consider convex-concave saddle point problems with a separable structure and non-strongly convex functions. We propose an efficient stochastic block coordinate descent method using adaptive primal-dual updates, which enables flexible parallel optimization for large-scale problems. Our method shares the efficiency an…
Differential privacy is concerned about the prediction quality while measuring the privacy impact on individuals whose information is contained in the data. We consider differentially private risk minimization problems with regularizers that induce structured sparsity. These regularizers are known to be convex but they…
Dual model predicts electricity spot prices using neural networks and wavelets.
problem Forecasting hourly electricity spot prices.
method Dual generalized long memory modelling with k-factor GARMA and G-GARCH models, using LLWNN and PSO for variance prediction.
result The hybrid k-factor GARMA-LLWNN model outperforms other methods in forecasting accuracy.
Neural model accelerates SDDP for stochastic optimization.
problem Exponential complexity of SDDP limits its applicability to low-dimensional problems.
method Trainable neural model maps problem instances to a low-dimensional piecewise linear value function.
result ν-SDDP significantly reduces problem solving cost without sacrificing solution quality.
New method estimates Schrödinger bridge potentials via empirical risk minimization.
problem Estimating Schrödinger bridge potentials from samples.
method Rewriting Schrödinger system as a fixed-point equation and estimating the potential via empirical risk minimization.
result Uniform concentration of empirical risk around population counterpart under sub-Gaussian assumptions.
This paper improves inverse problem solving with weakly convex regularisers and proves convergence.
problem Improving solution methods for inverse problems.
method Generalised formulation of convergent regularisation using weakly convex regularisers, and proof of convergence for primal-dual hybrid gradient method.
result Proves convergence of primal-dual hybrid gradient method for variational problems and shows improved performance with IWCNNs.
A distributed algorithm for online multi-task learning reduces communication and runtime costs.
problem Heavy communication and high runtime complexity in online multi-task learning.
method Adaptive primal-dual algorithm that synchronizes data across geographically distributed tasks.
result The proposed algorithm achieves optimal regret and is effective on real-world datasets.
Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evalua…
Extends convex clustering to graph-structured data.
problem Handling graph-structured data with convex clustering.
method Formulates a convex objective and uses a proximal dual algorithm for efficient recovery.
result Demonstrates the effectiveness of the method on real-life datasets.
Optimizes stochastic linear bandits with efficient, asymptotically optimal algorithm.
problem Optimizing stochastic linear bandits with multiple actions.
method Frequentist information-directed sampling (IDS) with a surrogate for information gain.
result Asymptotically optimal and nearly worst-case optimal in finite time.