Develops DP-SCD for stochastic coordinate descent, making it differentially private.
problem Privacy leak in auxiliary information during stochastic coordinate descent training.
method Develops DP-SCD, leveraging independent noise addition and decoupling/parallelizing coordinate updates.
result Demonstrates competitive performance against DP-SGD with less tuning.
Normalization layers improve the accuracy of Differentially Private training of deep neural networks.
problem Reduced accuracy in deep neural networks with Differentially Private training.
method Proposed a novel method for integrating batch normalization with Differentially Private Stochastic Gradient Descent (DPSGD) without additional privacy loss.
result Training deeper networks with better utility-privacy trade-off is possible.
Algorithm selects public datasets for private machine learning.
problem Choosing the most suitable public dataset for private machine learning.
method Measures gradient subspace distance between public and private datasets.
result Excess risk scales with the subspace distance between gradients.
Paper bridges statistical inference for DP-SGD, a privacy-preserving machine learning method.
problem Asymptotic statistical inference for Differentially Private Stochastic Gradient Descent (DP-SGD).
method Established asymptotic properties of SGD under randomized subsampling, extended to DP-SGD, proposed methods for constructing valid confidence intervals.
result Valid confidence intervals for DP-SGD output achieve nominal coverage rates while maintaining privacy.
Paper improves privacy in SGD with low noise, achieving optimal risk rates.
problem Privacy-preserving machine learning with good performance.
method Differentially private SGD with low-noise analysis.
result Achieves optimal excess risk rates for non-smooth losses.
Gradient sparsification enhances privacy-preserving machine learning models.
problem Improving performance of differentially-private machine learning models under privacy constraints.
method Gradient sparsification combined with compressed sensing and additive Laplace noise.
result Gradient sparsification can improve performance of differentially-private machine learning models for small privacy budgets.
DP-SGD provides privacy guarantees for all data points, but we propose output-specific DP to better account for individual examples.
problem Accounting for individual privacy guarantees in DP-SGD.
method Output-specific (ε,δ)-DP and an efficient algorithm to investigate individual privacy across datasets. result Most examples enjoy stronger privacy guarantees than the worst-case bound, and there is a correlation between training loss and privacy parameter.
New algorithm corrects bias in LDP-released data for better analysis.
problem Bias in data released under Local Differential Privacy (LDP).
method Inverse Weierstrass Private Stochastic Gradient Descent (IWP-SGD).
result Converges to true population risk minimizer at O(1/n) rate. DP-RandP improves privacy-utility tradeoff in DP-SGD by learning priors from random processes.
problem Improving the performance of differentially private stochastic gradient descent (DP-SGD) on private data.
method A three-phase approach that learns priors from images generated by random processes and transfers these priors to private data.
result New state-of-the-art accuracy on CIFAR10, CIFAR100, MedMNIST, and ImageNet for various privacy budgets.
New method reduces privacy impact on model accuracy for underrepresented groups.
problem Privacy mechanisms disproportionately affect underrepresented groups in machine learning models.
method Proposes DPSGD-F, a modified DPSGD that adjusts group contributions based on clipping bias.
result DPSGD-F removes disparate impact of differential privacy on model accuracy for protected groups.
Differentially private random block coordinate descent improves utility in machine learning.
problem Lack of privacy in classical CD methods when handling sensitive information.
method Proposes a differentially private random block coordinate descent method using sketch matrices and importance sampling.
result Demonstrates improved convergence rates and utility guarantees compared to non-private methods.
Paper introduces a differentially private generative model using gradient flow and sliced Wasserstein distance.
problem Protecting privacy in sensitive training data for generative models.
method Gradient flow in the space of probability measures, Gaussian-smoothed Sliced Wasserstein Distance, and numerical scheme for SDE.
result Demonstrates higher-fidelity data generation at low privacy budget compared to existing methods.
New algorithm maintains privacy while improving model performance in selective release.
problem Privacy degradation and slow convergence in DPSGD.
method Differentially Private Selective Release based on Clipped Gradients (DPSR-CG).
result Maintains strict privacy guarantees while achieving exceptional model performance.
Improved linear regression with privacy and robustness guarantees.
problem Private and robust linear regression with adversarial corruption.
method Differentially private stochastic gradient descent with full-batch gradient descent and adaptive clipping.
result Near optimal sample complexity for both private and robust linear regression.
Private adaptive methods improve on traditional SGD for convex optimization.
problem Differential privacy constraints in gradient optimization.
method Differentially private variants of SGD and AdaGrad with adaptive stepsizes and non-isotropic clipping.
result Private AdaGrad outperforms private SGD in high-dimensional problems.
Enhanced stability improves privacy in machine learning.
problem Improving privacy in machine learning training while maintaining accuracy.
method Study of stability in private empirical risk minimization, focusing on strongly-convex loss functions and uniform stability.
result An algorithm with uniform stability of β implies a bound of O(√β) on the scale of noise required for differential privacy.
New method for differentially private optimization with general Lipschitz conditions.
problem Differentially private optimization under general Lipschitz conditions.
method Generalized Lipschitz condition for per-sample gradients, tuning clip norm based on minimum per-sample Lipschitz constant.
result Efficacy of the recommended clip norm tuning method verified on 8 datasets.
The paper introduces a differentially private method for optimization on Riemannian manifolds.
problem Differential privacy in optimization constrained to Riemannian manifolds.
method Adding Gaussian noise to the Riemannian gradient on the tangent space, with privacy and utility guarantees.
result Privacy and utility guarantees for differentially private Riemannian optimization.
New method for uncertainty quantification in private SGD.
problem Uncertainty quantification for differentially private SGD.
method Block bootstrap for SGD under local differential privacy.
result Valid method for uncertainty quantification in private SGD.
DPDMs use DP-SGD to generate private synthetic data.
problem Generating private synthetic data from sensitive datasets.
method Introduced DP-SGD for DMs, investigated DM parameterization and sampling, proposed noise multiplicity.
result Achieved state-of-the-art performance in image generation benchmarks.
New DP-CD method outperforms DP-SGD in solving composite DP-ERM problems.
problem Privacy-preserving machine learning with differential privacy.
method Differentially Private proximal Coordinate Descent (DP-CD) for composite Empirical Risk Minimization (ERM).
result DP-CD outperforms DP-SGD due to larger step sizes and better gradient exploitation.
Gradient perturbation, widely used for differentially private optimization, injects noise at every iterative update to guarantee differential privacy. Previous work first determines the noise level that can satisfy the privacy requirement and then analyzes the utility of noisy gradient updates as in the non-private cas…
Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradie…
This work improves privacy-generalization bounds for DP-SGD.
problem Understanding the trade-off between privacy and generalization in machine learning.
method Proved a linear max-information bound for DP-SGD, derived PAC-Bayes and generalization bounds.
result Explicit and controlled complexity terms for DP-SGD-trained models.
We improve private training accuracy with learning rate schedules and matrix factorizations.
problem Private training with learning rate schedules and correlated noise.
method General upper and lower bounds for learning rate schedules, memory-efficient constructions, and schedule-aware factorizations.
result Schedule-aware factorizations improve accuracy in private training.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
In this paper, we consider efficient differentially private empirical risk minimization from the viewpoint of optimization algorithms. For strongly convex and smooth objectives, we prove that gradient descent with output perturbation not only achieves nearly optimal utility, but also significantly improves the running …
DIFF2 improves differential privacy in nonconvex optimization with better utility bounds.
problem Improving differential privacy in nonconvex optimization with better utility bounds.
method DIFF2 constructs a differential private global gradient estimator using gradient differences.
result DIFF2 achieves a utility of \(\widetilde O(d^{2/3}/(n\varepsilon_{\mathrm{DP}})^{4/3})\), significantly better than \(\widetilde O(\sqrt{d}/(n\varepsilon_{\mathrm{DP}}))\).
While machine learning has achieved remarkable results in a wide variety of domains, the training of models often requires large datasets that may need to be collected from different individuals. As sensitive information may be contained in the individual's dataset, sharing training data may lead to severe privacy conc…
Second-order methods improve differential privacy in convex optimization.
problem Improving differential privacy in convex optimization.
method Developed a private variant of the regularized cubic Newton method for strongly convex loss functions.
result Achieves quadratic convergence and optimal excess loss for strongly convex loss functions.
This paper analyzes convergence of DP-SGD with adaptive quantile clipping.
problem Empirical success of adaptive clipping methods lacks theoretical understanding.
method Comprehensive convergence analysis of SGD with quantile clipping (QC-SGD).
result Establishes theoretical guarantees for DP-QC-SGD, revealing relationships between quantile selection, step size, and convergence.
We propose an algorithm for the adaptation of the learning rate for stochastic gradient descent (SGD) that avoids the need for validation set use. The idea for the adaptiveness comes from the technique of extrapolation: to get an estimate for the error against the gradient flow which underlies SGD, we compare the resul…
Differentially private method for synthetic data generation from vertically partitioned data.
problem Generating synthetic data from vertically partitioned data while preserving privacy.
method Differentially private stochastic gradient descent (DP-SGD) algorithm combined with secure multiparty computation (MPC).
result Comparable accuracy to non-partitioned data, demonstrating privacy-preserving synthetic data generation.
While many solutions for privacy-preserving convex empirical risk minimization (ERM) have been developed, privacy-preserving nonconvex ERM remains a challenge. We study nonconvex ERM, which takes the form of minimizing a finite-sum of nonconvex loss functions over a training set. We propose a new differentially private…
AdaDPIGU improves privacy in deep learning by adaptively clipping and pruning gradients.
problem Privacy in deep learning models, especially in high-dimensional settings.
method Importance-based gradient updates, adaptive clipping, differentially private SGD.
result AdaDPIGU achieves high accuracy while maintaining privacy, outperforming non-private models.
While significant progress has been made separately on analytics systems for scalable stochastic gradient descent (SGD) and private SGD, none of the major scalable analytics frameworks have incorporated differentially private SGD. There are two inter-related issues for this disconnect between research and practice: (1)…
We consider the problem of minimizing a convex risk with stochastic subgradients guaranteeing ε-locally differentially private (ε-LDP). While it has been shown that stochastic optimization is possible with ε-LDP via the standard SGD (Song et al., 2013), its convergence rate largely depends on the learning rate, w…
Optimizes private learning with differential privacy for LASSO problems.
problem Private optimization of convex functions over ℓ1-bounded domains. method Combines iterative localization with private regularized mirror descent and variance-reduced Frank-Wolfe algorithm.
result Achieves optimal excess population loss rates in ℓ1 geometry. Novel algorithm reduces privacy noise in machine learning.
problem High privacy noise in machine learning algorithms.
method Robust statistics, specifically median and trimmed mean, to bound sensitivity of SGD iterates.
result Improved privacy-utility trade-off with reduced noise and computational efficiency.
New DP framework using data truncation for efficient estimation.
problem Differential privacy in unbounded data support.
method Data truncation, exponential family distributions, maximum likelihood estimation, DP stochastic gradient descent.
result Near-optimal sample complexity for Gaussian mean and covariance estimation.
Differentially private algorithms protect model explanations from leaking training data.
problem Model explanations can leak training data, compromising privacy.
method Adaptive differentially private gradient descent algorithm to produce accurate, private explanations.
result Privacy amplification and reduction of overall privacy loss on explanation data.
Private minimum Hellinger distance estimators maintain robustness and efficiency while ensuring privacy.
problem Ensuring privacy in robust statistical estimation.
method Derive private minimum Hellinger distance estimators satisfying Hellinger differential privacy.
result Private minimum Hellinger distance estimators retain robustness and efficiency under privacy constraints.
Zeroth-order optimization methods lack inherent privacy guarantees.
problem Ensuring differential privacy in zeroth-order optimization methods.
method Analyzing ZO-GD with and without random initialization for convex and strongly convex objectives.
result ZO-GD is not differentially private for strongly convex objectives and can have superlinear privacy loss.
Iterative algorithms, like gradient descent, are common tools for solving a variety of problems, such as model fitting. For this reason, there is interest in creating differentially private versions of them. However, their conversion to differentially private algorithms is often naive. For instance, a fixed number of i…
New algorithm improves privacy in high-dimensional machine learning models.
problem Privacy issues in learning large machine learning models.
method Differentially private greedy coordinate descent (DP-GCD) algorithm.
result Achieves logarithmic dependence on dimension for quasi-sparse solutions.
Paper relaxes SGD privacy and generalization guarantees for non-smooth convex losses.
problem Privacy and generalization in SGD for non-smooth convex losses.
method Relaxes Lipschitz and strong smoothness assumptions to Hölder smoothness, proving (ε,δ)-DP and optimal excess risk. result Noisy SGD with α-Hölder smooth losses achieves optimal excess risk with linear gradient complexity for α≥1/2. DP-SGD can update fewer coordinates while maintaining privacy.
problem How to update fewer coordinates in DP-SGD without losing optimization signal.
method TP-TopK (Two-Phase TopK DP-SGD), a two-phase method for coordinate-sparse private training.
result Private training can update fewer coordinates without losing optimization signal, scaling noise with active dimension \(k\) instead of full dimension \(d\).
New tool for parallel and private stochastic convex optimization reduces query complexity.
problem Parallel and private stochastic convex optimization with reduced query complexity.
method Reweighted Stochastic Query (ReSQue) estimator combined with ball oracle acceleration.
result Achieves state-of-the-art complexities for SCO in parallel and private settings.