DOPPLER optimizes DP training with low-pass filtering, improving model accuracy.
problem Privacy concerns in deep learning models and performance degradation of DP optimizers.
method Developed DOPPLER, a low-pass filter for DP optimizers, to reduce privacy noise and enhance model quality.
result DOPPLER optimizers outperform non-DOPPLER counterparts by 3%-10% in test accuracy.
Framework purifies approximate differential privacy to pure differential privacy.
problem Achieving pure differential privacy from approximate differential privacy.
method Randomized post-processing with calibrated noise to eliminate δ parameter.
result First statistically and computationally efficient reduction from approximate DP to pure DP.
Unified bounds for DP risks reduce noise and improve accuracy.
problem Difficult interpretation and calibration of DP mechanisms.
method Hypothesis-testing interpretation of DP (f-DP) and unified bounds. result Unified bounds are tighter and tunable for specific risks.
DiSK improves DP optimizers by simplifying Kalman filtering for better performance.
problem Performance drop of DP optimizers in large-scale training due to noise injection.
method DiSK uses Kalman filtering to denoise privatized gradients and refine gradient estimations.
result DiSK achieves significant performance improvements over standard DP optimizers in large-scale training.
Differentially private learning avoids iterative optimization in parameter space.
problem Differentially private training of neural networks
method Hypernetworks trained on public datasets
result Significantly reduces noise in parameter space
Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private …
This paper provides a method for noise-calibrated inference from DP synthetic data.
problem Inference from DP synthetic data is often miscalibrated and lacks principled uncertainty quantification.
method Release DP sufficient statistics, perform noise-calibrated likelihood-based inference, and optional synthetic data generation.
result Asymptotic normality and valid confidence intervals for the plug-in DP MLE.
Optimal DP mechanisms for vector queries are found to be staircase distributions.
problem Designing optimal additive mechanisms for vector-valued queries under differential privacy.
method Reduction to radially symmetric distributions and convex rearrangement theory.
result Staircase mechanisms are optimal for any norm and cost function.
Machine learning (ML) models trained by differentially private stochastic gradient descent (DP-SGD) have much lower utility than the non-private ones. To mitigate this degradation, we propose a DP Laplacian smoothing SGD (DP-LSSGD) to train ML models with differential privacy (DP) guarantees. At the core of DP-LSSGD is…
DPMC improves inverse problem solving with MCMC, reducing error in noisy conditions.
problem Inaccurate posterior approximation in inverse problems with high noise levels.
method DPMC uses Annealed MCMC to sample through a series of intermediate distributions, reducing accumulated error.
result DPMC outperforms DPS in various inverse problems, reducing error and evaluations.
Improved DP-SGD for variational inference reduces noise and variance.
problem Poor convergence and high variance in variational parameter outputs due to gradient noise in DP-SGD.
method Introduced aligned gradients and iterate averaging to reduce DP-induced noise, and noise-aware posteriors.
result Less noisy gradient estimator and improved parameter estimates for variational inference.
New bounds for KANs trained with DP-SGD, addressing correlated noise.
problem Risk bounds for Kolmogorov-Arnold Networks trained by DP-SGD with correlated noise.
method Established new optimization and population risk analysis for KANs trained with DP-SGD, addressing correlated noise.
result First optimization and population risk analysis of correlated-noise mechanisms for DP training in non-convex settings, including neural networks.
Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
A new method combines synthetic data analysis and DP generation to produce accurate uncertainty estimates.
problem Invalid inferences from DP synthetic data analysis.
method Combining synthetic data analysis techniques from MI and NA Bayesian modeling with a novel noise-aware synthetic data generation algorithm.
result Accurate confidence intervals from DP synthetic data are produced, wider with tighter privacy.
Proposes a method to generate private synthetic data in a decentralized setting using correlated noise.
problem Challenges of generating private synthetic data in a decentralized setting with limited client data.
method Integrates CAPE protocol into federated DP-CDA framework to generate anti-correlated noise.
result Improves privacy-utility trade-off in federated setting compared to centralized approach.
This paper improves privacy accounting in decentralized FL using f-Differential Privacy.
problem Challenges in accurately quantifying privacy budget in decentralized FL.
method Develops two new f-DP-based accounting methods for decentralized FL.
result Yields tighter (ε,δ) bounds and improved utility compared to existing methods.
DP-GD improves CNN training accuracy with privacy, especially with low signal-to-noise ratios.
problem Privacy-preserving training of neural networks with crowdsourced data.
method Differentially private gradient descent (DP-GD) algorithm applied to two-layer CNNs.
result DP-GD can achieve superior generalization performance compared to GD, especially with low signal-to-noise ratios.
Paper proposes DP-SGD and DP-NSGD for differentially private non-convex optimization.
problem Mitigating privacy risks in large model learning.
method Clip or normalize per-sample gradients and add noise for differential privacy.
result Achieved convergence rate of gradient norm for non-convex optimization.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
DP-SPRT improves privacy in sequential tests with near-optimal error rates.
problem Privacy constraints in sequential probability ratio tests.
method A wrapper for SPRT that uses a private mechanism to determine when to stop based on predefined intervals.
result DP-SPRT achieves near-optimal error rates and privacy guarantees.
New RDP guarantees for heavy-tailed SDEs and SGD.
problem Characterizing differential privacy for heavy-tailed noise in learning algorithms.
method Rényi flow computations and fractional Poincaré inequalities.
result First RDP guarantees for heavy-tailed SDEs with weaker dependence on dimension.
Noise-aware DP inference improves accuracy for complex models.
problem Inaccurate results and biases in DP inference for complex models.
method Noise-aware stochastic gradient variational inference.
result Accurate coverages and predictive probabilities for complex models.
Projective DP-SGD reduces privacy error by identifying low-dimensional gradient subspaces.
problem Differentially private SGD's error rate scales with model's dimensionality, problematic for over-parameterized models.
method Projective DP-SGD, projecting noisy gradients to a low-dimensional subspace identified from a public dataset.
result The method reduces the dependence on model dimensionality, improving accuracy in high privacy regimes.
Develops DP-SCD for stochastic coordinate descent, making it differentially private.
problem Privacy leak in auxiliary information during stochastic coordinate descent training.
method Develops DP-SCD, leveraging independent noise addition and decoupling/parallelizing coordinate updates.
result Demonstrates competitive performance against DP-SGD with less tuning.
Improved regret bounds for DP-KLUCB and DP-IMED in Bernoulli bandits.
problem Minimizing regret in stochastic bandits under ε-global Differential Privacy.
method Developed DP versions of KLUCB and IMED, proving tighter lower bounds and matching upper bounds.
result DP-KLUCB and DP-IMED achieve asymptotically optimal regret under ε-global DP.
AdaDPSyn generates synthetic examples to protect private data in ICL.
problem Protecting private data in in-context learning with large language models.
method Data-adaptive differentially private algorithm that dynamically adjusts noise level based on data properties.
result AdaDPSyn outperforms existing methods in preserving high ICL accuracy while maintaining differential privacy.
Paper addresses FL over MAC with DP constraints, proposing a novel consensus scheme.
problem Federated learning over a multiple access channel with differential privacy constraints.
method Proposes a novel consensus scheme using digital distributed stochastic gradient descent (D-DSGD) with artificial noise to preserve DP.
result Demonstrates improved convergence rate and DP level for a given MAC capacity.
A major challenge for machine learning is increasing the availability of data while respecting the privacy of individuals. Here we combine the provable privacy guarantees of the differential privacy framework with the flexibility of Gaussian processes (GPs). We propose a method using GPs to provide differentially priva…
Improved DP-SGD on large models achieves high accuracy on image classification tasks.
problem Differentially private image classification often degrades performance.
method Careful hyper-parameter tuning and signal propagation techniques.
result Achieved 81.4% top-1 accuracy on CIFAR-10 under (8, 10^{-5})-DP.
This paper improves deep learning models' accuracy with differential privacy using gradient encoding and denoising.
problem Deep learning models leak sensitive information about their training datasets.
method Gradient encoding to map gradients to a smaller vector space, and denoising for post-processing.
result Our technique achieves better model accuracy with differential privacy guarantees compared to state-of-the-art methods.
Balls-and-Bins sampling improves DP-SGD privacy and utility.
problem Improving privacy and utility in DP-SGD implementations.
method Introducing Balls-and-Bins sampling as an alternative to shuffling in DP-SGD.
result Balls-and-Bins sampling achieves utility comparable to shuffling while offering better privacy amplification.
FedHDPrivacy uses DP to improve FL in IoT, maintaining high accuracy.
problem Privacy threats in FL, especially in IoT environments.
method Integrates DP with neuro-symbolic computing, actively monitoring and adjusting noise.
result Maintains high performance in manufacturing monitoring, surpassing other FL methods.
Paper simplifies DP composition for adaptive privacy budgets, enabling better privacy and accuracy in deep learning.
problem Tension between efficiency and flexibility in DP composition theorems.
method Rényi Differential Privacy (RDP) for adaptive privacy budgets, proving simpler composition theorem with smaller constants.
result Practical DP composition for adaptive privacy budgets, enabling better privacy and accuracy in deep learning.
New method optimizes privacy and compute trade-offs for deep learning.
problem Privacy and compute trade-offs in deep learning training.
method Decoupling privacy analysis and experimental behavior, using TAN and scaling laws for DP-SGD.
result Stronger privacy guarantees with significant reduction in computational budget.
DP-SGD can update fewer coordinates while maintaining privacy.
problem How to update fewer coordinates in DP-SGD without losing optimization signal.
method TP-TopK (Two-Phase TopK DP-SGD), a two-phase method for coordinate-sparse private training.
result Private training can update fewer coordinates without losing optimization signal, scaling noise with active dimension \(k\) instead of full dimension \(d\).
Expands differential privacy mechanisms to include the Generalized Gaussian mechanism for improved private machine learning.
problem Improving privacy in machine learning algorithms while maintaining utility.
method Introduces and analyzes the Generalized Gaussian (GG) mechanism for differential privacy.
result The GG mechanism provides better performance than the Laplace and Gaussian mechanisms across various values of β.
New method improves privacy in linear regression with optimal error bounds.
problem Differentially private linear regression with suboptimal error bounds.
method One-pass mini-batch stochastic gradient descent (DP-AMBSSGD) with adaptive clipping.
result Nearly optimal error bounds in terms of key parameters like dimensionality, number of points, and noise standard deviation.
This paper shows how differential privacy can be achieved naturally in federated learning over fading channels without artificial noise.
problem Achieving differential privacy in federated learning over fading channels without artificial noise.
method Study of AirFL over multiple-access fading channels with a multi-antenna base station, deriving novel bounds on differential privacy.
result DP can be achieved naturally in federated learning over fading channels without artificial noise, revealing convergence-privacy trade-offs.
New method audits DP guarantees without noise or subsampling info.
problem Auditing DP guarantees of ML models without prior info.
method Histogram-based density estimation for lower bounds.
result Natural generalization of membership inference auditing.
Private statistics estimation faces a bias, accuracy, and privacy trilemma.
problem Balancing privacy, accuracy, and bias in statistical estimation.
method Use differential privacy (DP) for private statistics, but clip samples to control sensitivity and add noise for privacy, introducing bias.
result No algorithm can simultaneously have low bias, low error, and low privacy loss for arbitrary distributions.
New analysis shows how to balance privacy and accuracy in deep learning.
problem Balancing privacy and accuracy in deep learning models.
method Continuous time analysis through neural tangent kernel (NTK) for arbitrary architectures.
result Large clipping norm improves calibration without sacrificing accuracy.
Privacy-preserving SGD with heavy-tailed noise achieves differential privacy guarantees.
problem Privacy preservation in noisy SGD with heavy-tailed noise.
method Differential privacy guarantees for SGD with heavy-tailed noise.
result SGD with heavy-tailed perturbations achieves (0,O(1/n))-DP. DPConvCNP learns to predict private data accurately and privately.
problem Balancing privacy and accuracy in machine learning models.
method Meta-learning combined with improved DP mechanism.
result DPConvCNP outperforms DP GP baseline, especially on non-Gaussian data.
New sampling scheme improves privacy in DP-SGD without sacrificing utility.
problem Suboptimal privacy amplification due to participation variance in Poisson subsampling.
method Balanced Iteration Subsampling (BIS) with structured randomness.
result BIS achieves stronger privacy amplification than Poisson subsampling and is optimal at both extremes of noise spectrum.
Enhances privacy in federated learning with Laplacian smoothing.
problem Protecting data privacy in federated learning while maintaining model accuracy.
method Laplacian smoothing for differentially private federated learning (DP-Fed-LS).
result Improves model accuracy with differential privacy guarantee and membership privacy.
Algorithm estimates top k eigenvectors of shared covariance matrices while preserving privacy.
problem Differentially private PCA with adaptive noise for arbitrary k.
method Iterative algorithm with adaptive noise reduction.
result First algorithm for estimating top k eigenvectors with near-optimal statistical error.
DP-RandP improves privacy-utility tradeoff in DP-SGD by learning priors from random processes.
problem Improving the performance of differentially private stochastic gradient descent (DP-SGD) on private data.
method A three-phase approach that learns priors from images generated by random processes and transfers these priors to private data.
result New state-of-the-art accuracy on CIFAR10, CIFAR100, MedMNIST, and ImageNet for various privacy budgets.
Decor protects decentralized learning models from curious users.
problem Privacy violation in decentralized learning.
method Decor uses correlated Gaussian noises to protect local models in decentralized SGD with differential privacy guarantees.
result Decor matches central DP optimal privacy-utility trade-off for arbitrary connected graphs.