Private distribution learning with public data, leveraging sample compression schemes.
problem Private distribution learning with public and private samples under differential privacy constraints.
method Connection to sample compression schemes and list learning.
result At least d public samples are necessary for private learnability of Gaussians in R^d.
Public pretraining improves private model training even in extreme distribution shift scenarios.
problem Improving private model training accuracy in settings with large distribution shift.
method Empirical evaluation and theoretical explanation of public representations improving private training accuracy.
result Public representations can improve private training accuracy by up to 67% over private training from scratch in settings with large distribution shift.
Paper introduces algorithms for private decision tree learning.
problem Private decision tree learning in distributed settings.
method Proposes DP-TopDown, NoisyCounts, and LocalRNM.
result First utility guarantees for private decision tree learning.
Locally private reinforcement learning protects individual environments from reverse engineering.
problem Protecting private information in distributed reinforcement learning environments.
method Locally differentially private algorithms that protect local agents' models from adversarial reverse engineering.
result Demonstrated that the proposed algorithm performs well under local differential privacy (LDP).
Gradient clipping helps private SGD converge despite potential bias.
problem Gradient clipping in private SGD can bias convergence.
method Theoretical analysis and empirical evaluation of gradient clipping effects.
result Gradient clipping can prevent convergence to stationary points and introduces bias.
Differentially-private Bayes consistency rule for binary classification and density estimation.
problem Privacy constraints limit private learning in the distribution-free PAC model.
method Constructs a universally Bayes consistent learning rule that satisfies differential privacy.
result Private learning is possible for arbitrary distributions, even with a single algorithm.
We present novel, computationally efficient, and differentially private algorithms for two fundamental high-dimensional learning problems: learning a multivariate Gaussian and learning a product distribution over the Boolean hypercube in total variation distance. The sample complexity of our algorithms nearly matches t…
We study a basic private estimation problem: each of n users draws a single i.i.d. sample from an unknown Gaussian distribution, and the goal is to estimate the mean of this Gaussian distribution while satisfying local differential privacy for each user. Informally, local differential privacy requires that each data …
Improved ADMM for convex distributed learning with differential privacy.
problem Privacy concerns in distributed learning with sensitive data.
method Approximate multi-step ADMM with calibrated noise.
result Higher utility and error bounds asymptotic to state-of-the-art.
We provide a differentially private algorithm for hypothesis selection. Given samples from an unknown probability distribution P and a set of m probability distributions H, the goal is to output, in a ε-differentially private manner, a distribution from H whose total variation di…
New bounds for private learning of high-dimensional Gaussian distributions.
problem Learning high-dimensional Gaussian distributions under differential privacy constraints.
method Analytic tools for constructing global covers from local covers, modified hypothesis selection techniques.
result Near-optimal sample complexity bounds for general Gaussians, conjectured to be near-optimal in the general case.
Due to massive amounts of data distributed across multiple locations, distributed machine learning has attracted a lot of research interests. Alternating Direction Method of Multipliers (ADMM) is a powerful method of designing distributed machine learning algorithm, whereby each agent computes over local datasets and e…
EIGAN learns private representations without centralized data, outperforming state-of-the-art.
problem Private representation learning with multiple ally and adversary attributes.
method Exclusion-Inclusion Generative Adversarial Network (EIGAN) and Distributed EIGAN (D-EIGAN).
result EIGAN and D-EIGAN outperform state-of-the-art methods in accuracy and scalability.
Distributed devices such as mobile phones can produce and store large amounts of data that can enhance machine learning models; however, this data may contain private information specific to the data owner that prevents the release of the data. We wish to reduce the correlation between user-specific private information…
Dynamic model considers private asset markets' complexities.
problem Understanding and optimizing private asset allocation.
method State-of-the-art dynamic model with machine learning.
result Optimal investment policies quantified over fund life.
Example shows learnable distributions not privately learnable.
problem Learnable distributions under non-private conditions not transferable to differential privacy.
method Example of a distribution class learnable up to constant error in total variation distance but not under differential privacy.
result Contradicts conjecture of Ashtiani on learnability under differential privacy.
New algorithm for differentially private distributed optimization of smooth, non-convex problems.
problem No differentially private distributed method for smooth, non-convex optimization problems.
method Smoothed normalization integrated with an error-feedback mechanism.
result Achieves superior convergence rate and first differentially private distributed optimization algorithm with provable convergence guarantees.
Improved image generation with private data using perceptual features.
problem Difficulty in training generative models with differential privacy.
method Use pre-trained perceptual features to learn private data distribution.
result Generative models can generate high-quality images with low privacy budget (ϵ≈2). Efficiently learns private models using public data.
problem Improving private learning performance with public data.
method Proves computationally efficient algorithms for private learning with public data.
result First computationally efficient algorithms for private learning with public data.
A novel Bayesian framework for private linear regression with MCMC.
problem Private linear regression in a distributed setting.
method Generative statistical model, MCMC algorithms, fast Bayesian estimation.
result The proposed methods provide well-rounded estimation and prediction.
New privacy-preserving learning model for mixtures of private and public data.
problem Learning from datasets with both private and public data, where privacy concerns differ.
method Designing a differential privacy-preserving learning algorithm for a mixture of private and public sub-populations.
result Linear classifiers can be learned with sample complexity comparable to non-private PAC-learning, even when privacy status correlates with labels.
PILLAR improves SP learning with less private data.
problem Efficiently learning with semi-private data under privacy constraints.
method Uses pre-trained public data features to reduce private data requirements.
result Significantly lower private labelled sample complexity achieved.
Distributed estimation and learning with privacy preserved.
problem Privacy-preserving distributed estimation and learning in a networked environment.
method Linear aggregation schemes with differential privacy constraints.
result Noise minimizes convergence time to best estimates, using Laplace noise.
Study on InstaHide's security, linking to phase retrieval problem.
problem Security of InstaHide scheme for private dataset sharing.
method Design of a provable algorithm for private vector recovery.
result Private vectors can be recovered using synthetic vectors and public vectors.
In many signal processing and machine learning applications, datasets containing private information are held at different locations, requiring the development of distributed privacy-preserving algorithms. Tensor and matrix factorizations are key components of many processing pipelines. In the distributed setting, diff…
Private estimation with public data reduces sample complexity.
problem Estimating private distributions with limited public data.
method Differentially private estimation with public data under constraints of pure or concentrated DP.
result Public data can significantly reduce private sample complexity for estimation.
Improved differentially private deep learning with group-wise clipping techniques.
problem Efficiency and privacy trade-offs in deep learning models.
method Group-wise clipping techniques (per-layer and per-device) to reduce compute time and memory overhead.
result Private learning with group-wise clipping achieves similar or better performance than non-private learning with less wall time.
Paper proposes differentially private quantile regression for high-dimensional data.
problem Privacy concerns in big data with heterogeneous sensitive personal information.
method Newton-type transformation for reformulating quantile regression into an OLS problem; iterative updates for estimation; debiased estimator for inference; communication-efficient bootstrap.
result Near-optimal statistical accuracy and formal privacy guarantees achieved.
First DP algorithm for Wasserstein barycenters on private data.
problem Computing Wasserstein barycenters on private datasets.
method Differentially private algorithms for Wasserstein barycenters.
result High-quality private barycenters with strong accuracy-privacy tradeoffs.
We survey distributed deep learning models for training or inference without accessing raw data from clients. These methods aim to protect confidential patterns in data while still allowing servers to train models. The distributed deep learning methods of federated learning, split learning and large batch stochastic gr…
This work investigates the case of a network of agents that attempt to learn some unknown state of the world amongst the finitely many possibilities. At each time step, agents all receive random, independently distributed private signals whose distributions are dependent on the unknown state of the world. However, it m…
Differentially private ensemble classifiers adapt to data streams while protecting privacy.
problem Adapting to evolving data characteristics while protecting private information.
method Unbounded ensemble updates, model agnostic approach.
result Outperforms competitors on various privacy, drift, and distribution settings.
New regularizer for machine learning using private data.
problem Machine learning with private data.
method Distributionally-robust optimization with locally-differentially-private datasets.
result New regularizer for training linear regression models.
Researchers developed a differentially private method for computing Wasserstein distances.
problem Computing divergences between distributions while preserving privacy.
method They focused on the Sliced Wasserstein Distance and added Gaussian perturbations to make it differentially private.
result They introduced a new differentially private distance, the Smoothed Sliced Wasserstein Distance, which performs well in generative models and domain adaptation.
Proposes a privacy-preserving sign selection method for distributed systems.
problem Sign selection in distributed differentially private settings.
method Iterative peeling of stability function combined with exponential mechanism.
result Recovery of support and signs with optimal signal-to-noise ratio.
Asynchronous algorithms reduce privacy costs in distributed machine learning.
problem Privacy concerns in training machine learning models on scattered private data.
method Differentially-private asynchronous algorithms for collaborative training.
result Cost of privacy is inversely proportional to dataset size and privacy budgets.
Private density estimation in Wasserstein distance for geographic populations.
problem Private estimation of population density distributions.
method Differentially private algorithms for Wasserstein distance, instance-optimal.
result Uniformly achievable instance-optimal rates in both 1D and 2D.
Paper develops private synthetic data for sensitive data analysis.
problem Analyzing sensitive data while preserving privacy.
method Adapting a private density estimation algorithm for synthetic data generation.
result Better computational guarantees for synthetic data generation.
FPFL mitigates unfairness in private federated learning.
problem Differential privacy degrades model performance on under-represented groups.
method Extends modified method of differential multipliers to private federated learning.
result FPFL reduces unfairness in trained models on private federated learning.
We present a provably optimal differentially private algorithm for the stochastic multi-arm bandit problem, as opposed to the private analogue of the UCB-algorithm [Mishra and Thakurta, 2015; Tossou and Dimitrakakis, 2016] which doesn't meet the recently discovered lower-bound of Ω(εKlog(T)) [Shar…
The paper proposes a method to learn differentially private variational autoencoders with term-wise gradient aggregation.
problem Learning variational autoencoders with differential privacy constraints and multiple divergences.
method Term-wise Differentially Private SGD (DP-SGD) that crafts randomized gradients for each loss term, keeping sensitivity at O(1).
result The method reduces the amount of noise needed for differential privacy, allowing better learning.
The Probably Approximately Correct (PAC) Bayes framework (McAllester, 1999) can incorporate knowledge about the learning algorithm and (data) distribution through the use of distribution-dependent priors, yielding tighter generalization bounds on data-dependent posteriors. Using this flexibility, however, is difficult,…
The paper proposes a fair and private decentralized deep learning framework.
problem Ensuring fairness and privacy in collaborative deep learning.
method A reputation system and differential privacy are used. FDPDDL framework is built with two stages: initialisation and update.
result FDPDDL achieves high fairness, comparable accuracy to centralised and distributed frameworks, and better accuracy than standalone.
Alternating Direction Method of Multipliers (ADMM) is a widely used tool for machine learning in distributed settings, where a machine learning model is trained over distributed data sources through an interactive process of local computation and message passing. Such an iterative process could cause privacy concerns o…
A new method for privacy-preserving Bayesian learning in federated learning.
problem Privacy-preserving learning of models from distributed sensitive data.
method Differentially private partitioned variational inference (DPVI) for federated learning.
result First general framework for federated Bayesian learning with differential privacy.
How to train a machine learning model while keeping the data private and secure? We present CodedPrivateML, a fast and scalable approach to this critical problem. CodedPrivateML keeps both the data and the model information-theoretically private, while allowing efficient parallelization of training across distributed w…
In many real-world applications of machine learning, data are distributed across many clients and cannot leave the devices they are stored on. Furthermore, each client's data, computational resources and communication constraints may be very different. This setting is known as federated learning, in which privacy is a …
Privacy is a major issue in learning from distributed data. Recently the cryptographic literature has provided several tools for this task. However, these tools either reduce the quality/accuracy of the learning algorithm---e.g., by adding noise---or they incur a high performance penalty and/or involve trusting externa…