Gradient sparsification enhances privacy-preserving machine learning models.
problem Improving performance of differentially-private machine learning models under privacy constraints.
method Gradient sparsification combined with compressed sensing and additive Laplace noise.
result Gradient sparsification can improve performance of differentially-private machine learning models for small privacy budgets.
Volume-preserving neural networks prevent gradient issues.
problem Vanishing and exploding gradients in deep neural networks.
method A new neural network architecture with volume-preserving sublayers.
result Volume-preserving neural networks maintain gradient stability.
New algorithm for privacy-preserving nonconvex optimization.
problem Privacy-preserving nonconvex empirical risk minimization.
method Differentially private stochastic gradient descent algorithm.
result Achieves strong privacy guarantees efficiently with improved utility.
New objective function preserves Bellman's principle for policy gradient.
problem Lack of objective function capturing Bellman's principle optimally.
method Proposed a new objective function and its gradient.
result Preserves Bellman's principle of optimality in policy gradient.
Gradient flow preserves speed for integral Menger curvature curves.
problem Optimizing curves with integral Menger curvature constraints.
method Projected Sobolev gradient flow in Hilbert space.
result Long-time existence and C1,1-bounds for the flow. In this paper we extend some well-known rigidity results for conformal changes of Einstein metrics to the class of generalized quasi-Einstein (GQE) metrics, which includes gradient Ricci solitons. In order to do so, we introduce the notions of conformal diffeomorphisms and vector fields that preserve a GQE structure. W…
New defense method protects client data privacy in federated learning.
problem Gradient leakage attacks in federated learning.
method Learning to obscure data to generate synthetic samples.
result Synthetic samples preserve predictive features and protect privacy.
New method preserves convergence rates in gradient-based optimization.
problem How to discretize gradient-based optimization systems while preserving stability and convergence rates.
method Geometric framework for dissipative symplectic integration.
result Dissipative symplectic integrators preserve rates of convergence up to a controlled error.
New neural network architecture preserves gradient norms to approximate Lipschitz functions.
problem Training neural networks with strict Lipschitz constraints to ensure robustness and generalization.
method Identified gradient norm preservation as a necessary property, combined with norm-constrained weight matrices and GroupSort activation function.
result Norm-constrained GroupSort architectures can approximate Lipschitz functions and achieve tighter Wasserstein distance estimates.
Gradient flow of curve length on Sobolev metrics preserves convexity.
problem Optimal low-regularity gradient flow of curve length.
method Explicit gradient formula, Picard-Lindelöf theorem, time-reparametrisation.
result Exponential decay of length and preservation of convexity.
Improved image translation using asymmetric gradient guidance.
problem Trade-off between style transformation and content preservation in diffusion models.
method Asymmetric gradient guidance to guide reverse diffusion sampling.
result Our method outperforms state-of-the-art models in image translation tasks.
Paper bridges statistical inference for DP-SGD, a privacy-preserving machine learning method.
problem Asymptotic statistical inference for Differentially Private Stochastic Gradient Descent (DP-SGD).
method Established asymptotic properties of SGD under randomized subsampling, extended to DP-SGD, proposed methods for constructing valid confidence intervals.
result Valid confidence intervals for DP-SGD output achieve nominal coverage rates while maintaining privacy.
Orthogonal initializations speed up neural network training by preserving gradients and activations.
problem Understanding and optimizing the initialization of neural networks to speed up training.
method Connection between Fisher information matrix, spectral radius, and gradient smoothness.
result Orthogonal initializations ensure gradients and activations are preserved, leading to faster training.
Mean curvature flow is not a gradient flow on two nondegenerate metric spaces.
problem Whether mean curvature flow is a gradient flow on nondegenerate metric spaces of simple closed plane curves.
method Examined two nondegenerate metric spaces: uniformness-preserving and curvature-weighted structures.
result Mean curvature flow is not a gradient flow on either metric space.
Privacy-preserving SGD with heavy-tailed noise achieves differential privacy guarantees.
problem Privacy preservation in noisy SGD with heavy-tailed noise.
method Differential privacy guarantees for SGD with heavy-tailed noise.
result SGD with heavy-tailed perturbations achieves (0,O(1/n))-DP. Paper improves privacy in SGD with low noise, achieving optimal risk rates.
problem Privacy-preserving machine learning with good performance.
method Differentially private SGD with low-noise analysis.
result Achieves optimal excess risk rates for non-smooth losses.
Study curves evolving on hypersurfaces with free boundaries, preserving length.
problem Evolution of curves on hypersurfaces with free boundaries.
method Nonlocal evolution equation with nonlinear boundary conditions, short-time existence, uniqueness, and parabolic energy estimates.
result Global existence and convergence to critical points proved.
Non-affine aggregation rules cannot preserve monotonicity in convex learning.
problem Designing non-affine aggregation rules that maintain monotonicity in convex learning.
method Proving that monotonicity of aggregated gradients is preserved only if the aggregation rule is positively affine.
result Non-affine aggregation prevents steady convergence and substantially degrades algorithmic stability.
Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.
problem Stagnation of gradient descent in low-precision computation.
method Proposed unbiased stochastic rounding schemes that trade zero bias for larger probability of preserving small gradients.
result Unbiased rounding methods typically improve convergence rate of gradient descent for convex problems.
A new framework explains why early pruning works well.
problem Understanding why early pruning of neural networks leads to good performance.
method Gradient flow framework to unify pruning measures.
result Magnitude-based pruning removes least contributing parameters, leading to faster convergence.
Study proves stability and uniqueness for a specific type of flow.
problem Volume-preserving mean curvature flow stability and uniqueness.
method New gradient flow calibrations for volume preservation, stability estimate in distributional solutions.
result Strong solutions are calibrated and stable under certain conditions.
Improves GBDT accuracy with differential privacy.
problem Balancing privacy and accuracy in GBDT models.
method Adaptive gradient control and novel boosting framework for privacy budget allocation.
result Achieves better model accuracy with differential privacy.
New method improves fairness in DP learning by preventing excessive gradient suppression.
problem Disparate impact on model predictions for minority groups in DP learning.
method Bounded adaptive clipping to prevent excessive gradient suppression.
result Improves worst-class accuracy by over 10 percentage points compared to existing methods.
The paper generalizes optimization algorithms using category theory.
problem Optimizing functions in a category-theoretic setting.
method Using the Cartesian reverse derivative to generalize gradient descent and Newton's method.
result Properties of optimization algorithms are preserved in the generalized setting, including invariances and convergence.
Prunes neural networks at initialization to save resources, achieving high accuracy.
problem Efficiently reducing resource requirements for neural networks at both training and test time.
method Gradient Signal Preservation (GraSP) to prune networks at initialization.
result Pruning 80% of VGG-16 weights on ImageNet with only a 1.6% drop in accuracy.
New algorithm reduces communication bandwidth for large-scale deep learning training.
problem Efficiently compressing gradients for ring all-reduce in large-scale clusters.
method Importance weighted pruning based on gradient and parameter size.
result Achieved significant gradient compression ratios (64X and 58.8X) on AlexNet and ResNet50.
The paper studies totally nonnegative parts of flag varieties and their topologies.
problem Understanding the topology of totally nonnegative flag varieties.
method Algebraic, geometric, and dynamical perspectives; orbit context; gradient flows; Riemannian metrics.
result Positivity is preserved in certain metrics on the totally nonnegative part of flag varieties.
Paper proposes a privacy-preserving method for estimating complex models.
problem Lack of flexibility in existing model classes for approximating data-generating processes.
method Privacy-preserving distributed estimation of generalized additive mixed models using component-wise gradient boosting.
result Proposed algorithm yields equivalent model estimates as component-wise gradient boosting on pooled data.
In this paper, we study two kind of L^2 norm preserved non-local heat flows on closed manifolds. We first study the global existence, stability and asymptotic behavior to such non-local heat flows. Next we give the gradient estimates of positive solutions to these heat flows.
Paper designs privacy-preserving deep learning systems for combined datasets without sharing local data.
problem Privacy-preserving deep learning over combined datasets without sharing local data.
method Designs systems using SGD with shared weights, not gradients, and proves privacy-preserving while maintaining accuracy.
result Achieves the same learning accuracy as SGD while preserving privacy.
G-PATE generates private data with high utility using teacher-discriminator aggregation.
problem Privacy concerns in large-scale data sharing for machine learning.
method Generative adversarial nets combined with private gradient aggregation among discriminators.
result Significantly improves privacy budget efficiency and data utility.
Cloak identifies essential features to preserve prediction privacy without provider collaboration.
problem Discovering the subset of features necessary for a prediction task.
method Gradient-based perturbation maximization method to identify essential features, followed by suppression of the rest using utility-preserving constant values.
result Cloak reduces mutual information between input and sifted representations by 85.01% with negligible utility loss.
AdaDPIGU improves privacy in deep learning by adaptively clipping and pruning gradients.
problem Privacy in deep learning models, especially in high-dimensional settings.
method Importance-based gradient updates, adaptive clipping, differentially private SGD.
result AdaDPIGU achieves high accuracy while maintaining privacy, outperforming non-private models.
DP-GD improves CNN training accuracy with privacy, especially with low signal-to-noise ratios.
problem Privacy-preserving training of neural networks with crowdsourced data.
method Differentially private gradient descent (DP-GD) algorithm applied to two-layer CNNs.
result DP-GD can achieve superior generalization performance compared to GD, especially with low signal-to-noise ratios.
Paper tightens privacy and generalization bounds for iterative learning.
problem Balancing privacy and generalization in iterative learning algorithms.
method Established alignment between generalization and privacy, derived composition theorems for iterative algorithms.
result Generalization bounds for iterative learning algorithms are strictly tighter than existing works.
A fast method for discrete OT with group-sparse regularization for class label preservation.
problem Efficiently measuring the distance between two discrete distributions with class labels.
method Fast discrete OT with group-sparse regularizers using gradient-based algorithms.
result Up to 8.6 times faster than original method without degrading accuracy.
Flow preserves isoperimetric ratio for immersed surfaces.
problem Preserving isoperimetric ratio in Willmore flow.
method Non-local L2-gradient flow for Willmore energy. result Long-time existence and convergence for spherical initial data.
Secure XGB for privacy-preserving machine learning in federated learning.
problem Privacy-preserving machine learning in federated learning with practical gradient tree boosting models.
method Secure multi-party computation, distributed model storage, secure permutation protocols.
result Our XGB models provide competitive accuracy and practical performance.
This paper investigates gradient recovery schemes for data defined on discretized manifolds. The proposed method, parametric polynomial preserving recovery (PPPR), does not require the tangent spaces of the exact manifolds, and they have been assumed for some significant gradient recovery methods in the literature. Ano…
New DP EM algorithm with statistical guarantees for mixture models.
problem Preserving privacy in EM algorithms for mixture models.
method Proposed a DP EM algorithm with statistical guarantees.
result Near optimal estimation error for GMM in DP model.
A new gradient descent method speeds up in flat regions and slows in steep directions.
problem Improving the speed and stability of gradient descent algorithms.
method Introducing a 'power gradient' where each gradient component is replaced by its H-th power, with 0<H<1. result The new gradient descent methods achieve significantly better performances, especially for Nesterov accelerated gradient and AMSGrad.
LEASGD improves privacy-preserving decentralized learning with lower communication costs.
problem Achieving efficient and private decentralized learning.
method Proposes LEASGD, a Leader-Follower Elastic Averaging Stochastic Gradient Descent algorithm.
result LEASGD outperforms state-of-the-art algorithms in terms of lower loss and reduced communication costs.
Gradient descent mostly converges to a small subspace of eigenvectors.
problem Understanding the dynamics of gradient descent in deep learning.
method Analyzing the convergence of gradients in various deep learning scenarios.
result Gradient descent converges to a small subspace spanned by a few top eigenvectors of the Hessian.
FedNew improves federated learning efficiency and privacy.
problem Low communication efficiency and privacy issues in Newton-type methods for federated learning.
method Introduces a two-level framework using ADMM for inverse Hessian-gradient approximation and Newton's method for global model updates, reducing communication overhead.
result FedNew achieves superior communication efficiency and privacy compared to existing methods.
GNMR controls runtime stability in low-precision language model training.
problem Efficient low-precision training faces numerical risks at specific operators.
method GNMR compares gradient norms to historical means, applying bounded recovery actions.
result GNMR preserves high-fidelity quality with sparse, budgeted recovery.
A new decentralized SGD algorithm for privacy-preserving machine learning.
problem Privacy concerns in sharing training data for machine learning models.
method Decentralized differentially private SGD without replacement.
result The proposed algorithm maintains both privacy and convergence in large-scale machine learning.
Study curves evolving by gradient flow of elastic energy, proving existence, smoothing, and convergence.
problem Evolution of curves with fixed length and clamped boundary conditions.
method Negative L2-gradient flow of elastic energy, existence, parabolic smoothing, constrained Lojasiewicz-Simon gradient inequality. result Convergence to a critical point as time tends to infinity.
PPGAN adds noise to GAN gradients to protect private data.
problem Protecting private data in GANs when training on sensitive datasets.
method Integrates differential privacy into GANs by adding noise to gradients and using Moments Accountant strategy.
result Demonstrates generation of high-quality synthetic data while maintaining privacy.