FDS tackles long horizon hyperparameter optimization issues.
problem Memory scaling and gradient degradation in long horizon tasks.
method Forward-mode differentiation with sharing (FDS).
result Significantly outperforms greedy gradient-based alternatives.
Combines Integrated Gradients and PatternAttribution into PGIG, outperforming alternatives.
problem Improving neural network explainability methods.
method Combines Integrated Gradients and PatternAttribution into Pattern-Guided Integrated Gradients (PGIG).
result PGIG outperforms other explainability methods in a large-scale image degradation experiment.
This paper improves federated learning for industrial predictive analytics by accommodating client heterogeneity.
problem Traditional federated models assume homogeneity in degradation processes, which doesn't apply to industrial settings.
method Personalized federated prognostic model using proximal gradient descent algorithm for joint parameter estimation.
result The proposed model enhances performance and provides comprehensive failure time distributions.
DGPs with variational inference suffer from SNR issues that degrade gradient estimates, leading to unreliable training.
problem SNR issues in gradient estimates for DGPs with variational inference.
method Adapted doubly reparameterized gradient estimators for DGP training.
result Fix improves predictive performance of DGP models.
New algorithm improves multitask learning across diverse agents.
problem Performance degradation in decentralized learning with heterogeneous objectives.
method Developed an exact subspace diffusion algorithm for multitask learning over networks.
result The algorithm outperforms alternatives in noisy gradient approximations.
This paper improves deep learning models' accuracy with differential privacy using gradient encoding and denoising.
problem Deep learning models leak sensitive information about their training datasets.
method Gradient encoding to map gradients to a smaller vector space, and denoising for post-processing.
result Our technique achieves better model accuracy with differential privacy guarantees compared to state-of-the-art methods.
New GPU algorithm boosts machine learning with larger datasets.
problem Limited GPU memory restricts training data size.
method Out-of-core GPU gradient boosting algorithm.
result Training larger datasets on GPUs without accuracy loss.
Develops new bounds for deterministic samplers in diffusion models.
problem Analyzing deterministic samplers in diffusion generative models.
method Operational interpretation of deterministic sampling; restoration and degradation steps.
result First polynomial convergence bounds for DDIM-type samplers.
AdaScale SGD adapts learning rates for large-batch training efficiently.
problem Adapting learning rates for large-batch training to balance speed-ups and model quality.
method Adaptive learning rate adaptation based on gradient variance.
result AdaScale achieves reliable speed-ups for a wide range of batch sizes without degrading model quality.
Study finds non-IID data causes FL performance issues.
problem Reduced performance in federated learning due to non-IID data.
method Investigated from IID to non-IID settings, categorized methods into two strategies.
result Inconsistencies in client loss landscapes are the primary cause of performance degradation.
Gradient coding is a technique for straggler mitigation in distributed learning. In this paper we design novel gradient codes using tools from classical coding theory, namely, cyclic MDS codes, which compare favorably with existing solutions, both in the applicable range of parameters and in the complexity of the invol…
Develops a nonlocal PINN framework using PDDO for better solution of PDEs with sharp gradients.
problem Dealing with sharp gradients in solutions of PDEs using traditional PINN approaches.
method Integrates long-range interactions (nonlocality) into PINN using Peridynamic Differential Operator (PDDO).
result Nonlocal PINN approach improves solution accuracy and parameter inference for problems with sharp gradients.
RTC-GTNLN model recovers traffic data from missing values and noise.
problem Simultaneous missing data and noise in traffic data.
method Gradient tensor nuclear L1-L2 norm for robust tensor completion.
result RTC-GTNLN model outperforms existing methods in complex recovery scenarios.
MGD improves ML training efficiency and accuracy.
problem Optimizing large-scale ML model training configurations.
method Metagradient descent and smooth model training.
result Significantly improved model performance and learning rate schedules.
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
Self-supervised method enhances ultrasound images without needing clean targets.
problem Multiplicative speckle, acquisition blur, and scanner artifacts hamper ultrasound interpretation.
method Physics-guided degradation model trained on rotated/cropped patches with synthesized inputs.
result Achieves highest PSNR/SSIM across Gaussian and speckle noise levels, with significant improvements in heavy noise conditions.
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose \emph{double quantization}, a general scheme for quantizing both model parameters and gradients. …
Artificial neural network (ANN) provides superior accuracy for nonlinear alternating current (AC) state estimation (SE) in smart grid over traditional methods. However, research has discovered that ANN could be easily fooled by adversarial examples. In this paper, we initiate a new study of adversarial false data injec…
DOPPLER optimizes DP training with low-pass filtering, improving model accuracy.
problem Privacy concerns in deep learning models and performance degradation of DP optimizers.
method Developed DOPPLER, a low-pass filter for DP optimizers, to reduce privacy noise and enhance model quality.
result DOPPLER optimizers outperform non-DOPPLER counterparts by 3%-10% in test accuracy.
SpecGD mitigates misalignment in phase retrieval models with anisotropic inputs.
problem Misalignment during gradient descent in phase retrieval models with anisotropic inputs.
method Spectral gradient descent modifies gradient updates to preserve directional information and remove spike amplification.
result SpecGD removes spike amplification, leading to stable alignment and accelerated noise contraction.
Measures DNA quality degradation effects.
problem Identifying degraded DNA sequence data.
method Novel quality quantification based on intentional degradation effects.
result Quantified measures of degradation can be used for multiple purposes.
Modeling coating degradation with fewer data points.
problem Efficiently forecasting material degradation from high-frequency sensor data.
method Discrete degradation events using Hawkes processes.
result Forecasting future coating failure with superior performance.
This work is a part of ICLR Reproducibility Challenge 2019, we try to reproduce the results in the conference submission PADAM: Closing The Generalization Gap of Adaptive Gradient Methods In Training Deep Neural Networks. Adaptive gradient methods proposed in past demonstrate a degraded generalization performance than …
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
Improved stochastic gradient estimation for deep learning in high dimensions.
problem Inadmissibility of mini-batch gradients in high-dimensional settings.
method Stein-rule shrinkage applied to gradient computation.
result The proposed SR-Adam outperforms Adam in large-batch settings.
Paper develops a robust federated recommendation system against poisoning attacks.
problem Low-cost poisoning attacks degrade federated recommendation systems' performance.
method Develops a robust learning strategy using gradients to filter out Byzantine clients.
result Empirically validated robust learning strategy on four datasets.
BinaryDuo improves BNNs by coupling binary activations, outperforming state-of-the-art models.
problem Gradient mismatch in BNNs due to binarizing activations.
method Using gradient of smoothed loss function to estimate gradient mismatch, proposing BinaryDuo scheme with coupled ternary activations.
result BinaryDuo outperforms state-of-the-art BNNs on various benchmarks.
Paper uses deep learning to model systems with degrading behavior.
problem Modeling systems with degrading hysteretic behavior and uncertainty.
method Uses low-fidelity data to train a deep operator network (DeepONet).
result Improves prediction error in degrading hysteretic systems with uncertainty.
Unified framework reduces NFEs for inverse problems.
problem High computational costs and degraded reconstruction quality in existing LDM-based inverse solvers.
method Consistency Regularised Gradient Flows for posterior sampling and prompt optimization.
result Significantly reduced computational cost with state-of-the-art performance.
New algorithm maintains privacy while improving model performance in selective release.
problem Privacy degradation and slow convergence in DPSGD.
method Differentially Private Selective Release based on Clipped Gradients (DPSR-CG).
result Maintains strict privacy guarantees while achieving exceptional model performance.
Develops a new method to improve performance in multi-objective learning problems.
problem Gradient bias in multi-objective learning leading to degraded performance.
method Stochastic Multi-objective gradient Correction (MoCo) method that guarantees convergence without increasing batch size.
result Demonstrates effectiveness of MoCo method in simulations on multi-task learning.
We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD) framework for optimizing deep neural networks. ESGD combines SGD and gradient-free evolutionary algorithms as complementary algorithms in one framework in which the optimization alternates between the SGD step and evolution step to improve th…
Statistical test detects model degradations in optimized language models.
problem Detecting model degradations in optimized language models.
method Statistical hypothesis testing framework based on McNemar's test.
result Even small accuracy degradations (0.3%) can be attributed to actual degradations, not noise.
BNCR-GAN improves GANs to generate clean images from degraded inputs.
problem Generating clean images from blurred, noisy, and compressed degraded inputs.
method Multiple-generator model with image, blur-kernel, noise, and quality-factor generators, using masking architectures and adaptive consistency losses.
result BNCR-GAN effectively learns clean image generators from degraded images without degradation parameters.
In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. We experimentally demonstrate that as meta-training progresses, the meta-test solutions, obtained after adapting the meta-train solution of the model, to new tasks via few…
Boosted GFlowNets improve exploration by sequentially training GFlowNets with residual rewards.
problem GFlowNets struggle to evenly explore reward landscapes, leading to poor coverage of high-reward areas.
method Sequential training of an ensemble of GFlowNets, each optimizing a residual reward.
result Boosted GFlowNets achieve better exploration and sample diversity on multimodal benchmarks and peptide design tasks.
Recent analyses of certain gradient descent optimization methods have shown that performance can degrade in some settings - such as with stochasticity or implicit momentum. In deep reinforcement learning (Deep RL), such optimization methods are often used for training neural networks via the temporal difference error o…
TSSM splits neural networks for parallel training with minimal accuracy loss.
problem Accuracy degradation in parallel training of deep neural networks.
method TSSM reformulates alternating minimization to achieve parallelism with minimal accuracy loss.
result TSSM achieves significant speedup without accuracy loss on multiple datasets.
While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent ar…
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differen…
The choice of batch-size in a stochastic optimization algorithm plays a substantial role for both optimization and generalization. Increasing the batch-size used typically improves optimization but degrades generalization. To address the problem of improving generalization while maintaining optimal convergence in large…
Framework optimizes battery storage for markets by separating long-term degradation from short-term market dynamics.
problem Intractable computation due to timescale mismatch between battery degradation and market dynamics.
method Approximate dynamic programming with value function approximation and pseudo-time encoding.
result Policy outperforms benchmarks in real-time market scenarios.
This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchan…
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to …
New analysis reveals batch size effects on stochastic conditional gradient methods.
problem Understanding the role of batch size in stochastic conditional gradient methods.
method Deriving a new analysis focusing on momentum-based stochastic conditional gradient algorithms (e.g., Scion).
result Increasing batch size initially improves optimization accuracy but can degrade performance beyond a critical threshold.
AdaDPIGU improves privacy in deep learning by adaptively clipping and pruning gradients.
problem Privacy in deep learning models, especially in high-dimensional settings.
method Importance-based gradient updates, adaptive clipping, differentially private SGD.
result AdaDPIGU achieves high accuracy while maintaining privacy, outperforming non-private models.
New algorithm eliminates sign function in PGD attacks, improving performance.
problem Improving robustness of neural networks against adversarial attacks.
method Proposes a new raw gradient descent (RGD) algorithm that eliminates the sign function in PGD attacks.
result The RGD algorithm outperforms PGD and other competitors in various settings.
Paper develops a hybrid DNN approach for RUL prediction with adaptive drift.
problem RUL estimation challenges in practice, especially online update and uncertainty quantification.
method Hybrid DNN approach with Wiener-based-degradation model and adaptive drift. LSTM-CNN for trajectory prediction and Bayesian inference for adaptive drift.
result Superior accuracy in RUL prediction demonstrated on turbofan engines data.