Paper tackles partial label learning with self-guided retraining.
problem Dealing with partially labeled examples where each instance has a set of candidate labels.
method Unified formulation with constraints for joint training and pseudo-labeling; maximum infinity norm regularization for automatic differentiation; convex-concave optimization problem; upper-bound surrogate objective function.
result Significantly outperforms state-of-the-art partial label learning approaches.
GRAC improves reinforcement learning by self-guiding and self-regularizing.
problem Learning divergence and slow updates in reinforcement learning algorithms.
method Self-regularized TD-learning and self-guided policy improvement.
result Achieved or outperformed state-of-the-art results on OpenAI gym tasks.
Self-guided ALPs improve MDP policies without domain knowledge.
problem Improving MDP policies with minimal domain knowledge.
method Self-guided sequence of ALPs with random basis functions and state-relevance distribution.
result High probability error bounds and improved policy performance.
Enhances belief propagation to find global optima without increasing computational burden.
problem Improving probabilistic inference accuracy on graphical models.
method Homotopy continuation method that gradually incorporates pairwise potentials.
result SBP finds the global optimum of the Bethe approximation for attractive models.
Self-guiding diffusion models improve time series forecasting, refinement, and generation.
problem Improving time series forecasting, refinement, and generation.
method Unconditionally-trained diffusion model with self-guidance mechanism.
result TSDiff outperforms task-specific conditional forecasting methods and maintains generative performance.
Proposes a method to retrain neural networks incrementally for continuous data flow.
problem Continuous data flow and the challenges of catastrophic forgetting and efficient retraining.
method Incremental retraining using multi-armed bandits to select important samples and weights, and a new regularization term for synapse and neuron importance.
result Mitigates catastrophic forgetting and boosts model performance.
Retraining with predicted labels improves model accuracy in noisy settings.
problem Improving model accuracy with noisy or corrupted labels.
method Retraining with predicted hard labels in a linearly separable binary classification setting.
result Retraining with predicted labels can increase model accuracy, as proven theoretically.
DeltaGrad rapidly retrain models with minimal data changes.
problem Rapid retraining of machine learning models with minimal data changes.
method DeltaGrad algorithm based on cached training information.
result DeltaGrad compares favorably to state-of-the-art methods.
Less frequent retraining improves forecast accuracy in retail demand forecasting.
problem Balancing forecast accuracy and computational efficiency in global models.
method Analysis of ten machine learning and deep learning models across two large retail datasets with various retraining scenarios.
result Less frequent retraining strategies maintain forecast accuracy while reducing computational costs.
This paper optimizes retraining models using their own predictions and noisy labels.
problem Improving model performance through optimal retraining of noisy labels.
method Developed a principled framework based on approximate message passing (AMP) to analyze iterative retraining procedures.
result Derivation of the Bayes optimal aggregator function to minimize prediction error.
K-priors enable quick adaptation with minimal retraining.
problem Machine learning models struggle to adapt to changes efficiently.
method Combines weight and function-space priors to reconstruct past gradients.
result Adaptation with K-priors achieves similar performance to full retraining with less data.
Two retraining techniques outperform fine-tuning in neural network pruning.
problem Improving accuracy and compression in neural network pruning.
method Weight rewinding and learning rate rewinding compared to fine-tuning.
result Rewinding techniques outperform fine-tuning in accuracy and compression.
Traditional classification algorithms assume that training and test data come from similar distributions. This assumption is violated in adversarial settings, where malicious actors modify instances to evade detection. A number of custom methods have been developed for both adversarial evasion attacks and robust learni…
Retraining stabilizes model influence on data.
problem Performativity in predictive models leads to feedback loops.
method Developed the stable signal principle to address retraining dynamics.
result Repeated risk minimization converges geometrically to stable signal direction.
This research shows loss weighting remains effective in last layer retraining despite model overparameterization.
problem Overcoming biases in machine learning models at scale.
method Theoretical and practical exploration of last layer retraining in an overparameterized setting.
result Loss weighting is still effective in last layer retraining, but weights must account for model overparameterization.
New method detects data distribution changes and retraining is advised.
problem Detecting when data distribution changes for retraining prediction algorithms.
method Exchangeable martingales and conformal prediction.
result Guaranteed validity of the method, with efficiency explored.
Paper tackles model collapse in synthetic data retraining.
problem Iterative retraining of generative models on synthetic data can lead to performance deterioration.
method Integrates an external synthetic data verifier to prevent model collapse.
result Synthetic retraining with a verifier can improve model performance initially but may lead to convergence to the verifier's knowledge center.
Hard thresholding remains efficient for DNN pruning, but smart pruning offers faster accuracy recovery.
problem Efficiently pruning deep neural networks while minimizing accuracy loss.
method Proposes a novel smart pruning algorithm based on difference of convex functions optimization.
result Smart pruning is often orders of magnitude faster than competing approaches while achieving low accuracy degradation.
Data augmentation methods improve worst-case model performance.
problem Ensuring fair predictions across subpopulations in large models.
method Linear last layer retraining with data augmentation techniques.
result Optimal worst-group accuracy achieved for Gaussian latent representation distribution.
Classifier learns to ignore unreliable feedback from end users.
problem Improving classifier performance by filtering unreliable feedback.
method Modeling end users as autonomous agents, periodically retraining classifier with filtered feedback.
result Classifier can identify and filter out unreliable feedback, improving performance.
Last layer retraining improves robustness to spurious correlations without high computational costs.
problem Neural networks can rely on spurious features like backgrounds for predictions.
method Simple last layer retraining on large models.
result Last layer retraining matches or outperforms state-of-the-art approaches on spurious correlation benchmarks.
Quantum computing speeds up neural network training and retraining.
problem Inefficient classical training and retraining of neural networks.
method Adiabatic quantum computing to optimize Kolmogorov-Arnold Networks using Bezier curves.
result Quantum optimization achieves 100x faster retraining compared to classical methods.
New framework for predicting decisions that influence their own outcomes.
problem Predictions that affect the outcomes they predict, leading to undesirable distribution shift.
method Risk minimization framework combining statistics, game theory, and causality.
result Necessary and sufficient conditions for retraining to converge to a performatively stable point of minimal loss.
A smaller, less-trained model guides image generation, improving quality without sacrificing variation.
problem Improving image quality and variation in diffusion models without compromising one for the other.
method Guiding a conditional model with a smaller, less-trained version of the same model.
result Significant improvements in ImageNet generation, setting record FIDs.
Paper presents efficient AL strategies using influence functions.
problem High computational costs in AL strategies.
method Influence functions for efficient model retraining approximation.
result Makes AL strategies applicable in practice.
New method uses neural tangent kernel for efficient active learning.
problem Efficiently approximating deep learning's look-ahead selection criteria.
method Approximates retraining with neural tangent kernel for active learning.
result Approximation works asymptotically and enables sequential active learning.
The paper shows how curated synthetic data can optimize human preferences in generative models.
problem Contamination of web-scale datasets by synthetic data affects future model training.
method Theoretical study of iterated retraining of generative models with curated synthetic data.
result Data curation can be seen as an implicit preference optimization mechanism, maximizing expected reward.
Improved method for efficient black-box optimization in latent space.
problem Efficiently optimize expensive black-box functions over complex input spaces.
method Optimize in latent space of deep generative models, retrain model periodically and weight data points.
result Significantly improved efficiency and performance on synthetic and real-world problems.
Efficiently removes specific data subsets without retraining for GDPR compliance.
problem Efficiently removing specific data subsets to comply with GDPR regulations.
method Statistical framework for machine unlearning with minimax optimality for squared loss.
result Developed Unlearning Least Squares (ULS) achieving minimax optimality for estimating model parameters.
Network Implosion reduces ResNet layers without accuracy loss.
problem High computation costs in Residual Networks.
method Static layer pruning and retraining to erase unimportant layers.
result Reduces ResNet layers by 24.00-42.86% without accuracy drop.
New method protects neural networks from adversarial attacks without generating adversarial examples.
problem Vulnerability of neural networks to adversarial examples.
method Entropic retraining, inspired by information theory.
result Significant increase in NNs' security and robustness.
This paper improves neural network efficiency by combining filter columns and retraining, boosting array utilization and accuracy.
problem Efficient implementation of sparse convolutional neural networks on systolic arrays.
method Column combining of filter matrices, retraining of remaining weights, joint optimization for high utilization and accuracy.
result Significantly increased systolic array utilization efficiency (e.g., ~4x) and maintained high classification accuracy.
Many active learning methods belong to the retraining-based approaches, which select one unlabeled instance, add it to the training set with its possible labels, retrain the classification model, and evaluate the criteria that we base our selection on. However, since the true label of the selected instance is unknown, …
The paper proposes a method to monitor deep learning predictions for retraining, reducing costs.
problem Reducing computational costs in deep learning by detecting when predictions are no longer valid.
method Sequential monitoring of network predictions based on projected second moments monitoring.
result The proposed method can drastically reduce computational costs in deep learning.
Study on self-consuming generative models with diverse human curation, focusing on convergence and stability.
problem Analyzing self-consuming generative models with heterogeneous human curation.
method Investigates the asymptotic behavior of retraining dynamics using nonlinear Perron--Frobenius theory and Banach contraction mapping.
result Improves convergence results and provides stability and non-stability analyses for the model.
Wide and Deep GNN learns from distributed graphs and retrain online.
problem Decentralized graph support changes over time, causing mismatch between training and testing graphs.
method Wide and Deep GNN architecture with distributed online learning.
result Convergence guarantees for online retraining of the wide part of the GNN.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
Paper outlines a system for ML models to learn continuously from evolving data.
problem Managing ML models in environments where data evolves.
method Describes a reference architecture for self-maintaining systems.
result Proposes a reference architecture for continual AutoML.
Decomposable-Net compresses neural networks without retraining for various sizes.
problem Performance degradation when changing model size after training.
method Decomposes weight matrices via SVD and adjusts ranks for different sizes.
result Maintains and improves performance across multiple model sizes.
Graph neural networks perform well with pretraining even when new nodes and edges are added later.
problem Evaluation of graph neural networks' performance with dynamic graphs.
method Comparison of pretrained and retrained models in an experimental setup with dynamic graphs.
result Pretrained models maintain high accuracy on unseen nodes.
Timber targets decision trees, outperforming existing attacks.
problem Poisoning decision trees to manipulate model predictions.
method Greedy attack strategy using sub-tree retraining for efficiency.
result Timber outperforms existing attacks in effectiveness and efficiency.
Conformal prediction fails under severe feature turnover in COVID-19 supply chain tasks.
problem Dealing with distribution shift in conformal prediction models.
method Using COVID-19 as a natural experiment across 8 supply chain tasks, analyzing SHAP explanations.
result Coverage drops vary widely (0% to 86.7%) and correlate with single-feature dependence.
This paper proposes a method to improve neural network quantization without retraining.
problem Handling outliers in quantized DNN weights and activations.
method Outlier Channel Splitting (OCS) which duplicates channels containing outliers and halves their values.
result OCS outperforms state-of-the-art clipping techniques with minimal overhead.
AIDEL improves scalability of learned indexes in storage systems.
problem Expensive retraining and heavy inter-model dependency in learned indexes limit scalability.
method Construct different linear regression models based on data distribution, making them independent and easier to partition.
result AIDEL improves insertion performance by about 2x and comparable lookup performance.
Gradient ascent method successfully removes specific data points from neural networks without retraining.
problem Addressing privacy and ethical concerns by removing specific data points from trained models.
method Gradient ascent approach to unlearning, leveraging the implicit bias of gradient descent towards margin maximization conditions.
result Gradient ascent method can successfully unlearn specific data points from two-layer ReLU neural networks without retraining.
Demon aligns diffusion models without retraining or backpropagation.
problem Aligning diffusion models with user preferences.
method Stochastic optimization to control noise distribution.
result Significantly improves aesthetics scores for text-to-image generation.
Algorithm improves model performance on shifted concepts without retraining.
problem Improving model performance on shifted concepts with limited source data.
method Model consolidation of intermediate internal distributions after adaptation.
result Effective improvement in model performance on shifted concepts.
Model compression has gained a lot of attention due to its ability to reduce hardware resource requirements significantly while maintaining accuracy of DNNs. Model compression is especially useful for memory-intensive recurrent neural networks because smaller memory footprint is crucial not only for reducing storage re…