arXiv research

Large learning rates prevent memorization in denoising score matching.

Framework prevents deep learning models from memorizing noisy labels.

Introduces Relational Privacy (RP) to control relation memorization in question answering models.

Diffusion models generalize well until a threshold is reached, preventing memorization.

LLMs can memorize economic data and recall exact values before their training cutoff.

Geometric framework explains memorization in generative models.

Since deep neural networks are over-parameterized, they can memorize noisy examples. We address such a memorization issue in the presence of label noise. From the fact that deep neural networks cannot generalize to neighborhoods of memorized features, we hypothesize that noisy examples do not consistently incur small l…

The paper shows averaging gradients leads to memorization, proposing an alternative algorithm to focus on invariances.

New method prevents deep learning forgetting past by remembering key examples.

In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation sometimes help, but do not prevent this behavior. If one considers neural network weights as random v…

Kernel-smoothed scores improve diffusion models by reducing memorization.

Neural networks memorize exceptions, leading to poor generalization.

Local data coverage governs memorization in diffusion models.

The paper identifies patterns in language model weights used for memorizing paragraphs.

Consistency distillation reduces memorization in diffusion models without harming sample quality.

Study shows how deep generative models can memorize data.

Recently deep neural networks have shown their capacity to memorize training data, even with noisy labels, which hurts generalization performance. To mitigate this issue, we provide a simple but effective baseline method that is robust to noisy labels, even with severe noise. Our objective involves a variance regulariz…

Overparameterized deep networks have the capacity to memorize training data with zero \emph{training error}. Even after memorization, the \emph{training loss} continues to approach zero, making the model overconfident and the test performance degraded. Since existing regularizers do not directly aim to avoid zero train…

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ po…

Memorizing rare examples helps neural networks generalize better.

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

New method reduces memorization in diffusion models without sacrificing image quality.

Logarithmic network width suffices for robust memorization.

New approach shows data memorization trade-offs in large models.

Study compares memorization of SimCLR to supervised and random labels training.

Generative diffusion models gradually memorize training data, losing independent dimensions.

Deep networks preferentially learn shared features, avoiding memorization in early layers.

Deep networks can memorize random labels; symmetric loss improves this.

This paper explores memorization in adversarial training and proposes a mitigation algorithm.

Study shows FL reduces unintended memorization by clustering data and using strong user-level privacy.

ResMem improves model generalization by explicitly memorizing residuals.

Diffusion models can memorize training data, limiting their creativity and privacy.

RAF model explains neural networks' dual rule learning and fact memorization.

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Me…

Gradient descent memorizes many Gaussians efficiently.

The bias potential model explains how generative models can generalize or memorize samples.

The study uncovers the conditions under which diffusion models memorize or generalize.

State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorizati…

Deep models memorize training data in geophysical inversion, leading to biased posterior distributions.

New training method for ReLU networks achieves optimal weight size for memorization.

Transformers with CoT don't enhance reasoning power across all tasks.

Study interprets neural network generalization and memorization on corrupted data.

Early stopping improves generalization in overparameterized diffusion models.

New experiments show deep networks benefit from memorizing rare data points.

The roles played by learning and memorization represent an important topic in deep learning research. Recent work on this subject has shown that the optimization behavior of DNNs trained on shuffled labels is qualitatively different from DNNs trained with real labels. Here, we propose a novel permutation approach that …

We improve deep threshold networks' memorization capacity exponentially.

Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. In this paper, to relieve this issue, we propose …

The IM effect helps detect anomalies by memorizing inliers early.