arXiv research

Geometric framework explains memorization in generative models.

Local data coverage governs memorization in diffusion models.

New approach shows data memorization trade-offs in large models.

LLMs can memorize economic data and recall exact values before their training cutoff.

Memorizing rare examples helps neural networks generalize better.

Study shows how deep generative models can memorize data.

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ po…

Generative diffusion models gradually memorize training data, losing independent dimensions.

Neural networks memorize exceptions, leading to poor generalization.

Study shows FL reduces unintended memorization by clustering data and using strong user-level privacy.

New method reduces memorization in diffusion models without sacrificing image quality.

The paper identifies patterns in language model weights used for memorizing paragraphs.

State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorizati…

Consistency distillation reduces memorization in diffusion models without harming sample quality.

Deep networks can memorize random labels; symmetric loss improves this.

New experiments show deep networks benefit from memorizing rare data points.

Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. In this paper, to relieve this issue, we propose …

Diffusion models can memorize training data, limiting their creativity and privacy.

The study uncovers the conditions under which diffusion models memorize or generalize.

Data selection boosts fact memorization in language models.

Study interprets neural network generalization and memorization on corrupted data.

Deep models memorize training data in geophysical inversion, leading to biased posterior distributions.

Large learning rates prevent memorization in denoising score matching.

The IM effect helps detect anomalies by memorizing inliers early.

Deep networks preferentially learn shared features, avoiding memorization in early layers.

This paper explores memorization in adversarial training and proposes a mitigation algorithm.

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differen…

INN method refines clean labeled data from noisy labels.

Introduces Relational Privacy (RP) to control relation memorization in question answering models.

This paper studies the relationship between the classification performed by deep neural networks (DNNs) and the decision of various classical classifiers, namely k-nearest neighbours (k-NN), support vector machines (SVM) and logistic regression (LR), at various layers of the network. This comparison provides us with ne…

Early stopping improves generalization in overparameterized diffusion models.

RAF model explains neural networks' dual rule learning and fact memorization.

It has been observed \citep{zhang2016understanding} that deep neural networks can memorize: they achieve 100\% accuracy on training data. Recent theoretical results explained such behavior in highly overparametrized regimes, where the number of neurons in each layer is larger than the number of training samples. In thi…

New training method for ReLU networks achieves optimal weight size for memorization.

Deep neural networks (DNNs) can easily fit a random labeling of the training data with zero training error. What is the difference between DNNs trained with random labels and the ones trained with true labels? Our paper answers this question with two contributions. First, we study the memorization properties of DNNs. O…

Diffusion models generalize well until a threshold is reached, preventing memorization.

New algorithm interpolates data with neural nets, independent of sample size.

Solves memorization in diffusion models for manifold data.

The roles played by learning and memorization represent an important topic in deep learning research. Recent work on this subject has shown that the optimization behavior of DNNs trained on shuffled labels is qualitatively different from DNNs trained with real labels. Here, we propose a novel permutation approach that …

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

Researchers validate ML scenario generators by checking dependencies and detecting memorization effects.

Logarithmic network width suffices for robust memorization.

New method reduces neural network memorization of noisy labels.

Over-parameterized models can memorize noisy labels and still generalize well, revealing a hidden structure.

New attack reveals memorization patterns in pre-trained LLMs.

This paper examines how Higher-Order Langevin Dynamics reduces memorization in diffusion models.

Study compares memorization of SimCLR to supervised and random labels training.

Paper tightens CMI bounds for learning algorithms.