arXiv research

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ po…

Local data coverage governs memorization in diffusion models.

Generative diffusion models gradually memorize training data, losing independent dimensions.

New attack reveals memorization patterns in pre-trained LLMs.

New experiments show deep networks benefit from memorizing rare data points.

The study uncovers the conditions under which diffusion models memorize or generalize.

The study examines how shallow neural nets converge to training samples or manifold points during diffusion.

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

New algorithm interpolates data with neural nets, independent of sample size.

Solves memorization in diffusion models for manifold data.

We improve deep threshold networks' memorization capacity exponentially.

New training method for ReLU networks achieves optimal weight size for memorization.

Geometric framework explains memorization in generative models.

New approach shows data memorization trade-offs in large models.

A new model improves recurrent neural networks' ability to memorize long sequences.

The paper simplifies influence computations for large-scale machine learning models.

LLMs can memorize economic data and recall exact values before their training cutoff.

New framework evaluates MIA without retraining, addressing biases.

Memorizing rare examples helps neural networks generalize better.

Study shows how deep generative models can memorize data.

Neural networks memorize exceptions, leading to poor generalization.

Study shows FL reduces unintended memorization by clustering data and using strong user-level privacy.

New method reduces memorization in diffusion models without sacrificing image quality.

The paper identifies patterns in language model weights used for memorizing paragraphs.

Sample selection approaches are popular in robust learning from noisy labels. However, how to properly control the selection process so that deep networks can benefit from the memorization effect is a hard problem. In this paper, motivated by the success of automated machine learning (AutoML), we model this issue as a …

State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorizati…

Consistency distillation reduces memorization in diffusion models without harming sample quality.

Deep networks can memorize random labels; symmetric loss improves this.

Spaced repetition is a technique for efficient memorization which uses repeated, spaced review of content to improve long-term retention. Can we find the optimal reviewing schedule to maximize the benefits of spaced repetition? In this paper, we introduce a novel, flexible representation of spaced repetition using the …

Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. In this paper, to relieve this issue, we propose …

Diffusion models can memorize training data, limiting their creativity and privacy.

Data selection boosts fact memorization in language models.

Study interprets neural network generalization and memorization on corrupted data.

Kernel-smoothed scores improve diffusion models by reducing memorization.

Deep models memorize training data in geophysical inversion, leading to biased posterior distributions.

Large learning rates prevent memorization in denoising score matching.

The IM effect helps detect anomalies by memorizing inliers early.

Deep networks preferentially learn shared features, avoiding memorization in early layers.

This paper explores memorization in adversarial training and proposes a mitigation algorithm.

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differen…

DMs emerge from DenseAMs, transitioning from memorization to generalization.

Many results in recent years established polynomial time learnability of various models via neural networks algorithms. However, unless the model is linear separable, or the activation is a polynomial, these results require very large networks -- much more than what is needed for the mere existence of a good predictor.…

INN method refines clean labeled data from noisy labels.

Introduces Relational Privacy (RP) to control relation memorization in question answering models.

This paper studies the relationship between the classification performed by deep neural networks (DNNs) and the decision of various classical classifiers, namely k-nearest neighbours (k-NN), support vector machines (SVM) and logistic regression (LR), at various layers of the network. This comparison provides us with ne…

Early stopping improves generalization in overparameterized diffusion models.

RAF model explains neural networks' dual rule learning and fact memorization.

It has been observed \citep{zhang2016understanding} that deep neural networks can memorize: they achieve 100\% accuracy on training data. Recent theoretical results explained such behavior in highly overparametrized regimes, where the number of neurons in each layer is larger than the number of training samples. In thi…