arXiv research

Neural networks memorize exceptions, leading to poor generalization.

The paper identifies patterns in language model weights used for memorizing paragraphs.

Neural networks can memorize random labels just as well as true labels.

Study compares memorization of SimCLR to supervised and random labels training.

New attack reveals memorization patterns in pre-trained LLMs.

ResMem improves model generalization by explicitly memorizing residuals.

Learning rate decay helps modern neural networks by suppressing memorization and improving complex pattern learning.

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differen…

A novel framework IMBoost improves outlier detection by leveraging the inlier memorization effect.

Paper tackles noisy label learning by exploiting memorization effect.

The roles played by learning and memorization represent an important topic in deep learning research. Recent work on this subject has shown that the optimization behavior of DNNs trained on shuffled labels is qualitatively different from DNNs trained with real labels. Here, we propose a novel permutation approach that …

The IM effect helps detect anomalies by memorizing inliers early.

The paper shows averaging gradients leads to memorization, proposing an alternative algorithm to focus on invariances.

We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identity-mapping task. We examine fully-connected and convolutional networks (FCN and CNN), both linear and nonlinear, initialized randomly and then trained to minimize th…

INN method refines clean labeled data from noisy labels.

Label noise may affect the generalization of classifiers, and the effective learning of main patterns from samples with noisy labels is an important challenge. Recent studies have shown that deep neural networks tend to prioritize the learning of simple patterns over the memorization of noise patterns. This suggests a …

New metric $m$-coherence measures gradient alignment during training, revealing surprising memorization patterns.

Large initial learning rate helps neural nets generalize better.

Deep neural networks can generalize by reducing high-frequency noise over time, not always following a monotonic learning bias.

ALTBI enhances outlier detection by maximizing the inlier-memorization effect.

A blindfolded LLM trading framework validates market signals without ticker memorization.

Deep ReLU networks have surprisingly few activation patterns at initialization.

Local data coverage governs memorization in diffusion models.

Geometric framework explains memorization in generative models.

Consistency distillation reduces memorization in diffusion models without harming sample quality.

Study shows how deep generative models can memorize data.

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ po…

Memorizing rare examples helps neural networks generalize better.

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

LLMs can memorize economic data and recall exact values before their training cutoff.

Transformers mimic Bayesian reasoning in controlled settings, revealing geometric mechanisms.

New method reduces memorization in diffusion models without sacrificing image quality.

Logarithmic network width suffices for robust memorization.

New approach shows data memorization trade-offs in large models.

Generative diffusion models gradually memorize training data, losing independent dimensions.

Deep networks preferentially learn shared features, avoiding memorization in early layers.

Deep networks can memorize random labels; symmetric loss improves this.

Learning requires memorizing labels, especially in long-tailed data.

This paper explores memorization in adversarial training and proposes a mitigation algorithm.

Study shows FL reduces unintended memorization by clustering data and using strong user-level privacy.

A new model improves recurrent neural networks' ability to memorize long sequences.

Diffusion models can memorize training data, limiting their creativity and privacy.

RAF model explains neural networks' dual rule learning and fact memorization.

Introduces Relational Privacy (RP) to control relation memorization in question answering models.

Gradient descent memorizes many Gaussians efficiently.

The bias potential model explains how generative models can generalize or memorize samples.

The study uncovers the conditions under which diffusion models memorize or generalize.

Deep models memorize training data in geophysical inversion, leading to biased posterior distributions.