arXiv research

RAF model explains neural networks' dual rule learning and fact memorization.

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ po…

New measure FTC quantifies how much a ReLU network can fine-tune.

Derives an empirical capacity model for self-attention neural networks.

Data selection boosts fact memorization in language models.

This paper explores memorization in adversarial training and proposes a mitigation algorithm.

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differen…

New algorithm interpolates data with neural nets, independent of sample size.

We improve deep threshold networks' memorization capacity exponentially.

Logarithmic network width suffices for robust memorization.

Generative diffusion models gradually memorize training data, losing independent dimensions.

New analysis tightens memory capacity of Hopfield models using spherical codes.

Algorithm performance in supervised learning is a combination of memorization, generalization, and luck. By estimating how much information an algorithm can memorize from a dataset, we can set a lower bound on the amount of performance due to other factors such as generalization and luck. With this goal in mind, we int…

Researchers analyze backdoor data poisoning attacks and identify a memorization capacity parameter.

Improved neural network capacity analysis using simplified RDT.

New NTK bounds show deep networks with minimum over-parameterization can still memorize and optimize.

Transformers with CoT don't enhance reasoning power across all tasks.

DMs emerge from DenseAMs, transitioning from memorization to generalization.

A Gaussian mixture model improves generalization for long-tailed data.

Deep neural networks can generalize by reducing high-frequency noise over time, not always following a monotonic learning bias.

Standard Transformers approximate Hölder functions and achieve optimal nonparametric regression rate.

Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of cl…

Overwhelming theoretical and empirical evidence shows that mildly overparametrized neural networks -- those with more connections than the size of the training data -- are often able to memorize the training data with $100\%$ accuracy. This was rigorously proved for networks with sigmoid activation functions and, very …

Fine-tuning LLMs on privacy-sensitive data introduces privacy risk, and synthetic data audits can quantify this risk.

Transformers mimic Bayesian reasoning in controlled settings, revealing geometric mechanisms.

New bounds on neural network capacity for treelike sign perceptrons using RDT.

Neural networks memorize exceptions, leading to poor generalization.

Local data coverage governs memorization in diffusion models.

Geometric framework explains memorization in generative models.

The paper identifies patterns in language model weights used for memorizing paragraphs.

Consistency distillation reduces memorization in diffusion models without harming sample quality.

Study shows how deep generative models can memorize data.

Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…

Recently deep neural networks have shown their capacity to memorize training data, even with noisy labels, which hurts generalization performance. To mitigate this issue, we provide a simple but effective baseline method that is robust to noisy labels, even with severe noise. Our objective involves a variance regulariz…

Memorizing rare examples helps neural networks generalize better.

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

LLMs can memorize economic data and recall exact values before their training cutoff.

New method reduces memorization in diffusion models without sacrificing image quality.

Paper explains adversarial training's robust overfitting through a minimax game perspective.

New approach shows data memorization trade-offs in large models.

Study compares memorization of SimCLR to supervised and random labels training.

Deep networks preferentially learn shared features, avoiding memorization in early layers.

Deep networks can memorize random labels; symmetric loss improves this.

Study shows FL reduces unintended memorization by clustering data and using strong user-level privacy.

ResMem improves model generalization by explicitly memorizing residuals.

Diffusion models can memorize training data, limiting their creativity and privacy.

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Me…

Gradient descent memorizes many Gaussians efficiently.