arXiv research

The paper identifies patterns in language model weights used for memorizing paragraphs.

Word2vec (Mikolov et al., 2013) has proven to be successful in natural language processing by capturing the semantic relationships between different words. Built on top of single-word embeddings, paragraph vectors (Le and Mikolov, 2014) find fixed-length representations for pieces of text with arbitrary lengths, such a…

A classical link in 3-space can be represented by a Gauss paragraph encoding a link diagram in a combinatorial way. A Gauss paragraph may code not a classical link diagram, but a diagram with virtual crossings. We present a criterion and a linear algorithm detecting whether a Gauss paragraph encodes a classical link. W…

A Gauss paragraph is a combinatorial formulation of a generic closed curve with multiple components on some surface. A virtual string is a collection of circles with arrows that represent the crossings of such a curve. Every closed curve has an underlying virtual string and every virtual string has an underlying Gauss …

Paper develops a framework for generating coherent image captions using visual features and hierarchical topics.

Classic Topic Models are built under the Bag Of Words assumption, in which word position is ignored for simplicity. Besides, symmetric priors are typically used in most applications. In order to easily learn topics with different properties among the same corpus, we propose a new line of work in which the paragraph str…

In this paper we use theory of embedded graphs on oriented and compact $PL$-surfaces to construct minimal realizations of signed Gauss paragraphs. We prove that the genus of the ambient surface of these minimal realizations can be seen as a function of the maximum number of Carter's circles. For the case of signed Gaus…

Feature extraction has gained increasing attention in the field of machine learning, as in order to detect patterns, extract information, or predict future observations from big data, the urge of informative features is crucial. The process of extracting features is highly linked to dimensionality reduction as it impli…

Neural networks memorize exceptions, leading to poor generalization.

Local data coverage governs memorization in diffusion models.

Geometric framework explains memorization in generative models.

Consistency distillation reduces memorization in diffusion models without harming sample quality.

Study shows how deep generative models can memorize data.

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ po…

Memorizing rare examples helps neural networks generalize better.

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

LLMs can memorize economic data and recall exact values before their training cutoff.

New method reduces memorization in diffusion models without sacrificing image quality.

Logarithmic network width suffices for robust memorization.

Learning latent representations from long text sequences is an important first step in many natural language processing applications. Recurrent Neural Networks (RNNs) have become a cornerstone for this challenging task. However, the quality of sentences during RNN-based decoding (reconstruction) decreases with the leng…

New approach shows data memorization trade-offs in large models.

Study compares memorization of SimCLR to supervised and random labels training.

Generative diffusion models gradually memorize training data, losing independent dimensions.

Deep networks preferentially learn shared features, avoiding memorization in early layers.

Deep networks can memorize random labels; symmetric loss improves this.

This paper explores memorization in adversarial training and proposes a mitigation algorithm.

Study shows FL reduces unintended memorization by clustering data and using strong user-level privacy.

We present an attention-based ranking framework for learning to order sentences given a paragraph. Our framework is built on a bidirectional sentence encoder and a self-attention based transformer network to obtain an input order invariant representation of paragraphs. Moreover, it allows seamless training using a vari…

ResMem improves model generalization by explicitly memorizing residuals.

Diffusion models can memorize training data, limiting their creativity and privacy.

We present a new local entity disambiguation system. The key to our system is a novel approach for learning entity representations. In our approach we learn an entity aware extension of Embedding for Language Model (ELMo) which we call Entity-ELMo (E-ELMo). Given a paragraph containing one or more named entity mentions…

RAF model explains neural networks' dual rule learning and fact memorization.

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Me…

Gradient descent memorizes many Gaussians efficiently.

Introduces Relational Privacy (RP) to control relation memorization in question answering models.

The bias potential model explains how generative models can generalize or memorize samples.

The study uncovers the conditions under which diffusion models memorize or generalize.

State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorizati…

Deep models memorize training data in geophysical inversion, leading to biased posterior distributions.

We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a large document corpus. This problem is often solved in two steps. The retrieval phase first reduces the solution space, returning a subset …

New training method for ReLU networks achieves optimal weight size for memorization.

Diffusion models generalize well until a threshold is reached, preventing memorization.

Large learning rates prevent memorization in denoising score matching.

Transformers with CoT don't enhance reasoning power across all tasks.

Study interprets neural network generalization and memorization on corrupted data.

Early stopping improves generalization in overparameterized diffusion models.

New experiments show deep networks benefit from memorizing rare data points.

The roles played by learning and memorization represent an important topic in deep learning research. Recent work on this subject has shown that the optimization behavior of DNNs trained on shuffled labels is qualitatively different from DNNs trained with real labels. Here, we propose a novel permutation approach that …