arXiv research

Improved neural keyphrase generation by beam search with reward functions.

New attention model removes softmax, improving sequence length bias.

Recurrent models can produce infinite sequences, causing bias; new methods prevent this.

Non-parametric estimators improve quickest changepoint detection under irregular sequence lengths.

TOQ-Nets learn to recognize complex temporal events with varying objects and sequences.

Transformers tend to learn more symmetric functions in sequence data.

Randomized positional encodings boost transformer performance on longer sequences.

Recurrent neural networks and sequence to sequence models require a predetermined length for prediction output length. Our model addresses this by allowing the network to predict a variable length output in inference. A new loss function with a tailored gradient computation is developed that trades off prediction accur…

Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge …

The paper examines conditions for compactness in sequences of warped product length spaces.

This research proves guarantees on sequence models' generalization to longer and novel sequences.

Using frequency distributions of daily closing price time series of several financial market indexes, we investigate whether the bias away from an equiprobable sequence distribution found in the data, predicted by algorithmic information theory, may account for some of the deviation of financial markets from log-normal…

A simple method reduces bias in LLM auto-evaluators by controlling output length.

We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limit…

The paper offers generalization bounds for Transformers that ignore sequence length.

Decouples critic chunk length from policy to improve policy reactivity and performance.

SSMs have a built-in bias towards low-frequency components, which can be adjusted.

New bounds show transformers need longer training for length generalization.

Transformers can interpolate finite input sequences exactly.

Method estimates dataset utility via minimal program length proxy.

Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods …

Study improves voice conversion model with Mel-spectrogram augmentation.

XL-Editor improves sentence post-editing using XLNet's variable-length insertion probability.

New techniques improve channel prediction in noisy wireless systems.

Accelerates signature kernel computation for sequences.

Deep architecture such as hierarchical semi-Markov models is an important class of models for nested sequential data. Current exact inference schemes either cost cubic time in sequence length, or exponential time in model depth. These costs are prohibitive for large-scale problems with arbitrary length and depth. In th…

Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been show…

We will give a weak energy identity for Sacks-Uhlenbeck approximation of harmonic maps and calculate the length of the necks.

New method trains GFlowNets from partial episodes to improve convergence and stability.

J.P. Levine introduced a clover link to investigate the indeterminacy of the Milnor invariants of a link. It is shown that for a clover link, the Milnor numbers of length at most $2k+1$ are well-defined if those of length at most $k$ vanish, and that the Milnor numbers of length at least $2k+2$ are not well-defined if …

Time-based model improves recommendation performance.

Labyrinth fractals are self-similar dendrites in the unit square that are defined with the help of a labyrinth set or a labyrinth pattern. In the case when the fractal is generated by a horizontally and vertically blocked pattern, the arc between any two points in the fractal has infinite length [Cristea\&Steinsky 2009…

New method for Transformer models to encode position information without sequential bias.

Study SGD dynamics in sequence models, revealing training phases and influence of sequence length.

The paper provides a theoretical justification for using stable SSM blocks in deep sequential models.

HGConv uses HRR to efficiently detect malware, outperforming existing methods.

Inspired by the possibility that generative models based on quantum circuits can provide a useful inductive bias for sequence modeling tasks, we propose an efficient training algorithm for a subset of classically simulable quantum circuit models. The gradient-free algorithm, presented as a sequence of exactly solvable …

Models such as Sequence-to-Sequence and Image-to-Sequence are widely used in real world applications. While the ability of these neural architectures to produce variable-length outputs makes them extremely effective for problems like Machine Translation and Image Captioning, it also leaves them vulnerable to failures o…

New approach reduces malware detection memory requirements and speeds up training.

The study reveals a persistent bias in the distribution of holonomy on compact hyperbolic 3-manifolds.

Spectral regularization simplifies sequence models by focusing on grammatical simplicity.

Phylogenetic tree inference using deep DNA sequencing is reshaping our understanding of rapidly evolving systems, such as the within-host battle between viruses and the immune system. Densely sampled phylogenetic trees can contain special features, including "sampled ancestors" in which we sequence a genotype along wit…

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper net…

Paper analyzes regret bounds for unconstrained online optimization.

This work is devoted to the study of deformations of hyperbolic cone structures under the assumption that the lengths of the singularity remain uniformly bounded over the deformation. Given a sequence $(M_{i}$ of pointed hyperbolic cone-manifolds with topological type $(M,Σ)$, where $M$ is a closed, orientab…

We present the Latent Sequence Decompositions (LSD) framework. LSD decomposes sequences with variable lengthed output units as a function of both the input sequence and the output sequence. We present a training algorithm which samples valid extensions and an approximate decoding algorithm. We experiment with the Wall …

Representation learning of pedestrian trajectories transforms variable-length timestamp-coordinate tuples of a trajectory into a fixed-length vector representation that summarizes spatiotemporal characteristics. It is a crucial technique to connect feature-based data mining with trajectory data. Trajectory representati…

BigBird improves transformer performance on NLP tasks with longer sequences.