arXiv research

The paper offers generalization bounds for Transformers that ignore sequence length.

Transformers can interpolate finite input sequences exactly.

This study examines how sequential correlations affect in-context learning in sequence models.

The paper provides a theoretical justification for using stable SSM blocks in deep sequential models.

Study of supervised learning from multiple non-independent sequences.

We study the asymptotic behavior of the asymptotic translation lengths on the curve complexes of pseudo-Anosov monodromies in a fibered cone of a fibered hyperbolic 3-manifold $M$ with $b_1(M) \geq 2$. For a sequence $(Σ_n, ψ_n)$ of fibers and monodromies in the fibered cone, we show that the asymptotic translation len…

Randomized positional encodings boost transformer performance on longer sequences.

Non-parametric estimators improve quickest changepoint detection under irregular sequence lengths.

Recurrent neural networks and sequence to sequence models require a predetermined length for prediction output length. Our model addresses this by allowing the network to predict a variable length output in inference. A new loss function with a tailored gradient computation is developed that trades off prediction accur…

The paper examines conditions for compactness in sequences of warped product length spaces.

Efficient sparse attention reduces self-attention complexity and improves model performance.

Mining tasks over sequential data, such as clickstreams and gene sequences, require a careful design of embeddings usable by learning algorithms. Recent research in feature learning has been extended to sequential data, where each instance consists of a sequence of heterogeneous items with a variable length. However, m…

This research proves guarantees on sequence models' generalization to longer and novel sequences.

We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limit…

Transformers can outperform feedforward and recurrent networks due to dynamic sparsity.

New bounds show transformers need longer training for length generalization.

We consider $N$ Bernoulli random variables, which are independent conditional on a common random factor determining their probability distribution. We show that certain expected functionals of the proportion $L_N$ of variables in a given state converge at rate $1/N$ as $N\rightarrow \infty$. Based on these results, we …

We prove that any Riemannian torus of dimension $m$ with unit volume admits $m$ homologically independent closed geodesics whose length product is bounded from above by $m^m$.

Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.

This article presents a novel method for predicting suicidal ideation from Electronic Health Records (EHR) and Ecological Momentary Assessment (EMA) data using deep sequential models. Both EHR longitudinal data and EMA question forms are defined by asynchronous, variable length, randomly-sampled data sequences. In our …

New proof shows Fuchsian groups have irrational length spectra.

Despite strong performance on a variety of tasks, neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. We study the related issue of receiving infinite-length sequences from a recurrent language model when using common decoding algorithm…

New findings on translation lengths in Teichmüller and curve graphs for pseudo-Anosovs.

A simple text model shows word lengths follow Zipf's law.

Dynamic time warping (DTW) can be used to compute the similarity between two sequences of generally differing length. We propose a modification to DTW that performs individual and independent pairwise alignment of feature trajectories. The modified technique, termed feature trajectory dynamic time warping (FTDTW), is a…

New techniques improve channel prediction in noisy wireless systems.

We introduce the problem of reconstructing a sequence of multidimensional real vectors where some of the data are missing. This problem contains regression and mapping inversion as particular cases where the pattern of missing data is independent of the sequence index. The problem is hard because it involves possibly m…

Proves a conjecture about graph complexes without specific cycle lengths.

Estimates missing mass in Markovian sequences with linear runtime and near-optimal risk.

Accelerates signature kernel computation for sequences.

Deep architecture such as hierarchical semi-Markov models is an important class of models for nested sequential data. Current exact inference schemes either cost cubic time in sequence length, or exponential time in model depth. These costs are prohibitive for large-scale problems with arbitrary length and depth. In th…

The study finds bounds on homologically independent loops on hyperelliptic hyperbolic surfaces.

We will give a weak energy identity for Sacks-Uhlenbeck approximation of harmonic maps and calculate the length of the necks.

The paper improves transformer generalization bounds using rank-dependent covering number bounds.

J.P. Levine introduced a clover link to investigate the indeterminacy of the Milnor invariants of a link. It is shown that for a clover link, the Milnor numbers of length at most $2k+1$ are well-defined if those of length at most $k$ vanish, and that the Milnor numbers of length at least $2k+2$ are not well-defined if …

Time-based model improves recommendation performance.

We prove that the exponential growth rate of the regular language of penetration sequences is smaller than the growth rate of the regular language of normal form words, if the acceptor of the regular language of normal form words is strongly connected. Moreover, we show that the latter property is satisfied for all irr…

A two-step approach efficiently selects hyperparameters for FCMs.

New attention model removes softmax, improving sequence length bias.

Labyrinth fractals are self-similar dendrites in the unit square that are defined with the help of a labyrinth set or a labyrinth pattern. In the case when the fractal is generated by a horizontally and vertically blocked pattern, the arc between any two points in the fractal has infinite length [Cristea\&Steinsky 2009…

Study SGD dynamics in sequence models, revealing training phases and influence of sequence length.

HGConv uses HRR to efficiently detect malware, outperforming existing methods.

Models such as Sequence-to-Sequence and Image-to-Sequence are widely used in real world applications. While the ability of these neural architectures to produce variable-length outputs makes them extremely effective for problems like Machine Translation and Image Captioning, it also leaves them vulnerable to failures o…

New approach reduces malware detection memory requirements and speeds up training.

Phylogenetic tree inference using deep DNA sequencing is reshaping our understanding of rapidly evolving systems, such as the within-host battle between viruses and the immune system. Densely sampled phylogenetic trees can contain special features, including "sampled ancestors" in which we sequence a genotype along wit…

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper net…

Analyzes geodesic lengths in sparse networks, deriving a distribution.

Paper analyzes regret bounds for unconstrained online optimization.