arXiv research

New bounds show transformers need longer training for length generalization.

Shorter adversarial prompts help protect LLMs from jailbreak attacks.

Task hinting improves transformer performance on longer tasks.

This research proves guarantees on sequence models' generalization to longer and novel sequences.

Paper trains a Transformer to add numbers of any length.

Transformer pretraining yields strong EB performance without explicit adaptation.

This paper introduces a new method for model selection and more generally hyperparameter selection in machine learning. Minimum description length (MDL) is an established method for model selection, which is however not directly aimed at minimizing generalization error, which is often the primary goal in machine learni…

Model learns brevity by exposing to easy problems, improving efficiency without explicit length penalties.

Training deep neural networks often requires careful hyper-parameter tuning and significant computational resources. In this paper, we propose ConvTimeNet (CTN): an off-the-shelf deep convolutional neural network (CNN) trained on diverse univariate time series classification (TSC) source tasks. Once trained, CTN can be…

The standard state-of-the-art backend for text-independent speaker recognizers that use i-vectors or x-vectors, is Gaussian PLDA (G-PLDA), assisted by a Gaussianization step involving length normalization. G-PLDA can be trained with both generative or discriminative methods. It has long been known that heavy-tailed PLD…

I-BERT extends Transformer's self-attention to arbitrary input lengths.

Current end-to-end deep Reinforcement Learning (RL) approaches require jointly learning perception, decision-making and low-level control from very sparse reward signals and high-dimensional inputs, with little capability of incorporating prior knowledge. This results in prohibitively long training times for use on rea…

Transformer struggles with arithmetic length but improves with explicit structure encoding.

New language model shows context length impacts generation quality and reasoning ability.

Randomized positional encodings boost transformer performance on longer sequences.

The paper offers generalization bounds for Transformers that ignore sequence length.

Study improves voice conversion model with Mel-spectrogram augmentation.

New insights show embedding lengths correlate with semantic properties.

Recurrent models can produce infinite sequences, causing bias; new methods prevent this.

SentenceMIM learns rich latent representations for variable-length language data.

Mamba struggles with long context lengths, but spectrum scaling improves performance.

Transformers converge linearly to optimal models for Gaussian mixtures classification.

Reasoning models generate differently based on problem difficulty, not just length.

Study evaluates how limited training data affects streamflow predictions.

To understand the fundamental trade-offs between training stability, temporal dynamics and architectural complexity of recurrent neural networks~(RNNs), we directly analyze RNN architectures using numerical methods of ordinary differential equations~(ODEs). We define a general family of RNNs--the ODERNNs--by relating t…

Neural nets solve braid untangling up to length 20.

Neural network outperforms traditional methods in chaotic dynamics classification.

Meta-learning framework improves short utterance speaker recognition.

Bayesian optimization reduces RNN architecture search time.

Transformers learn chain-of-thought reasoning for longer problems, proving length generalization.

New bounds on efficiency for conformalized regression methods.

New approach reduces malware detection memory requirements and speeds up training.

Boosted conformal procedure improves prediction intervals.

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a …

New statistical models for predicting ranked preferences from partial orders.

We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms. In particular, given variable length data sequences, we first pass these sequences through our LSTM based structure and obtain fixed length sequences. We then find a decision functi…

New attention model removes softmax, improving sequence length bias.

Random Transformers behave like polynomial models in ICL with asymptotic growth.

One of the ubiquitous representation of long DNA sequence is dividing it into shorter k-mer components. Unfortunately, the straightforward vector encoding of k-mer as a one-hot vector is vulnerable to the curse of dimensionality. Worse yet, the distance between any pair of one-hot vectors is equidistant. This is partic…

We study the use of a time series encoder to learn representations that are useful on data set types with which it has not been trained on. The encoder is formed of a convolutional neural network whose temporal output is summarized by a convolutional attention mechanism. This way, we obtain a compact, fixed-length repr…

Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge …

The paper provides a theoretical justification for using stable SSM blocks in deep sequential models.

In current clinical practices, electroencephalograms (EEG) are reviewed and analyzed by trained neurologists to provide supports for therapeutic decisions. Manual reviews can be laborious and error prone. Automatic and accurate seizure/non-seizure classification methods are desirable. A critical challenge is that seizu…

We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves. We study instantiations of this framework based on random forests and Bayesian recur…

Recurrent auto-encoder model summarises sequential data through an encoder structure into a fixed-length vector and then reconstructs the original sequence through the decoder structure. The summarised vector can be used to represent time series features. In this paper, we propose relaxing the dimensionality of the dec…

Proving that next-token prediction makes language models generate coherent long documents.

New string kernels discover global properties through random feature maps, avoiding quadratic complexity.

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper net…