arXiv research

LSTM LMs improve speech recognition by 8% using lattice rescoring.

The dominant language models (LMs) such as n-gram and neural network (NN) models represent sentence probabilities in terms of conditionals. In contrast, a new trans-dimensional random field (TRF) LM has been recently introduced to show superior performances, where the whole sentence is modeled as a random field. In thi…

Improved neural language models trained with dynamic noise-contrastive estimation.

LM-SNNs use lattice maps to classify and cluster images.

KiloGrams finds top-k large n-grams for malware classification.

We present NN-grams, a novel, hybrid language model integrating n-grams and neural networks (NN) for speech recognition. The model takes as input both word histories as well as n-gram counts. Thus, it combines the memorization capacity and scalability of an n-gram model with the generalization ability of neural network…

Proposes a faster Transformer decoding method by truncating target-side self-attention windows.

Improved speech recognition model with better performance.

Improved neural TRF LMs for speech recognition with NCE and CNN integration.

This paper investigates teacher hacking during language model distillation and proposes methods to mitigate it.

New deep architectures inspired by numerical differential equations improve performance.

Here we present a novel approach to statistical analysis of financial time series. The approach is based on $n$-grams frequency dictionaries derived from the quantized market data. Such dictionaries are studied by evaluating their information capacity using relative entropy. A specific quantization of (originally conti…

Let LM be the semigroup of non-degenerate based loops with a fixed initial/final frame in a Riemannian manifold M of dimension at least three. We compare the topology of LM to that of the loop space Omega FTM on the bundle of frames in the tangent bundle of M. We show that Omega FTM is the group completion of LM, and p…

Multilingual LM improves low-resource language modeling.

We introduce a probabilistic approach to the LMS filter. By means of an efficient approximation, this approach provides an adaptable step-size LMS algorithm together with a measure of uncertainty about the estimation. In addition, the proposed approximation preserves the linear complexity of the standard LMS. Numerical…

We consider adaptive system identification problems with convex constraints and propose a family of regularized Least-Mean-Square (LMS) algorithms. We show that with a properly selected regularization parameter the regularized LMS provably dominates its conventional counterpart in terms of mean square deviations. We es…

Let M be a closed, connected manifold, and LM its loop space. In this paper we describe closed string topology operations in h_*(LM), where h_* is a generalized homology theory that supports an orientation of M. We will show that these operations give h_*(LM) the structure of a unital, commutative Frobenius algebra wit…

This paper shows RL with KL penalties is equivalent to Bayesian inference for fine-tuning LMs.

End-to-end ASR system uses context n-grams for better speech recognition.

Improved neural model predicts gender from tweets.

Neural TRFs improve speech recognition models with fewer parameters and faster inference.

In this paper, we prove that for every Finsler $n$-dimensional sphere $(S^{n},F)$ with reversibility $\lm$ and flag curvature $K$ satisfying $\left(\frac{\lm}{1+\lm}\right)^2<K\le 1$, either there exist infinitely many closed geodesics, or there exist at least two elliptic closed geodesics and each linearized Poincaré …

We propose a version of least-mean-square (LMS) algorithm for sparse system identification. Our algorithm called online linearized Bregman iteration (OLBI) is derived from minimizing the cumulative prediction error squared along with an l1-l2 norm regularizer. By systematically treating the non-differentiable regulariz…

Paper proves a spinorial version of Aubin's estimate for the Yamabe problem.

Let M be a closed, oriented, n -manifold, and LM its free loop space. Chas and Sullivan defined a commutative algebra structure in the homology of LM, and a Lie algebra structure in its equivariant homology. These structures are known as the string topology loop product and string bracket, respectively. In this paper w…

Study compares N-gram and RNN models for chord prediction on large datasets.

New methods improve integration of external LMs with AED models.

LMs perform poorly in true few-shot learning without held-out examples.

Using the Wodzicki residue, we build Wodzicki-Chern-Simons (WCS) classes in $H^{2k-1}(LM)$ associated to the residue Chern character on the loop space $LM$ of a Riemannian manifold $M^{2k-1}$. These WCS classes are associated to the $L^2$ connection and the Sobolev $s=1$ connections on $LM.$ The WCS classes detect seve…

We present power low rank ensembles (PLRE), a flexible framework for n-gram language modeling where ensembles of low rank matrices and tensors are used to obtain smoothed probability estimates of words in context. Our method can be understood as a generalization of n-gram modeling to non-integer n, and includes standar…

Recent work on end-to-end automatic speech recognition (ASR) has shown that the connectionist temporal classification (CTC) loss can be used to convert acoustics to phone or character sequences. Such systems are used with a dictionary and separately-trained Language Model (LM) to produce word sequences. However, they a…

Paper improves natural language understanding with less data using a new training method.

Generative Adapter adapts LMs with a single forward pass, reducing inference overhead.

Paper criticizes mFLMS algorithm for power signal estimation.

We study the existence of $S^1$-equivariant characteristic classes on certain natural infinite rank bundles over the loop space $LM$ of a manifold $M$. We discuss the different $S^1$-equivariant cohomology theories in the literature and clarify their relationships. We attempt to use $S^1$-equivariant Chern-Weil techniq…

Study shows challenges in converting RNNs to FSMs due to computational complexity.

Neural language models (LMs) based on recurrent neural networks (RNN) are some of the most successful word and character-level LMs. Why do they work so well, in particular better than linear neural LMs? Possible explanations are that RNNs have an implicitly better regularization or that RNNs have a higher capacity for …

Given a smooth closed manifold M with a family {L_i} of closed submanifolds, we consider the free loop space LM and the spaces PM(L_i,L_j) of open strings (paths g:[0,1]->M with g(0) in L_i, and g(1) in L_j). We construct string topology operations resulting in an open-closed TQFT on the family (h_*(LM),h_*(PM(L_i,L_j)…

Paper reduces communication costs in distributed LMS algorithms.

New framework assesses LM uncertainty without thresholding.

Chas and Sullivan have defined an intersection-type product on the homology of the free loop space LM of an oriented manifold M. In this paper we show how to extend this construction to a topological conformal field theory of degree d. In particular, we get operations on the homology of LM which are parameterized by th…

Paper shows how LSTM can remember long sequences by attending to persisted information.

Let $G \to P \to M$ be a flat principal bundle over a closed and oriented manifold $M$ of dimension $m=2d$. We construct a map of Lie algebras $Ψ: \H_{2\ast} (L M) \to ø(\Mc)$, where $\H_{2\ast} (LM)$ is the even dimensional part of the equivariant homology of $LM$, the free loop space of $M$, and $\Mc$ is the Maurer-C…

Let M be one of the projective spaces CP^n, HP^n for n>1 or the Cayley projective plane OP^2, and let LM denote the free loop space on M. Using Morse theory methods, we prove that the suspension spectrum of (LM)_+ is homotopy equivalent to the suspension spectrum of M_+ wedge a family of Thom spaces of explicit vector …

In LM, we proved a family version of the famous Witten rigidity theorems and several family vanishing theorems for elliptic genera. In this paper, we gerenalize our theorems LM in two directions. First we establish a family rigidity theorem for the Dirac operator on loop space twisted by general positive energy loop gr…

In this work we give a comprehensive overview of the time consistency property of dynamic risk and performance measures, focusing on a the discrete time setup. The two key operational concepts used throughout are the notion of the LM-measure and the notion of the update rule that, we believe, are the key tools for stud…

Let $M$ be a closed, oriented manifold of dimension $d$. Let $LM$ be the space of smooth loops in $M$. Chas and Sullivan recently defined a product on the homology $H_*(LM)$ of degree $-d$. They then investigated other structure that this product induces, including a Batalin -Vilkovisky structure, and a Lie algebra str…

Proposes CLRS-Text, a new benchmark for evaluating LM reasoning capabilities.