A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Hierarchical pretraining with slow-fast ODEs
Transformers with linear space and time complexity for accurate attention estimation.
SpGAT learns graph representations using spectral attention for efficiency.
PRISMA uses PDE residuals for fast, robust, and accurate inference.
WildCat efficiently compresses neural network attention mechanisms.
Efficiently accelerates attention calculation for Transformers with relative positional encoding.
New Performer model tackles long-sequence protein modeling.
FEM improves attention mechanisms by applying value-driven log-linear tilts.
FAST-DAD distills complex ensemble models into faster, more accurate individual models.
Gradient-based meta-learning has proven to be highly effective at learning model initializations, representations, and update rules that allow fast adaptation from a few samples. The core idea behind these approaches is to use fast adaptation and generalization -- two second-order metrics -- as training signals on a me…
Minimum attention improves reinforcement learning performance in high-dimensional dynamics.
Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for att…
Clustered attention improves transformer efficiency for large sequences.
The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…
Gradient descent converges geometrically to optimal self-attention parameters.
Fast Bayesian inference with adaptable priors for real-time applications.
Transformers become faster by linearizing self-attention.
FastGAT reduces GNN computation time by 10x using graph sparsification.
AReLU uses attention-based rectification to improve neural network performance.
Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to . We also introduce a) a variation on architecture and initialization to train deeper net…
Introduces Spectral Attention for better long-range time series forecasting.
Current deep learning based text classification methods are limited by their ability to achieve fast learning and generalization when the data is scarce. We address this problem by integrating a meta-learning procedure that uses the knowledge learned across many tasks as an inductive bias towards better natural languag…
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses …
New method prevents entropy collapse in Transformer training, leading to more stable and robust models.
Self-attention optimizers converge to optimal weights, revealing bias patterns.
Unified framework for sequence models using test-time regression.
Attention mechanisms and non-local mean operations in general are key ingredients in many state-of-the-art deep learning techniques. In particular, the Transformer model based on multi-head self-attention has recently achieved great success in natural language processing and computer vision. However, the vanilla algori…
Random utility theory models an agent's preferences on alternatives by drawing a real-valued score on each alternative (typically independently) from a parameterized distribution, and then ranking the alternatives according to scores. A special case that has received significant attention is the Plackett-Luce model, fo…
Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received increasing attention in the field of optimization for developing optimization algorithms with fast convergence. However, the studies of EBC in st…
Develops a privacy-preserving algorithm for sparse robust regression.
RNA-binding proteins (RBPs) play crucial roles in many biological processes, e.g. gene regulation. Computational identification of RBP binding sites on RNAs are urgently needed. In particular, RBPs bind to RNAs by recognizing sequence motifs. Thus, fast locating those motifs on RNA sequences is crucial and time-efficie…
A transformer model improves spell correction with hierarchical attention.
Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced seque…
Neural architecture search (NAS) searches architectures automatically for given tasks, e.g., image classification and language modeling. Improving the search efficiency and effectiveness have attracted increasing attention in recent years. However, few efforts have been devoted to understanding the generated architectu…
Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen consid…
New algorithms speed up attention computation for large models by limiting matrix entries.
Conditional density estimation generalizes regression by modeling a full density f(yjx) rather than only the expected value E(yjx). This is important for many tasks, including handling multi-modality and generating prediction intervals. Though fundamental and widely applicable, nonparametric conditional density estimat…
OLS is a special case of Transformer, revealing its linear nature.
iSTFTNet2 improves iSTFTNet's speed and lightness with 1D-2D CNN.
Paper tackles fast convergence for non-convex strongly-concave min-max problems.
A new, fast kernel test for large data.
Object detection in still images has drawn a lot of attention over past few years, and with the advent of Deep Learning impressive performances have been achieved with numerous industrial applications. Most of these deep learning models rely on RGB images to localize and identify objects in the image. However in some a…
Learning representation from relative similarity comparisons, often called ordinal embedding, gains rising attention in recent years. Most of the existing methods are based on semi-definite programming (\textit{SDP}), which is generally time-consuming and degrades the scalability, especially confronting large-scale dat…
We present a scalable and robust Bayesian inference method for linear state space models. The method is applied to demand forecasting in the context of a large e-commerce platform, paying special attention to intermittent and bursty target statistics. Inference is approximated by the Newton-Raphson algorithm, reduced t…
FastSGT improves accuracy in BERT-based DST for SGD datasets.
Clustering with fast algorithms large samples of high dimensional data is an important challenge in computational statistics. Borrowing ideas from MacQueen (1967) who introduced a sequential version of the -means algorithm, a new class of recursive stochastic gradient algorithms designed for the -medians loss cri…
Graph classification has recently received a lot of attention from various fields of machine learning e.g. kernel methods, sequential modeling or graph embedding. All these approaches offer promising results with different respective strengths and weaknesses. However, most of them rely on complex mathematics and requir…
Financial time-series forecasting has long been a challenging problem because of the inherently noisy and stochastic nature of the market. In the High-Frequency Trading (HFT), forecasting for trading purposes is even a more challenging task since an automated inference system is required to be both accurate and fast. I…