A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
gLSTM improves graph neural networks by increasing storage capacity to prevent over-squashing.
problem Over-squashing in GNNs collapses information from a large receptive field into a single vector, creating an information bottleneck.
method Introduced a new synthetic task to measure over-squashing and adapted ideas from sequence modeling to develop gLSTM, a novel GNN architecture with improved capacity.
result gLSTM architecture demonstrates strong performance on synthetic and real-world graph benchmarks, mitigating over-squashing.
Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent neural networks. To evaluate the models in the lifelong learning setting, we propo…
We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet. TabNet uses sequential attention to choose which features to reason from at each decision step, enabling interpretability and more efficient learning as the learning capacity is used for the most salient fea…
We investigate regularized algorithms combining with projection for least-squares regression problem over a Hilbert space, covering nonparametric regression over a reproducing kernel Hilbert space. We prove convergence results with respect to variants of norms, under a capacity assumption on the hypothesis space and a …
We propose a metric, Layer Saturation, defined as the proportion of the number of eigenvalues needed to explain 99% of the variance of the latent representations, for analyzing the learned representations of neural network layers. Saturation is based on spectral analysis and can be computed efficiently, making live ana…
We extend the adaptive regression spline model by incorporating saturation, the natural requirement that a function extend as a constant outside a certain range. We fit saturating splines to data using a convex optimization problem over a space of measures, which we solve using an efficient algorithm based on the condi…
A homogeneously saturated equation for the time development of the price of a financial asset is presented and investigated for the pricing of European call options using noise that is distributed as a Student's t-distribution. In the limit that the saturation parameter of the equation equals zero, the standard model o…
Water saturation is an important property in reservoir engineering domain. Thus, satisfactory classification of water saturation from seismic attributes is beneficial for reservoir characterization. However, diverse and non-linear nature of subsurface attributes makes the classification task difficult. In this context,…
A recent paper suggests that Deep Neural Networks can be protected from gradient-based adversarial perturbations by driving the network activations into a highly saturated regime. Here we analyse such saturated networks and show that the attacks fail due to numerical limitations in the gradient computations. A simple s…
Modelling long-term dependencies is a challenge for recurrent neural networks. This is primarily due to the fact that gradients vanish during training, as the sequence length increases. Gradients can be attenuated by transition operators and are attenuated or dropped by activation functions. Canonical architectures lik…
The time development of the price of a financial asset is considered by constructing and solving Langevin equations for a homogeneously saturated model, and for comparison, for a standard model and for a logistic model. The homogeneously saturated model uses coupled rate equations for the money supply and for the price…
Estimates the generalization error of deep neural networks without relying on capacity measures.
problem Understanding how generalization error scales with training data for deep neural networks.
method Derives estimates of generalization error for deep networks based on two assumptions: zero training error and error probability proportional to distance to nearest training point.
result Estimates the generalization error of DNNs as O(1/(δN^{1/d})), matching experimental behavior.
Service-induced congestion in memory-constrained LLM serving
problem Service-induced congestion in memory-constrained large language model (LLM) serving
method Developing a discrete-time dynamical model of memory-constrained LLM inference
result The system converges to a unique worst-case limit cycle that is asymptotically stable outside a Lebesgue-measure-zero exact-capture set, with throughput losses as large as 50%.
A general nonlinear logistic equation has been proposed to model long-time saturation in industrial growth. An integral solution of this equation has been derived for any arbitrary degree of nonlinearity. A time scale for the onset of nonlinear saturation in industrial growth can be estimated from an equipartition cond…
As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networ…
Sign information is the key to overcoming the inevitable saturation error in compressive sensing systems, which causes information loss and results in bias. For sparse signal recovery from saturation, we propose to use a linear loss to improve the effectiveness from existing methods that utilize hard constraints/hinge …
We give a method to compute presentations of saturated cluster modular groups. Using this, we obtain finite presentations of the saturated cluster modular groups of finite mutation type X6 and X7. We verify that the cluster modular groups of finite mutation type E6, E7, $\widetilde{E…
Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions t…