arXiv research

Paper proves KRR saturation effect for smooth functions.

We extend the adaptive regression spline model by incorporating saturation, the natural requirement that a function extend as a constant outside a certain range. We fit saturating splines to data using a convex optimization problem over a space of measures, which we solve using an efficient algorithm based on the condi…

Modelling long-term dependencies is a challenge for recurrent neural networks. This is primarily due to the fact that gradients vanish during training, as the sequence length increases. Gradients can be attenuated by transition operators and are attenuated or dropped by activation functions. Canonical architectures lik…

Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions t…

As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networ…

This paper explores saturation effects in spectral algorithms over large dimensions.

A method to analyze neural network performance by measuring layer saturation.

To improve how neural networks function it is crucial to understand their learning process. The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task. However, empirical evid…

Generative adversarial networks (GANs) can be interpreted as an adversarial game between two players, a discriminator D and a generator G, in which D learns to classify real from fake data and G learns to generate realistic data by "fooling" D into thinking that fake data is actually real data. Currently, a dominating …

Noiseless KRR achieves optimal rates and exhibits saturation effects.

Study network equilibria in saturated systems, revealing how small shocks can trigger major losses.

New method resolves density ratio estimation saturation issues.

New insights into neural network training efficiency.

We propose a metric, Layer Saturation, defined as the proportion of the number of eigenvalues needed to explain 99% of the variance of the latent representations, for analyzing the learned representations of neural network layers. Saturation is based on spectral analysis and can be computed efficiently, making live ana…

This work shows MLPs can approximate monotonic functions without bounded activations.

The paper normalizes Poisson saturation of coregular submanifolds.

Extends return extrapolation to nonlinear, asymmetric functions under stochastic volatility.

This paper deals with robust regression and subspace estimation and more precisely with the problem of minimizing a saturated loss function. In particular, we focus on computational complexity issues and show that an exact algorithm with polynomial time-complexity with respect to the number of data can be devised for r…

The paper accelerates regression algorithms by identifying saturated coordinates.

A homogeneously saturated equation for the time development of the price of a financial asset is presented and investigated for the pricing of European call options using noise that is distributed as a Student's t-distribution. In the limit that the saturation parameter of the equation equals zero, the standard model o…

33 curves on a 3-genus surface, all intersecting at most once.

Water saturation is an important property in reservoir engineering domain. Thus, satisfactory classification of water saturation from seismic attributes is beneficial for reservoir characterization. However, diverse and non-linear nature of subsurface attributes makes the classification task difficult. In this context,…

A recent paper suggests that Deep Neural Networks can be protected from gradient-based adversarial perturbations by driving the network activations into a highly saturated regime. Here we analyse such saturated networks and show that the attacks fail due to numerical limitations in the gradient computations. A simple s…

Neural marked point processes show saturation with complexity, leading to new simple architectures.

NS-GAN mode collapse due to sample weighting inversion, solved with MM-nsat.

We extend return extrapolation to incorporate asymmetry and saturation, finding that asymmetric nonlinear extrapolation leads to lower welfare loss.

Entrocraft addresses RL performance saturation in LLMs by customizing entropy curves.

New insights explain speedup saturation in distributed learning with large batches and delays.

The time development of the price of a financial asset is considered by constructing and solving Langevin equations for a homogeneously saturated model, and for comparison, for a standard model and for a logistic model. The homogeneously saturated model uses coupled rate equations for the money supply and for the price…

Proposes a method to improve pWCET estimation for heavy-tailed distributions.

New theory explains GAN's high quality but low diversity.

A general nonlinear logistic equation has been proposed to model long-time saturation in industrial growth. An integral solution of this equation has been derived for any arbitrary degree of nonlinearity. A time scale for the onset of nonlinear saturation in industrial growth can be estimated from an equipartition cond…

The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tra…

Sign information is the key to overcoming the inevitable saturation error in compressive sensing systems, which causes information loss and results in bias. For sparse signal recovery from saturation, we propose to use a linear loss to improve the effectiveness from existing methods that utilize hard constraints/hinge …

We give a method to compute presentations of saturated cluster modular groups. Using this, we obtain finite presentations of the saturated cluster modular groups of finite mutation type $X_6$ and $X_7$. We verify that the cluster modular groups of finite mutation type $\widetilde{E}_6$, $\widetilde{E}_7$, $\widetilde{E…

New examples show deletion type admissible pairs can be rigid under rational saturation.

Much effort has been devoted to understanding the decisions of deep neural networks in recent years. A number of model-aware saliency methods were proposed to explain individual classification decisions by creating saliency maps. However, they are not applicable when the parameters and the gradients of the underlying m…

We define the representation ring of a saturated fusion system $\mathcal F$ as the Grothendieck ring of the semiring of $\mathcal F$-stable representations, and study the dimension functions of $\mathcal F$-stable representations using the transfer map induced by the characteristic idempotent of $\mathcal F$. We find a…

Study shows scaling up models doesn't always improve downstream tasks.

Advances of modern sensing and sequencing technologies generate a deluge of high dimensional space-temporal physiological and next-generation sequencing (NGS) data. Physiological traits are observed either as continuous random functions, or on a dense grid and referred to as function-valued traits. Both physiological a…

Soft-Radial Projection solves gradient saturation in constrained deep learning.

Unified theory of deep neural networks with diverse activations.

The paper extends kernel ridge regression to product kernels and reveals new convergence behaviors.

Study shows how close functions are to optimal in Riemannian manifolds.

New formulations for Ricci flows without smoothness.

For a stable marginally outer trapped surface (MOTS) in an axially symmetric spacetime with cosmological constant $Λ> 0$ and with matter satisfying the dominant energy condition, we prove that the area $A$ and the angular momentum $J$ satisfy the inequality $8π|J| \le A\sqrt{(1-ΛA/4π)(1-ΛA/12π)}$ which is saturated pre…

New methods for estimating causal effects in hidden variable DAGs.

Recurrent neural networks trained on regular languages exhibit stable states that can recover from noise.