arXiv research

AdamP optimizes momentum-based optimizers for scale-invariant weights, improving model performance.

New method trains deep networks robustly without adaptive methods.

We consider a variant of online convex optimization in which both the instances (input vectors) and the comparator (weight vector) are unconstrained. We exploit a natural scale invariance symmetry in our unconstrained setting: the predictions of the optimal comparator are invariant under any linear transformation of th…

We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.

It is well known that neural networks with rectified linear units (ReLU) activation functions are positively scale-invariant. Conventional algorithms like stochastic gradient descent optimize the neural networks in the vector space of weights, which is, however, not positively scale-invariant. This mismatch may lead to…

Using the monotonicity formulas of Colding and Minicozzi, we prove that on any complete, non-parabolic Riemannian manifold $(M^3, g)$ with non-negative Ricci curvature, the asymptotic weighted scaling invariant integral of scalar curvature has an explicit bound in form of asymptotic volume ratio.

In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradi…

Scales attention for long contexts in LLMs.

Three training regimes found for scale-invariant neural networks on the sphere.

In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation …

Bilateral trade relationships in the international level between pairs of countries in the world give rise to the notion of the International Trade Network (ITN). This network has attracted the attention of network researchers as it serves as an excellent example of the weighted networks, the link weight being defined …

Proves uniqueness of Ricci flow with scaling invariant estimates.

Critical points of scale-invariant curvature energies in 4D are analytic.

New bounds on self-normalized martingales improve online linear regression performance.

Proves conditions for Willmore surfaces to have finite ends or finite total curvature.

This paper extends compositional data analysis using graph signal processing.

Power iteration has been generalized to solve many interesting problems in machine learning and statistics. Despite its striking success, theoretical understanding of when and how such an algorithm enjoys good convergence property is limited. In this work, we introduce a new class of optimization problems called scale …

Investigates energy minimizers and critical points of scale-invariant tangent-point energies for knots.

We study scale invariant but not necessarily conformal invariant deformations of non-relativistic conformal field theories from the dual gravity viewpoint. We present the corresponding metric that solves the Einstein equation coupled with a massive vector field. We find that, within the class of metric we study, when w…

New learning dynamics achieve fast convergence in games without needing to know utility scales.

Study shows how near crushing singularities, Kasner-like regions can exist.

Adam performs better with equal momentum parameters, revealing a gradient scale invariance principle.

SAM improves deep learning tasks by promoting balancedness, reducing outlier impact.

The importance of the power law has been well realized in econophysics over the last decade. For instance, the distribution of the rate of stock price variation and of personal assets show the power law. While these results reveal the striking scale invariance of financial markets, the behaviour of price in real econom…

The condition number predicts efficient information encoding in neural units, aiding model fine-tuning.

The paper analyzes various normalization methods in deep learning.

We develop a scale-invariant truncated Lévy (STL) process to describe physical systems characterized by correlated stochastic variables. The STL process exhibits Lévy stability for the probability density, and hence shows scaling properties (as observed in empirical data); it has the advantage that all moments are fini…

ASAM improves deep neural network generalization by adapting sharpness to scale.

The study proves a new inequality and formula for manifolds with non-negative Ricci curvature.

It was empirically confirmed by Keskar et al.\cite{SharpMinima} that flatter minima generalize better. However, for the popular ReLU network, sharp minimum can also generalize well \cite{SharpMinimacan}. The conclusion demonstrates that the existing definitions of flatness fail to account for the complex geometry of Re…

Unified framework for scale-invariant representation learning using MAPCA.

The paper studies minimal resistance dynamics in radial fields, finding unique solutions for incompressible flows.

Proves long-time Ricci flow existence and topological rigidity for pinched integral curvature manifolds.

New approach improves classification guarantees by focusing on direction rather than regression risk.

We develop a regularity theory for extremal knots of scale invariant knot energies defined by J. O'hara in 1991. This class contains as a special case the Möbius energy. For the Möbius energy, due to the celebrated work of Freedman, He, and Wang, we have a relatively good understanding. Their approch is crucially based…

A new metric framework for weighted projective spaces improves clustering and analysis.

This paper analyzes implicit bias in Deep Linear Discriminant Analysis.

Wavelet scattering spectra model non-Gaussian time-series, proving scale invariance for self-similar processes.

A new method detects small holes in noisy data.

The study improves Poincaré and log-Sobolev inequalities on hyperbolic spaces.

Proves planarity and convexity for ancient solutions of mean curvature flow.

NOTEARS fails to identify true causal relationships from data.

The results of R^2 dynamical random surface model (2-dimensional quantum gravity with a $R^2$ term) are applied to explain the personal income distribution. A scale invariance exists if there is not the $R^2$ term in the action. The R^2 term provides a typical scale and breaks the scale invariance explicitly in the low…

This paper investigates the effectiveness of decoupled weight decay at the start of training.

We propose a new high dimensional semiparametric principal component analysis (PCA) method, named Copula Component Analysis (COCA). The semiparametric model assumes that, after unspecified marginally monotone transformations, the distributions are multivariate Gaussian. COCA improves upon PCA and sparse PCA in three as…

The statistical properties of the multipliers of the absolute returns are investigated using one-minute high-frequency data of financial time series. The multiplier distribution is found to be independent of the box size $s$ when $s$ is larger than some crossover scale, providing direct evidence of the existence of sca…

We develop an entropic framework to model the dynamics of stocks and European Options. Entropic inference is an inductive inference framework equipped with proper tools to handle situations where incomplete information is available. The objective of the paper is to lay down an alternative framework for modeling dynamic…

Reduces symplectic Hamiltonian systems to contact systems, realizing Poincaré's dream.