This paper precisely estimates transformer derivatives for explicit learning guarantees.
problem Computing fully-explicit generalization bounds for transformers with precise higher-order derivative estimates.
method Analyzes and estimates all higher-order derivatives of transformers with multiple attention heads and layer normalization.
result Obtains explicit pathwise generalization bounds for transformers learning from non-i.i.d. samples.
New properties of weighted Hilbert transform derived, useful for imaging applications.
problem Properties of weighted Hilbert transform in L2 spaces.
method Derivation of Plancherel-like equations, coerciveness, iterative sequences.
result Iterative sequences for inversion are applicable to specific cases.
We reduce boundary determination of an unknown function and its normal derivatives from the (possibly weighted and attenuated) broken ray data to the injectivity of certain geodesic ray transforms on the boundary. For determination of the values of the function itself we obtain the usual geodesic ray transform, but for…
Paper derives a simplified formula for Expected Improvement using log-transformed data.
problem Challenges in enhancing Bayesian optimization with Expected Improvement.
method Derives a closed form of Expected Improvement for Gaussian process trained on log-transformed objective.
result Provides a simplified formula for Expected Improvement.
We derive bounds for a notion of adversarial risk, designed to characterize the robustness of linear and neural network classifiers to adversarial perturbations. Specifically, we introduce a new class of function transformations with the property that the risk of the transformed functions upper-bounds the adversarial r…
On simple geodesic disks of constant curvature, we derive new functional relations for the geodesic X-ray transform, involving a certain class of elliptic differential operators whose ellipticity degenerates normally at the boundary. We then use these relations to derive sharp mapping properties for the X-ray transform…
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
The spherical Radon transform on the unit sphere can be regarded as a member of the analytic family of suitably normalized generalized cosine transforms. We derive new formulas for these transforms and apply them to study classes of intersections bodies in convex geometry.
Study projective derivative cocycles for circle diffeomorphisms.
problem Understanding reducibility and almost reducibility in circle diffeomorphisms.
method Computing precise expressions for projective derivative cocycles and extending to 3-torus.
result Generalization of results to diagonal action on 3-torus.
Unified method for deriving ridgelet transforms for various neural network architectures.
problem Deriving closed-form expressions for ridgelet transforms in modern neural network architectures.
method Unified Fourier slice method to derive ridgelet transforms for diverse neural network types.
result Systematic method to derive ridgelet transforms for various neural network architectures.
Study light ray transform in pseudo-Euclidean space, derive inversion formula, and prove stability.
problem Analyzing light ray transform in pseudo-Euclidean space.
method Investigate normal operator, derive inversion formula, analyze as Fourier Integral Operator.
result Derive an inversion formula and prove stability estimates.
Study Lp boundedness of Riesz transform on differential forms for certain manifolds.
problem Investigate Lp-boundedness of the covariant Riesz transform on differential forms. method Analyze Lp-boundedness on weighted Riemannian manifolds under curvature-dimension and lower bound conditions. result Derive Calderón-Zygmund inequality for 1<p≤2 under curvature-dimension condition. Most of the empirical studies on stochastic volatility dynamics favor the 3/2 specification over the square-root (CIR) process in the Heston model. In the context of option pricing, the 3/2 stochastic volatility model is reported to be able to capture the volatility skew evolution better than the Heston model. In this …
A generalization of exterior calculus is considered by allowing the partial derivatives in the exterior derivative to assume fractional orders. That is, a fractional exterior derivative is defined. This is found to generate new vector spaces of finite and infinite dimension, fractional differential form spaces. The def…
Despite being studied for over a century, the use of quadrupoles have been limited to Cartesian coordinates in flat spacetime due to the incorrect transformation rules used to define them. Here the correct transformation rules are derived, which are particularly unusual as they involve second derivatives of the coordin…
Formulae for mass and angular momentum transformations under BMS transformations derived from curvature and metric.
problem Deriving transformation formulae for mass and angular momentum under BMS transformations.
method Two approaches: from curvature tensor and metric coefficients.
result Exact expressions for Drey-Streubel angular momentum of a general section.
In this paper, we generalize the polar transforms of spacelike isothermic surfaces in Q14 to n-dimensional pseudo-Riemannian space forms Qrn. We show that there exist c−polar spacelike isothermic surfaces derived from a spacelike isothermic surface in Qrn, which are into Srn+1(c), Hr−1n+1(c)…
A new EM gradient algorithm for mixture models with skewed components.
problem Fitting mixture models with skewed components derived from the Manly transformation.
method Proposes an alternative EM gradient algorithm using Newton's method for better parameter updates.
result Shows improved convergence and parameter estimation compared to the Nelder-Mead optimization.
This work further develops the properties of fractional differential forms. In particular, finite dimensional subspaces of fractional form spaces are considered. An inner product, Hodge dual, and covariant derivative are defined. Coordinate transformation rules for integral order forms are also computed. Matrix order f…
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
Derives integral formulae on weighted manifolds.
problem No specific problem stated; focuses on mathematical derivations.
method Introduces weighted mean sigma-r curvature and uses weighted Newton transformations.
result Derives integral formulae generalizing previous work.
Let M be a complete non-compact Riemannian manifold. In this paper, we derive sufficient conditions on metric perturbation for stability of Lp-boundedness of the Riesz transform, p∈(2,∞). We also provide counter-examples regarding in-stability for Lp-boundedness of Riesz transform.
A new method for pricing derivatives using self-exciting dynamics and finite-difference transforms.
problem Pricing derivatives with accumulated marks using a self-exciting marked point process.
method Derive discounted pricing equation as a PIDE, transform to one-dimensional PIDEs, use Laplace/Fourier transform, approximate jump term, solve using finite difference scheme.
result Efficiently price derivatives with accumulated marks using a novel finite-difference and transform approach.
We study Christoffel and Darboux transforms of discrete isothermic nets in 4-dimensional Euclidean space: definitions and basic properties are derived. Analogies with the smooth case are discussed and a definition for discrete Ribaucour congruences is given. Surfaces of constant mean curvature are special among all iso…
Normality equations describe Newtonian dynamical systems admitting normal shift of hypersurfaces. These equations were first derived in Euclidean geometry. Then very soon they were rederived in Riemannian and in Finslerian geometry. Recently I have found that normality equations can be derived in geometry given by clas…
Researchers present and compare different representations of dissipative Hamiltonian DAE systems.
problem Understanding and transforming dissipative Hamiltonian DAE systems.
method Global geometric and algebraic points of view, translations between representations, characterizations, and numerical methods for computing structural information.
result A general DAE system can be transformed into a dissipative Hamiltonian or port-Hamiltonian DAE system.
Small neural networks embed arbitrary metric spaces into Gaussian mixtures.
problem Embedding arbitrary metric spaces into a fixed space with low distortion.
method Probabilistic transformers of small depth and width.
result Embeddings with low metric distortion for various metric spaces.
Estimates for covariant derivatives and Riesz transforms on differential forms.
problem Bounding covariant derivatives and Riesz transforms on differential forms.
method Use Bismut derivative formula to prove heat kernel bounds and Riesz transform boundedness.
result Formulate and prove conjecture on boundedness of covariant local Riesz-transforms in L^p.
Learning multiple tasks across heterogeneous domains is a challenging problem since the feature space may not be the same for different tasks. We assume the data in multiple tasks are generated from a latent common domain via sparse domain transforms and propose a latent probit model (LPM) to jointly learn the domain t…
Measures equivariance in vision models using Lie derivative.
problem Understanding the role of equivariance in recent vision models.
method Introducing Lie derivative to measure equivariance with strong mathematical foundations and minimal hyperparameters.
result Many violations of equivariance can be linked to spatial aliasing in network layers, and larger models tend to display more equivariance.
This paper addresses anisotropy in Transformer models, providing geometric insights and empirical support.
problem Anisotropy phenomenon in Transformer models, challenging their geometric interpretation.
method Derive geometric arguments and use concept-based mechanistic interpretability during training.
result Activation-derived directions capture large gradient energy and a larger share of gradient anisotropy than normal controls.
Develops a new stochastic volatility model for temperature derivatives.
problem Assessing risk related to temperature volatility.
method Conditional Least Squares and Fourier transform techniques.
result Better assessment of temperature volatility risk.
The paper derives Pizzetti formulae and inverts the Radon transform on spheres.
problem Inverting the Radon transform on spheres.
method Obtained Pizzetti-type formulae on sphere regions, used delta distributions, and derived inversion formulae.
result Derived Pizzetti formulae and inversion formulae for the Radon transform on spheres.
In the present paper we consider the problem of local equivalence of second order ODEs which are cubic in second derivative under the action of the pseudogroup of contact transformations. We show how it may be reduced to the equivalence problem of 2-webs in R3 under the action of finite-dimensional group, a…
In the 3-gauge theory, a 3-connection is given by a 1-form A valued in the Lie algebra g, a 2-form B valued in the Lie algebra h and a 3-form C valued in the Lie algebra l, where (g,h,l) constitutes a differential 2-crossed modu…
The paper derives statistics of multi-factor functions from their Fourier transforms.
problem Deriving statistics of multi-factor functions from Fourier transforms.
method Developed an m-Coefficient/Index Annihilation Theorem to analyze the moments of a function from its Fourier transform.
result The mth moment of a function becomes a series of terms, each with precisely m Fourier coefficients, and the indices sum to zero.
Researchers derive an explicit Laplace transform for integrated Volterra Wishart process.
problem Modeling and pricing financial instruments with complex covariance structures.
method Explicit expression for conditional Laplace transform of integrated Volterra Wishart process, linking to matrix Riccati equations.
result Derivation of Laplace transform for a special case of convolution kernel, leading to efficient pricing methods.
We solve for functions from their truncated Hilbert transforms using Chebyshev series.
problem Finding functions from their truncated Hilbert transforms.
method Express functions in Chebyshev series and numerically estimate coefficients.
result Numerical methods work well for extrapolating functions from truncated Hilbert transforms.
Transformers interpreted as probabilistic Laplacian Eigenmaps steps.
problem Improving transformer performance through probabilistic interpretation.
method Probabilistic Laplacian Eigenmaps model derivation and graph diffusion step.
result Subtracting identity from attention matrix improves transformer performance.
We derive precise transformation formulas for synthetic lower Ricci bounds under time change. More precisely, for local Dirichlet forms we study how the curvature-dimension condition in the sense of Bakry-Emery will transform under time change. Similarly, for metric measure spaces we study how the curvature-dimension c…
Survey on inverse exponential Radon transform methods.
problem Analytical methods for inverse exponential Radon transform.
method Derivation of classical inversion formula, finite Hilbert transform, exact reconstruction from partial measurements, diverging-beam data.
result Exact reconstruction from 180 degree data using finite Hilbert transform.
Derives an empirical capacity model for self-attention neural networks.
problem Theoretical capacity of large transformer models is not fully utilized by current optimization algorithms.
method Analyzes memory capacity of transformers using synthetic training data and common training algorithms.
result Derives an empirical capacity model (ECM) for a generic transformer.
New PAC-Bayes bounds derived using Legendre transform and f-divergences.
problem Deriving PAC-Bayes bounds under various assumptions.
method Combining Legendre transform and Fenchel--Young inequality to derive change-of-measure inequalities.
result Extended PAC-Bayesian guarantees under tailored assumptions.
A new metric HSW derived from hierarchical Radon Transform addresses computational bottlenecks in sliced Wasserstein.
problem Computational inefficiency of sliced Wasserstein in high-dimensional settings with few supports.
method Hierarchical Radon Transform (HRT) and bottleneck projections to reduce projection number.
result HSW metric derived from HRT is computationally efficient and maintains metric properties.
Estimates heat kernel gradients on fractal-like cable systems.
problem Bounding gradients of heat kernels on complex fractal structures.
method Pointwise upper estimates for heat kernel gradients.
result Derives Lp-boundedness of quasi-Riesz transforms. We derive a permutability theorem for the Christoffel, Goursat and Darboux transformations of isothermic surfaces. As a consequence we obtain a simple proof of a relation between Darboux pairs of minimal surfaces in Euclidean space, curved flats in the 2-sphere and flat fronts in hyperbolic space.
Transformers use a unique Hessian structure that differs from classical networks, affecting optimization.
problem Understanding the unique optimization landscape of Transformers.
method Theoretical Hessian analysis of a single self-attention layer in Transformers.
result Transformers have a highly non-linear Hessian structure, distinguishing them from classical networks.
The geometry of the total space of a principal bundle with regard to the action of the bundle's structure group is elegantly described by the bundle's operation, a collection of derivations consisting of the de Rham differential and the contraction and Lie derivatives of all vertical vector fields and satisfying the si…