Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
Paper proposes HTAF for stable training of binary neural networks.
problem Challenges in training binary neural networks with gradient-based optimization.
method HTAF is a smooth approximation to the Heaviside function that enables stable training.
result HTAF enables stable training of various binary neural networks with gradient-based optimization.
Training feedforward neural networks with standard logistic activations is considered difficult because of the intrinsic properties of these sigmoidal functions. This work aims at showing that these networks can be trained to achieve generalization performance comparable to those based on hyperbolic tangent activations…
We developed safe and ranged approximations for ReLU, tanh, and sigmoid functions to reduce neural network training time.
problem Expensive computation of hyperbolic tangent and sigmoid functions in neural networks.
method Function approximation techniques to create safe and ranged approximations.
result 10% to 37% improvement in training times on CPU and 20% to 53% improvement in ranged cases.
We simplify word embeddings by removing sigmoid in SGNS, revealing connections to hyperbolic spaces.
problem Improving word embeddings quality and understanding their relationship with hyperbolic spaces.
method Analyzing squashed shifted PMI matrix and its relation to graph properties and hyperbolic geometry.
result Word embeddings can be connected to hyperbolic spaces through squashed shifted PMI matrix.
Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value …
The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tra…
Study shows how certain foliations in unit tangent bundles behave.
problem Characterizing behavior of foliations in unit tangent bundles.
method Analyzing intersections and properties of foliations.
result Certain partially hyperbolic diffeomorphisms are collapsed Anosov flows.
We equip the whole tangent space TM to a hyperbolic manifold M (of constant sectional curvature -1) with a natural metric in an intrinsic way, so that the isometries of M extend to isometries of TM by holomorphic continuation. The image to the tangent space to a geodesic is equivalent to a hyperbolic disk. In t…
Researchers create metrics on hyperbolic space's tangent bundle.
problem Constructing metrics on the unit tangent bundle of hyperbolic space.
method Using Hopf coordinates and Busemann functions, they constructed a flow-invariant metric.
result The unit tangent bundle of hyperbolic space is a homogeneous space under specific groups.
We show, using two different approaches, that there exists a family of Riemannian metrics on the tangent bundle of a two-sphere, which induces metrics of constant curvature on its unit tangent bundle. In other words, given such a metric on the tangent bundle of a two-sphere, the Hopf map is identified with a Riemannian…
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of …
Abstract Neural Networks (ANNs) improve DNN verification efficiency.
problem Efficiently verify safety-critical DNNs without slowing exponentially.
method Introduces ANNs that use abstract domains and activation functions to overapproximate DNNs.
result ANNs can soundly overapproximate DNNs with fewer nodes, improving verification efficiency.
Gating is a key technique used for integrating information from multiple sources by long short-term memory (LSTM) models and has recently also been applied to other models such as the highway network. Although gating is powerful, it is rather expensive in terms of both computation and storage as each gating unit uses a…
Study extends GNN VC dimension bounds to Pfaffian activation functions.
problem Bounding GNN VC dimension for new activation functions.
method Pfaffian function theory applied to GNNs with sigmoid and hyperbolic tangent activations.
result Bounds on GNN VC dimension for various architectures and graph properties.
Projective manifolds with specific bundles are isomorphic to simpler spaces.
problem Characterizing projective manifolds with tangent bundles containing strictly nef subsheaves.
method Analyzing the structure of the tangent bundle and using properties of strictly nef subsheaves.
result Projective manifolds with the described bundles are isomorphic to projective bundles over hyperbolic manifolds or projective spaces.
Combining M-algebra and hyperbolic involutory algebra extends exceptional tangent spaces to 11 dimensions.
problem Combining symmetries in M-theory to extend exceptional tangent spaces.
method Combining known results to show hyperbolic involutory algebra acts on M-algebra through brane-rotating symmetry.
result Extends the hierarchy of exceptional tangent spaces from n ≤ 7 to n = 11.
Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
The paper finds lower bounds for volumes of complex geometric structures.
problem Estimating the volume of complex geometric structures.
method Reduction to a counting problem in the unit tangent bundle, solved using exponential multiple mixing for the geodesic flow.
result First known lower bound for the volume of these manifolds in terms of curve length.
This paper is a continuation of the previous paper of the author[M]. We show that an affine deformation space of a hyperbolic surface of type (g,b) can be parametrized by Margulis invariants and affine twist parameters with a certain decomposition of the surface, which are associated with the Fenchel-Nielsen coordinate…
For an n-dimensional real hyperbolic manifold M, we calculate the Zariski tangent space of a character variety χ(π1(M),SL(n+1,R)),n>2 at Fuchisan loci to show that the tangent space consists of cubic forms. Furthermore we prove the Weil's local rigidity theorem for uniforml hyperbolic lattices using rea…
Partial coverings of hyperbolic surfaces equidistribute with geodesics.
problem Equidistribution of partial coverings defined from geodesics.
method Sequence of geodesics equidistributing in unit tangent bundle implies equidistribution of associated partial coverings.
result Partial coverings equidistribute with a sequence of geodesics.
The paper extends Descartes' circle theorem to n-flower configurations using hyperbolic geometry.
problem Extending Descartes' circle theorem to n-flower configurations.
method Spinorial description of horospheres in hyperbolic geometry.
result An explicit equation satisfied by the curvatures of n-flower configurations.
The paper examines when NTK theory applies to real finite-width neural networks.
problem Understanding when NTK theory accurately predicts the behavior of finite-width neural networks.
method Empirical study of fully-connected ReLU and sigmoid DNNs with various hyperparameters and depths.
result NTK theory does not always apply to sufficiently deep networks with exploding gradients, and the kernel changes significantly during training.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
New normalizing flows in hyperbolic space improve posterior modeling for hierarchical data.
problem Limited flexibility of existing normalizing flows in Euclidean space for hierarchical data.
method Elevated normalizing flows to hyperbolic spaces using coupling transforms and Wrapped Hyperboloid Coupling.
result Improved performance on density estimation and hierarchical graph data.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
New surfaces show horocyclic flow isn't always minimal.
problem Complex dynamics on infinite fineness surfaces.
method Construction of infinite hyperbolic surfaces.
result Horocyclic flow is not minimal on infinite fineness surfaces.
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Deep convolutional neural networks have led to breakthrough results in numerous practical machine learning tasks such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction …
For a compact riemannian manifold of negative curvature, the geodesic foliation of its unit tangent bundle is independent of the negatively curved metric, up to Holder bicontinuous homeomorphism. However, the riemannian metric defines a natural transverse measure to this foliation, the Liouville transverse measure, whi…
Study of harmonic maps with extreme Kerr-like singularities.
problem Analyzing harmonic maps with specific singularities.
method Asymptotic analysis of harmonic maps from 3D Euclidean space to hyperbolic plane.
result Existence and classification of tangent harmonic maps at extreme black hole horizons.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios…
We develop a new approach to learn the parameters of regression models with hidden variables. In a nutshell, we estimate the gradient of the regression function at a set of random points, and cluster the estimated gradients. The centers of the clusters are used as estimates for the parameters of hidden units. We justif…
Study volumes of hyperbolic 3-manifolds formed by curves on surfaces.
problem Understanding volumes of hyperbolic 3-manifolds formed by curves on surfaces.
method Study of stratified hyperbolic links in Seifert-fibered spaces and comparison of volumes to distances in pants graphs.
result Volume of PT(S)∖Γ^ is coarsely comparable to Weil-Petersson distance between strata in Teichmüller space. We consider the normalized Ricci flow evolving from an initial metric which is conformally compactifiable and asymptotically hyperbolic. We show that there is a unique evolving metric which remains in this class, and that the flow exists up to the time where the norm of the Riemann tensor diverges. Restricting to initi…
Defines horocyclic evolutes, parallels, and involutes of spacelike frontals in hyperbolic 2-space.
problem None explicitly stated; focuses on definitions and relations.
method Using enveloid theorem, defines horocyclic parallel and involute as normal envelopes of horocycles.
result Investigates relations among horocyclic evolutes, parallels, and involutes.
Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing m…
We show that the empirical risk minimization (ERM) problem for neural networks has no solution in general. Given a training set s1,…,sn∈Rp with corresponding responses t1,…,tn∈Rq, fitting a k-layer neural network νθ:Rp→Rq involves estimation of…
The study limits the cohomological dimension of certain affine manifolds with partially hyperbolic holonomy groups.
problem Understanding cohomological dimensions of affine manifolds with specific holonomy groups.
method Analyzing the tangent bundle structure and using coarse geometry techniques.
result The cohomological dimension is bounded by the dimension minus the index of the holonomy group.
New examples of real hypersurfaces found in complex hyperbolic quadrics.
problem Existence of specific types of real hypersurfaces in complex hyperbolic quadrics.
method Construction of a one-parameter family of homogeneous Hopf hypersurfaces.
result First known examples of real hypersurfaces with integrable maximal complex subbundle in irreducible Kahler manifolds.
A new framework for hyperbolic neural networks using the Klein model is introduced.
problem Previous works focused on Poincaré and hyperboloid models, neglecting the Klein model.
method Formulation of operations using the Klein model, study of the Klein linear layer, and comparison with Poincaré ball model.
result The Klein HNN performs similarly to the Poincaré ball model, offering a third option.
In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation …
Deep conditional generative models are developed to simultaneously learn the temporal dependencies of multiple sequences. The model is designed by introducing a three-way weight tensor to capture the multiplicative interactions between side information and sequences. The proposed model builds on the Temporal Sigmoid Be…
Explains exceptional surgeries connecting maps and knot orbifolds.
problem Exceptional surgeries connecting maps and knot orbifolds.
method Explains connections between the cat-bat map and figure-eight knot via hyperbolic orbispheres.
result Shows how the suspension of the cat-bat map is related to figure-eight knot orbifolds.