The paper simplifies the Fisher information matrix for random deep networks, speeding up learning.
problem Learning deep neural networks efficiently with large parameter spaces.
method Statistical neurodynamical method to reveal Fisher information properties, proving unit-wise block diagonal structure and explicit inverse.
result Explicit natural gradient formula without matrix inversion, speeding up learning.
A new measure of model complexity based on Fisher Information.
problem Model complexity measurement in statistical models.
method Effective dimension defined by the number of cubes needed to cover the model space.
result The effective dimension is scale-dependent and measures model complexity.
Two Fisher information matrix estimators are analyzed for neural networks, focusing on their variances and trade-offs.
problem Estimating the Fisher information matrix in neural networks due to its high computational cost.
method Examined two popular diagonal Fisher information matrix estimators and their variances in neural networks for regression and classification.
result The variances of the estimators depend on the non-linearity with respect to different parameter groups and should not be neglected.
Improved SGLD for neural networks using Fisher matrix approximations.
problem Overfitting in neural networks.
method Natural Langevin dynamics with Fisher matrix preconditioning.
result Fisher matrix preconditioning improves SGLD for large neural networks.
NGD improves multivariate Gaussian inference by optimizing Fisher information.
problem Efficiently optimizing multivariate Gaussian models.
method Natural Gradient Descent applied to multivariate Gaussian parameters.
result NGD updates are more efficient for symmetric covariance matrices.
The Fisher-Rao geometry is applied to elliptical distributions for optimization and classification.
problem Optimizing and classifying covariance matrices using geometric tools.
method Riemannian optimization and intrinsic Cramér-Rao bounds.
result Geometric tools enhance covariance matrix estimation and classification.
Disputes the empirical Fisher approximation for natural gradient descent.
problem The empirical Fisher approximation fails to capture second-order information in general.
method Comparison of empirical Fisher and Fisher information matrices.
result The empirical Fisher does not generally approximate the Fisher or Hessian.
Unified framework for two types of matrix Lie group preconditioners in SGD.
problem Improving optimization efficiency in machine learning models.
method Unified framework for Newton and Fisher type preconditioners on matrix Lie groups.
result Efficient estimation of preconditioners on matrix Lie groups.
We propose a modified χβ-divergence, give some of its properties, and show that this leads to the definition of a generalized Fisher information. We give generalized Cramér-Rao inequalities, involving this Fisher information, an extension of the Fisher information matrix, and arbitrary norms and power of the estimat…
We introduce RSE to measure robustness in estimation problems.
problem Estimating statistical models from observed data.
method Developed theory for spectral functions of measures to compute RSE.
result RSE reveals a reciprocal relationship with problem complexity.
New methods improve Fisher Matrix approximations for neural networks at low cost.
problem High cost of solving Fisher Information Matrix (FIM) in neural networks.
method Direct minimization via Kronecker product singular value decomposition.
result Improved approximations to FIM provide more accurate and faster optimization.
Early training phase affects deep neural network optimization and generalization.
problem The choice of learning rate influences generalization in deep learning models.
method Showed that SGD implicitly penalizes the trace of the Fisher Information Matrix (FIM) from the start of training, and explicitly penalizing the trace of FIM improves generalization.
result Catastrophic Fisher explosion (large trace of FIM early in training) is linked to poor generalization.
Paper identifies key function spaces for ReLU networks based on Fisher information.
problem Understanding the structure of Fisher information matrices in ReLU networks.
method Spectral decomposition of Fisher information matrices, focusing on the first three eigenspaces.
result The first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, corresponding to spherical harmonic functions of order ≤2.
A family of probability distributions parametrized by an open domain Λ in Rn defines the Fisher information matrix on this domain which is positive semi-definite. In information geometry the standard assumption has been that the Fisher information matrix tensor is positive definite defining in this way a Riemannia…
Optimal preconditioning improves Langevin sampling efficiency.
problem Improving sampling efficiency in high-dimensional target distributions.
method Optimal preconditioning using Fisher information, applied to MALA.
result Adaptive MCMC scheme significantly outperforms other methods.
Paper improves matrix-valued data classification using nonparametric LDA.
problem Classification of matrix-valued data in neuroimaging and signal processing.
method Nonparametric LDA based on NPMLE for vectorized and scaled matrices.
result Improves classification performance across various data structures.
New loss function reduces adversarial examples by controlling Fisher information matrix eigenvalues.
problem Defending against adversarial attacks in neural networks.
method Adding a term to the loss function representing the trace of the Fisher information matrix.
result Effective and robust defensive capability, reducing adversarial example fooling ratio.
Exploring how noise and curvature affect optimization and generalization.
problem The interaction between noise and curvature in optimization and generalization.
method Analyzing the speed of minimizing expected loss with stochastic methods, distinguishing between Fisher, Hessian, and gradient covariance matrices.
result Clarifying the role of curvature and noise in estimating the generalization gap.
This paper solves the intractability barrier in non-parametric information geometry by introducing a novel framework.
problem The intractability barrier in non-parametric information geometry due to the Fisher-Rao metric being a functional.
method Introducing an Orthogonal Decomposition of the Tangent Space and deriving the Covariate Fisher Information Matrix (cFIM).
result Established a rigorous foundation for the G-entropy and provided fundamental limits of variance for semi-parametric estimators.
A new iterative K-FAC algorithm reduces training time and memory usage.
problem Training deep learning models efficiently.
method Uses conjugate gradient to approximate Fisher information matrix without generating the matrix or factors.
result Time and memory complexity of iterative CG-FAC is less than standard K-FAC.
Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2…
New method for natural policy gradients converges linearly.
problem Improving natural policy gradient methods for better convergence.
method Fisher-Rao gradient flow applied to state-action distributions.
result Linear convergence rate with geometry-dependent factor.
Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function. Unfortunately, the exact natural gradient is impractical to compute for large models, and most approximations either require an expensive it…
Paper presents a rank-1 approximation method for natural policy gradients in deep RL.
problem Computing natural gradients requires inverting the Fisher Information Matrix, which is computationally expensive.
method Develops a rank-1 approximation to the inverse Fisher Information Matrix for efficient natural policy optimization.
result The rank-1 approximation converges faster and has similar sample complexity to stochastic policy gradient methods.
A new method improves uncertainty estimation in deep learning, especially for hard-to-label samples.
problem Improving uncertainty estimation for hard-to-label samples in deep learning.
method Introduces Fisher Information Matrix (FIM) to dynamically reweight objective loss terms.
result Consistently outperforms traditional evidential neural networks in uncertainty estimation tasks.
New method learns quantum states using neural networks, revealing hidden dynamics.
problem High-precision ground state estimation of quantum many-body problems.
method Stochastic reconfiguration method with neural network Ansatz states.
result Learning landscape modes with least entanglement have largest eigenvalues, suggesting correlations are encoded in large flat valleys.
NG+ method improves deep learning efficiency and accuracy.
problem Efficiency and accuracy in deep learning models.
method Proposes NG+ method using matrix-product natural gradient approach.
result Established global convergence and provided regret bound.
The study uses Fisher information to estimate volatility uncertainty in Heston model.
problem Estimating volatility uncertainty in financial models like Heston.
method Fit likelihood function on VIX options, compute Fisher information matrices from Heston model Greeks.
result Option prices can reliably estimate volatility when it's large, but become unreliable below a critical value.
We analyze the variance of Fisher information estimators in deep learning models.
problem Understanding the variance of Fisher information in deep learning models.
method Investigated two unbiased and consistent estimators of Fisher information matrix.
result The variance of estimators is influenced by the model's parametric structure.
Estimates metric tensor on neuromanifolds using Fisher information and random methods.
problem Computing the metric tensor on high-dimensional neuromanifolds efficiently and accurately.
method Deterministic bounds and unbiased random estimators based on Hutchinson's trace method.
result An efficient random estimator with bounded standard deviation.
Hierarchical parametric models consisting of observable and latent variables are widely used for unsupervised learning tasks. For example, a mixture model is a representative hierarchical model for clustering. From the statistical point of view, the models can be regular or singular due to the distribution of data. In …
The paper analyzes how quantization affects the Fisher Information Matrix's dominant eigenvalue.
problem The impact of quantization on the Fisher Information Matrix's dominant eigenvalue.
method The study examines spectral perturbation of the empirical Fisher Information Matrix under in-distribution input and quantized parameter perturbations.
result A bound on the eigenvalue under quantization noise, showing it strictly exceeds the unperturbed value at leading order.
Kernel networks' stability edge linked to Fisher Information singularity.
problem Understanding the stability edge in high-capacity kernel Hopfield networks.
method Statistical manifold analysis and Riemannian geometry.
result The Ridge of Optimization corresponds to the Edge of Stability, revealing a dual equilibrium.
The paper uses Fisher information to explain and defend against adversarial attacks.
problem Vulnerability of deep learning models to adversarial attacks.
method Proposes OSSA for adversarial attack and eigenvalues for detection.
result Fisher information reveals model vulnerability through eigenvalues.
Deep neural networks reveal a low-dimensional manifold structure in data.
problem Understanding the structure of data for better model performance.
method Model-centric analysis of the data manifold using the local data matrix and Fisher information matrix.
result The dataset lies on a data leaf with a dimension bounded by the number of labels.
Parallel unlearning framework for inherited models reduces computational overhead.
problem Challenges in unlearning complex, evolving model networks.
method Chronologically Directed Acyclic Graph (DAG) and Fisher Inheritance Unlearning (FIUn) method.
result Significant reduction in computational overhead and efficient parallel unlearning.
New tensor framework connects Fisher information, hypergraphs, and multi-observable correlations.
problem Missing structure in pairwise Fisher graphs for multi-observable radiation patterns.
method Higher-order Fisher tensors and natural exponential-family coordinates.
result Exact triality of Fisher tensors, cumulants, and hypergraphs.
It is well known that in a supervised classification setting when the number of features is smaller than the number of observations, Fisher's linear discriminant rule is asymptotically Bayes. However, there are numerous modern applications where classification is needed in the high-dimensional setting. Naive implementa…
A new iterative algorithm improves RFDA for high-dimensional data.
problem High-dimensional data challenges conventional FDA and RFDA.
method Iterative sketching-based algorithm with accuracy guarantees.
result Accurate approximations can be achieved with smaller sample sizes.
This paper analyzes Barlow Twins' representation efficiency using information-geometric methods.
problem Understanding and comparing the efficiency of self-supervised learning methods.
method Introduces an information-geometric framework to quantify representation efficiency and applies it to Barlow Twins.
result Proves that Barlow Twins achieves optimal representation efficiency (η=1).
Study active learning for multi-level user preferences in recommendation systems.
problem Efficiently learning user preferences through active querying in recommendation systems.
method Proposes a theoretically optimal active learning strategy based on Fisher information matrix for collective matrix factorization.
result Demonstrates strong improvements over active learning methods in personalized, cold-start, and noisy data settings.
Paper proposes efficient method to calculate Fisher-Bingham distribution normalizing constant.
problem Efficiently calculating the normalizing constant of Fisher-Bingham distributions.
method Numerical integration with continuous Euler transform to Fourier-type integral representation.
result The method is fast and accurate, applicable to high-dimensional distributions.
Paper establishes limits for accurately estimating low-rank matrices from noisy, non-linear data.
problem Estimating low-rank matrices from noisy, non-linear observations.
method Proves strong universality result with equivalent Gaussian model and effective prior parameters.
result Signal-to-noise ratio requirement grows as $N^{ rac 12 (1-1/k_F)}$ for accurate reconstruction.
The space of probability densities is an infinite-dimensional Riemannian manifold, with Riemannian metrics in two flavors: Wasserstein and Fisher--Rao. The former is pivotal in optimal mass transport (OMT), whereas the latter occurs in information geometry---the differential geometric approach to statistics. The Rieman…
Study reveals pathological eigenvalue spectra in FIM and its variants of DNNs.
problem Understanding sharp local shapes in DNN loss landscapes.
method Analysis of FIM and its variants in regression and classification DNNs.
result Pathological eigenvalue spectra appear in FIM and its variants, indicating sharp local shapes in specific directions.
The paper sets bounds on how much regret is unavoidable in adaptive LQR with unknown B-matrix.
problem Understanding the limits of adaptive LQR with unknown B-matrix.
method Local asymptotic minimax regret lower bounds using van Trees' inequality and Bellman error representation.
result Logarithmic regret is impossible if the parametrization induces an uninformative optimal policy.
The paper analyzes convergence of Langevin dynamics with time-dependent metrics.
problem Analyzing convergence of Langevin dynamics with time-dependent metrics.
method Formulated a modified gradient flow of the Kullback-Leibler divergence, selected a time-dependent relative Fisher information functional, and developed a time-dependent Hessian matrix condition.
result Proved convergence conditions for various Langevin dynamics.
Develops information geometry for Lévy processes in finance.
problem Understanding the statistical properties of Lévy processes for financial modeling.
method Deriving α-divergences from Lévy triplets, identifying Fisher information matrix and α-connection. result Identifies statistical implications and differential-geometric structures of Lévy processes.