New method approximates diffusion process posteriors using moment functions.
problem Approximating posteriors of stochastic differential equations.
method Constructs variational process as controlled prior, approximates posterior with moment functions, uses natural gradient descent.
result Richer variational approximations for state-dependent diffusion terms.
In this communication, we describe some interrelations between generalized q-entropies and a generalized version of Fisher information. In information theory, the de Bruijn identity links the Fisher information and the derivative of the entropy. We show that this identity can be extended to generalized versions of en…
Market strategies minimize Fisher information to minimize risk.
problem Applying minimum Fisher information principle to market dynamics.
method Analytical extension to quantum harmonic oscillator eigenstates and Gibbs distribution.
result Minimizing Fisher information reduces information and risk.
We propose a modified χβ-divergence, give some of its properties, and show that this leads to the definition of a generalized Fisher information. We give generalized Cramér-Rao inequalities, involving this Fisher information, an extension of the Fisher information matrix, and arbitrary norms and power of the estimat…
Blog post discusses various implementations of Fisher Information for EWC in continual learning.
problem Improving Elastic Weight Consolidation (EWC) results by optimizing Fisher Information computation.
method Empirically compares different implementations of Fisher Information for EWC.
result Many reported EWC results can be improved by changing Fisher Information computation methods.
This paper shows any Kähler metric can be a Fisher information metric.
problem Establishing a new characterization of Kähler and coKähler manifolds.
method Statistical approach using Fisher information and exponential families.
result Any Kähler metric is a Fisher information metric.
In information theory, Fisher information and Shannon information (entropy) are respectively used to quantify the uncertainty associated with the distribution modeling and the uncertainty in specifying the outcome of given variables. These two quantities are complementary and are jointly applied to information behavior…
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between ap…
TopoFisher learns topological summaries by maximizing Fisher information, improving parameter efficiency and inference quality.
problem Simulation-based inference misses key information in low-order statistics, especially for non-Gaussian fields.
method TopoFisher uses a differentiable persistent-homology pipeline that learns topological summaries by maximizing local Gaussian Fisher information.
result TopoFisher recovers much of the available information and outperforms fixed topological vectorizations in weak gravitational lensing.
A deep neural network is a hierarchical nonlinear model transforming input signals to output signals. Its input-output relation is considered to be stochastic, being described for a given input by a parameterized conditional probability distribution of outputs. The space of parameters consisting of weights and biases i…
Paper identifies key function spaces for ReLU networks based on Fisher information.
problem Understanding the structure of Fisher information matrices in ReLU networks.
method Spectral decomposition of Fisher information matrices, focusing on the first three eigenspaces.
result The first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, corresponding to spherical harmonic functions of order ≤2.
The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.
problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.
method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.
result Obtained an approximation formula for functions represented by 2-layer neural networks.
Paper introduces a new gradient estimator for SNNs.
problem High variance in score function gradient estimator impedes SNNs training.
method Developed a differentiable point process to derive path-wise gradient estimator.
result Demonstrated effectiveness of path-wise gradient estimator through simulations.
Study improves sampling from non-log-concave distributions using Fisher information.
problem Sampling from non-log-concave distributions with high Fisher information guarantees.
method Proximal sampler with RGO implementation, leveraging log-concave sampling results.
result Improved complexity guarantee in relative Fisher information for non-log-concave sampling.
We present a novel synthesis of Fisher information and asset pricing theory that yields a practical method for reconstructing the probability density implicit in security prices. The Fisher information approach to these inverse problems transforms the search for a probability density into the solution of a differential…
Machine learning approximates phase transitions using Fisher information.
problem Understanding phase transitions from data using machine learning.
method Information geometry and Fisher information.
result Machine learning indicators approximate the square root of Fisher information.
Two Fisher information matrix estimators are analyzed for neural networks, focusing on their variances and trade-offs.
problem Estimating the Fisher information matrix in neural networks due to its high computational cost.
method Examined two popular diagonal Fisher information matrix estimators and their variances in neural networks for regression and classification.
result The variances of the estimators depend on the non-linearity with respect to different parameter groups and should not be neglected.
The paper explores geometry of probability measures and barycenter maps.
problem Understanding the space of probability measures and their barycenter.
method Information geometry, Fisher metric, dualistic structures, divergences, geodesics.
result Recent developments in the geometry of probability measures and barycenter.
We show a general relation between the spatially disjoint product of probability density functions and the sum of their Fisher information metric tensors. We then utilise this result to give a method for constructing the probability density functions for an arbitrary Riemannian Fisher information metric tensor. We note…
New tensor framework connects Fisher information, hypergraphs, and multi-observable correlations.
problem Missing structure in pairwise Fisher graphs for multi-observable radiation patterns.
method Higher-order Fisher tensors and natural exponential-family coordinates.
result Exact triality of Fisher tensors, cumulants, and hypergraphs.
Improved mean estimation for symmetric distributions with finite-sample guarantees.
problem Estimating the mean of a symmetric distribution from samples.
method Using Fisher information rate for finite-sample guarantees.
result Finite-sample convergence close to subgaussian with variance 1/(n * I_r), where I_r is r-smoothed Fisher information.
Paper discusses the Fisher metric and differentiability in statistical models.
problem Understanding the relationship between Fisher metric and differentiability in statistical models.
method Comparison of different concepts and models in Information Geometry, mathematical statistics, and measure theory.
result Discussion of various models and their differentiability properties.
Paper improves Fisher information estimation methods.
problem Estimating Fisher information for location parameters.
method Revisits and improves Bhattacharya estimator, introduces clipped estimator.
result Clipped estimator shows superior convergence rates in Gaussian noise.
Paper proposes a method to verify PINN fidelity using Fisher information from dynamical systems.
problem Quantifying PINN fidelity beyond simple trajectory prediction.
method Employing Fisher information for differentiable dynamical systems to compare PINN's learned equations with analytical models.
result PINN fidelity is verified by matching Fisher information landscapes of learned equations and analytical models.
Inequalities linking entropy, Fisher info, Stein discrepancy, and Wasserstein distance on Riemannian manifolds.
problem Linking entropy, Fisher info, Stein discrepancy, and Wasserstein distance on Riemannian manifolds.
method Deriving inequalities linking these measures on Riemannian manifolds.
result Strengthening and extending existing inequalities to Riemannian manifolds.
New method detects and clusters market regimes in multidimensional data.
problem Detecting and clustering market regimes in complex data structures.
method Non-parametric online market regime detection and clustering using path-wise two-sample tests and maximum mean discrepancy.
result Successfully detected and clustered market regimes in various data structures.
Data processing inequalities link Fisher information to local differential privacy constraints.
problem Understanding how Fisher information scales with local differential privacy constraints.
method Developed data processing inequalities for Fisher information under local differential privacy.
result Implications for private estimation with optimal bounds and error rates.
Cosine schedule is optimal for discrete diffusion models.
problem Choosing the best discretization schedule for diffusion models.
method Optimized using Fisher-Rao geometry.
result Cosine schedule is Fisher-Rao optimal.
FedFisher improves one-shot FL by using Fisher information.
problem Reducing communication rounds in federated learning.
method Bayesian perspective, Fisher information matrices, diagonal Fisher, K-FAC approximation.
result FedFisher achieves vanishingly small error in two-layer neural networks.
We analyze the variance of Fisher information estimators in deep learning models.
problem Understanding the variance of Fisher information in deep learning models.
method Investigated two unbiased and consistent estimators of Fisher information matrix.
result The variance of estimators is influenced by the model's parametric structure.
The paper sets lower bounds for sampling non-log-concave distributions using Fisher information.
problem Understanding the complexity of sampling non-log-concave distributions.
method Proves two lower bounds using Fisher information in the context of sampling.
result Lower bounds on the complexity of sampling non-log-concave distributions, ruling out high-accuracy algorithms.
The probability distribution function (PDF) for prices on financial markets is derived by extremization of Fisher information. It is shown how on that basis the quantum-like description for financial markets arises and different financial market models are mapped by quantum mechanical ones.
We examine Generative Adversarial Networks (GANs) through the lens of deep Energy Based Models (EBMs), with the goal of exploiting the density model that follows from this formulation. In contrast to a traditional view where the discriminator learns a constant function when reaching convergence, here we show that it ca…
Estimates metric tensor on neuromanifolds using Fisher information and random methods.
problem Computing the metric tensor on high-dimensional neuromanifolds efficiently and accurately.
method Deterministic bounds and unbiased random estimators based on Hutchinson's trace method.
result An efficient random estimator with bounded standard deviation.
The Fisher-Rao geometry is applied to elliptical distributions for optimization and classification.
problem Optimizing and classifying covariance matrices using geometric tools.
method Riemannian optimization and intrinsic Cramér-Rao bounds.
result Geometric tools enhance covariance matrix estimation and classification.
SQFA learns features maximizing Fisher-Rao distance for better classification.
problem Improving classification accuracy through feature learning.
method SQFA learns linear features maximizing Fisher-Rao distance between class-conditional distributions.
result SQFA-H features achieve the best classification accuracy.
Paper adds Fisher Information to mean field optimization for faster convergence.
problem Mean field optimization in neural networks training.
method Developed energy-dissipation method and gradient flow on probability space.
result Marginal distributions converge exponentially to minimizer.
Concrete distribution properties examined on simplex.
problem Properties of Concrete distribution on simplex.
method Reflection and location-scale transformation of uniform distribution; explicit parameterization to Poincaré half-space.
result Fisher information and information metric are hyperbolic space; Fisher-Rao geodesic distance computed.
New method isolates epistemic uncertainty in diffusion models, improving plausibility scores.
problem Uncertainty quantification in diffusion models, especially epistemic uncertainty.
method Fisher information based approach using FLARE (Fisher-Laplace Randomized Estimator).
result FLARE improves uncertainty estimation in synthetic time-series generation tasks.
This paper solves the intractability barrier in non-parametric information geometry by introducing a novel framework.
problem The intractability barrier in non-parametric information geometry due to the Fisher-Rao metric being a functional.
method Introducing an Orthogonal Decomposition of the Tangent Space and deriving the Covariate Fisher Information Matrix (cFIM).
result Established a rigorous foundation for the G-entropy and provided fundamental limits of variance for semi-parametric estimators.
PWSHAP provides targeted explanations for complex models.
problem Inability of black-box models to explain targeted effects in sensitive domains.
method Augments model with DAG, uses Shapley values for causal pathway identification.
result Establishes error bounds and demonstrates resolution, interpretability, and locality.
This study explains why approximate NGD works well in wide neural networks.
problem Understanding why NGD with approximate Fisher information converges fast in wide neural networks.
method Analyzing asymptotic training dynamics in function space via the neural tangent kernel.
result NGD with approximate Fisher information achieves the same fast convergence as exact NGD under specific conditions.
The cost of belief changes with precision and is a hyperbolic geometry.
problem The cost of belief changes with precision and is a hyperbolic geometry.
method The cost of belief changes with precision and is a hyperbolic geometry.
result The cost of belief changes with precision and is a hyperbolic geometry.
FisherNet extends Autoencoder using Fisher information for better data reconstruction.
problem Data reconstruction accuracy and model scalability in high-dimensional latent spaces.
method Introduces FisherNet architecture that uses Fisher information to quantify and account for latent space uncertainty.
result FisherNet produces more accurate reconstructions and scales better with latent space dimensions compared to VAE.
Kernel networks' stability edge linked to Fisher Information singularity.
problem Understanding the stability edge in high-capacity kernel Hopfield networks.
method Statistical manifold analysis and Riemannian geometry.
result The Ridge of Optimization corresponds to the Edge of Stability, revealing a dual equilibrium.
FADE adapts machine learning models to evolving data efficiently.
problem Sequential covariate shift in dynamic environments.
method FADE uses Fisher information geometry for robust learning under SCS.
result FADE achieves up to 19% higher accuracy under severe shifts.
FIRE method improves model performance in federated learning by penalizing fragmentation-induced covariate shifts.
problem Performance degradation in federated learning due to data fragmentation and covariate shift.
method FIRE method accumulates fragmentation-induced covariate shift divergences via approximate Fisher information and uses it as a per-fragment loss penalty.
result FIRE outperforms importance weighting and federated learning benchmarks by up to 5.3% on shifted validation sets.
Many deep learning models are vulnerable to the adversarial attack, i.e., imperceptible but intentionally-designed perturbations to the input can cause incorrect output of the networks. In this paper, using information geometry, we provide a reasonable explanation for the vulnerability of deep learning models. By consi…