In this paper we study the consistency of an empirical minimum error entropy (MEE) algorithm in a regression setting. We introduce two types of consistency. The error entropy consistency, which requires the error entropy of the learned function to approximate the minimum error entropy, is shown to be always true if the…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “minimum error entropy”
Paper proposes deep neural networks for nonparametric regression from dependent data.
problem Nonparametric regression from strongly mixing observations.
method Minimum error entropy principle applied to deep neural networks.
result Deep neural networks achieve minimax optimal convergence rates for Gaussian errors.
We consider the minimum error entropy (MEE) criterion and an empirical risk minimization learning algorithm in a regression setting. A learning theory approach is presented for this MEE algorithm and explicit error bounds are provided in terms of the approximation ability and capacity of the involved hypothesis space w…
Proposes a method to solve deep neural networks' local minimum problem.
problem Local minimum problem in deep neural networks training.
method Transforms cross-entropy loss into risk-averse error criterion, adjusts RSI, and uses convexity region.
result Trained deep learning machine is expected to be inside a global minimum's attraction basin.
The minimum error entropy (MEE) criterion has been verified as a powerful approach for non-Gaussian signal processing and robust machine learning. However, the implementation of MEE on robust classification is rather a vacancy in the literature. The original MEE only focuses on minimizing the Renyi's quadratic entropy …
Comparing with traditional learning criteria, such as mean square error (MSE), the minimum error entropy (MEE) criterion is superior in nonlinear and non-Gaussian signal processing and machine learning. The argument of the logarithm in Renyis entropy estimator, called information potential (IP), is a popular MEE cost i…
We study the problem of identifying the causal relationship between two discrete random variables from observational data. We recently proposed a novel framework called entropic causality that works in a very general functional model but makes the assumption that the unobserved exogenous variable has small entropy in t…
While the channel capacity reflects a theoretical upper bound on the achievable information transmission rate in the limit of infinitely many bits, it does not characterise the information transfer of a given encoding routine with finitely many bits. In this note, we characterise the quality of a code (i. e. a given en…
Inference for normal and Monte Carlo distributions using minimum relative entropy.
problem Inference from partial information on expectations and covariances.
method Minimum relative entropy sub-manifolds, analytical formulas, Monte Carlo simulations.
result Improved numerical implementation for inference from partial information.
We consider the problem of identifying the causal direction between two discrete random variables using observational data. Unlike previous work, we keep the most general functional model but make an assumption on the unobserved exogenous variable: Inspired by Occam's razor, we assume that the exogenous variable is sim…
Paper compares two entropy concepts for finite presentation groups.
problem Comparing two entropy concepts for groups of finite presentation.
method Analyzes and contrasts minimum volume entropy for geometrically finite groups.
result Two entropy concepts coincide in dimension 1 but differ in others.
Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we argue that for classiffication tasks, mutual information is a more suitable informa…
Relative cluster entropy for power-law correlated sequencescond-mat.stat-mech
A measure called relative cluster entropy distinguishes between correlated and uncorrelated sequences.
problem Distinguishing between sequences with different correlation degrees.
method Minimum relative entropy principle applied to cluster partitions of power-law correlated sequences.
result Optimal Hurst exponents are selected for market price series, indicating non-markovianity.
New method uncovers zero entropy in dependent observations after finite samples.
problem Understanding uncertainty reduction in dependent observations.
method Minimum list entropy coupling, greedy algorithm.
result Zero entropy achieved with O(log(1/P_min)) samples for dependent observations.
Sample size determination for a data set is an important statistical process for analyzing the data to an optimum level of accuracy and using minimum computational work. The applications of this process are credible in every domain which deals with large data sets and high computational work. This study uses Bayesian a…
Sequence-to-sequence models, such as attention-based models in automatic speech recognition (ASR), are typically trained to optimize the cross-entropy criterion which corresponds to improving the log-likelihood of the data. However, system performance is usually measured in terms of word error rate (WER), not log-likel…
Minimum Entropy Density Method for the Time Series Analysisphysics.data-an
The entropy density is an intuitive and powerful concept to study the complicated nonlinear processes derived from physical systems. We develop the minimum entropy density method (MEDM) to detect the structure scale of a given time series, which is defined as the scale in which the uncertainty is minimized, hence the p…
Statistical mechanical analysis of sparse linear regression as a variable selection problemcond-mat.dis-nn
An algorithmic limit of compressed sensing or related variable-selection problems is analytically evaluated when a design matrix is given by an overcomplete random matrix. The replica method from statistical mechanics is employed to derive the result. The analysis is conducted through evaluation of the entropy, an expo…
Minimum Rényi Entropy Portfoliosq-fin.PM
Accounting for the non-normality of asset returns remains challenging in robust portfolio optimization. In this article, we tackle this problem by assessing the risk of the portfolio through the "amount of randomness" conveyed by its returns. We achieve this by using an objective function that relies on the exponential…
Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcs.LG
Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the art results for many problems. Nonetheless, Stochastic Gradient Descent typically cannot find the global minimum, thus its empirical effect…
Robust diffusion adaptive estimation algorithms based on the maximum correntropy criterion (MCC), including adaptation to combination MCC and combination to adaptation MCC, are developed to deal with the distributed estimation over network in impulsive (long-tailed) noise environments. The cost functions used in distri…
The entropy of minimal surfaces is minimized in hyperbolic manifolds.
problem Counting essential minimal surfaces in closed negatively curved manifolds.
method Defining minimal surface entropy and computing it for various spaces.
result Minimal surface entropy is minimized in hyperbolic manifolds.
Consider the problem of estimating the minimum entropy of pseudo-Anosov maps on a surface of genus with punctures. We determine the behaviour of this minimum number for a certain large subset of the plane, up to a multiplicative constant. In particular it has been shown that for fixed , this minimum …
Tree Index evaluates cluster quality by creating decision trees from data.
problem Evaluating the quality of cluster results from various techniques.
method Tree Index creates a decision tree from clustered data, combining entropy and depth of leaves.
result Tree Index discriminates between sensible and non-sensible clusters on brain dataset.
In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking,…
Many recent models of trade dynamics use the simple idea of wealth exchanges among economic agents in order to obtain a stable or equilibrium distribution of wealth among the agents. In particular, a plain analogy compares the wealth in a society with the energy in a physical system, and the trade between agents to the…
Motivated by Bonahon's result for hyperbolic surfaces, we construct an analogue of the Patterson-Sullivan-Bowen-Margulis map from the Culler-Vogtmann outer space into the space of projectivized geodesic currents on a free group. We prove that this map is a topological embedding. We also prove that for every $…
The study analyzes robustness of estimators in linear models with adversarial errors.
problem Analyzing robustness of estimators in linear models with adversarial errors.
method Develops a general theory for minimum norm interpolating estimators and RERM in linear models without conditions on errors.
result Quantitative bound for the prediction error relating it to Rademacher complexity, norm of minimum norm interpolator of errors, and subdifferential size.
Conservation laws improve diffusion model training by optimizing likelihood.
problem Training diffusion models with denoising objectives.
method Developed conservation laws based on GEXIT functions for memoryless noise processes.
result Unified characterization of diffusion model likelihood, reducing training to learning marginal posteriors.
Explicit construction of the minimum error variance estimator for stochastic LTI state-space systemsmath.OC
Optimal estimator derived for partially observable LTI systems.
problem Optimal estimator for partially observable LTI systems.
method State-space representation for derivation of optimal estimator.
result Derivation of minimum error variance estimator for partially observable LTI systems.
Entropy minimization has been widely used in unsupervised domain adaptation (UDA). However, existing works reveal that entropy minimization only may result into collapsed trivial solutions. In this paper, we propose to avoid trivial solutions by further introducing diversity maximization. In order to achieve the possib…
Study tightens bounds for interpolating noisy data using minimum l1-norm.
problem Predicting noisy data with minimum l1-norm interpolation.
method Provided matching upper and lower bounds for prediction error.
result Tight consistency up to negligible terms for .
Time-series Random Process Complexity Ranking Using a Bound on Conditional Differential Entropyeess.SP
This paper improves entropy bounds for ranking time-series complexity.
problem Ranking the complexity of time series processes.
method Building on information theoretic bounds, the paper improves the upper bound of conditional differential entropy using Hadamard's inequality and covariance matrix properties.
result The improved bounds can be used to rank the complexity of time series processes.
A novel framework infers causal direction from symbolic sequences using pattern entropy.
problem Challenges in discovering causal direction from temporal symbolic data.
method Dictionary Based Pattern Entropy () framework integrating AIT and Shannon Information Theory.
result Minimizing pattern level uncertainty yields a robust framework for causal discovery.
The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networksstat.ML
We study the generalization properties of minimum-norm solutions for three over-parametrized machine learning models including the random feature model, the two-layer neural network model and the residual network model. We proved that for all three models, the generalization error for the minimum-norm solution is compa…
We study the problem of discovering the simplest latent variable that can make two observed discrete variables conditionally independent. The minimum entropy required for such a latent is known as common entropy in information theory. We extend this notion to Renyi common entropy by minimizing the Renyi entropy of the …
In this work, three lattice-free (LF) discriminative training criteria for purely sequence-trained neural network acoustic models are compared on LVCSR tasks, namely maximum mutual information (MMI), boosted maximum mutual information (bMMI) and state-level minimum Bayes risk (sMBR). We demonstrate that, analogous to L…
This article proposes a method to quantify the structure of a bipartite graph using a network entropy per link. The network entropy of a bipartite graph with random links is calculated both numerically and theoretically. As an application of the proposed method to analyze collective behavior, the affairs in which parti…
Theoretical analysis of entropy approximation for Gaussian mixtures.
problem Lack of theoretical guarantees for entropy approximation of Gaussian mixtures.
method Theoretical analysis of the error between true and approximate entropy.
result The error converges to zero as the ratios of means to variances tend to infinity, providing a guarantee for high-dimensional problems.
Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errorscs.LG
A new method reduces compounding errors in model-based reinforcement learning.
problem Compounding errors in long horizon predictions from model-based reinforcement learning.
method Maximum Entropy Model Rollouts (MEMR) with non-uniform sampling and prioritized experience replay.
result Significantly reduces computation requirements compared to other model-based methods.
Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priorsstat.ML
The study improves representation learning bounds using data-dependent Gaussian mixtures.
problem Improving generalization in representation learning.
method Established bounds using relative entropy and MDL of latent variables.
result The approach significantly improves generalization over existing methods.
Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcs.LG
Entropy-regularized NPG converges linearly with linear function approximation.
problem Analyzing convergence of entropy-regularized NPG with function approximation.
method Established finite-time convergence analyses with entropy regularization and linear function approximation.
result Entropy-regularized NPG achieves linear convergence up to a function approximation error.
New proof shows how to identify DAGs with weakly increasing errors.
problem Identifying the true DAG in models with weakly increasing error variances.
method Minimum-trace DAG method and hill climbing algorithm with R2R neighborhood.
result Hill climbing algorithm without strict local optima under weakly increasing error variances.
Inflating the minimum norm interpolator improves linear regression generalization error.
problem Highly anisotropic covariances and diverging in linear regression.
method Inflating the minimum norm interpolator by a constant greater than one.
result Inflating the minimum norm interpolator improves generalization error.
We propose a general framework for neural network compression that is motivated by the Minimum Description Length (MDL) principle. For that we first derive an expression for the entropy of a neural network, which measures its complexity explicitly in terms of its bit-size. Then, we formalize the problem of neural netwo…
Minimal surface entropy and applications of Ricci flow on finite-volume hyperbolic 3-manifoldsmath.DG
The paper studies minimal surface entropy on hyperbolic 3-manifolds and compares it to the hyperbolic case.
problem Minimal surface entropy on hyperbolic 3-manifolds and its comparison to the hyperbolic case.
method Analysis of Ricci flow convergence and comparison of metrics with sectional and scalar curvature constraints.
result The entropy is maximized at the hyperbolic metric under certain curvature conditions.
AR-DAE approximates entropy gradient for machine learning models.
problem Intractable computation of entropy gradient for continuous distributions.
method Amortized residual denoising autoencoder (AR-DAE) to approximate entropy gradient.
result AR-DAE provides an unbiased gradient approximation for entropy.
State-level minimum Bayes risk (sMBR) training has become the de facto standard for sequence-level training of speech recognition acoustic models. It has an elegant formulation using the expectation semiring, and gives large improvements in word error rate (WER) over models trained solely using cross-entropy (CE) or co…