A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Minimum Description Length prevents overfitting in noisy data.
Study shows LLC correlates with neural network compressibility.
Time-invariant linear dynamical system arises in many real-world applications,and its usefulness is widely acknowledged. A practical limitation with this model is that its latent dimension that has a large impact on the model capability needs to be manually specified. It can be demonstrated that a lower-order model cla…
Method estimates dataset utility via minimal program length proxy.
PCA (Principal Component Analysis) and its variants areubiquitous techniques for matrix dimension reduction and reduced-dimensionlatent-factor extraction. One significant challenge in using PCA, is thechoice of the number of principal components. The information-theoreticMDL (Minimum Description Length) principle gives…
Neural networks generalize on simple data generated by a programming language.
The Fisher information approximation (FIA) is an implementation of the minimum description length principle for model selection. Unlike information criteria such as AIC or BIC, it has the advantage of taking the functional form of a model into account. Unfortunately, FIA can be misleading in finite samples, resulting i…
New method improves bivariate causal discovery by accurately estimating cause variable complexity.
We tackle the problem of penalty selection of regularization on the basis of the minimum description length (MDL) principle. In particular, we consider that the design space of the penalty function is high-dimensional. In this situation, the luckiness-normalized-maximum-likelihood(LNML)-minimization approach is favorab…
This paper introduces a new method for model selection and more generally hyperparameter selection in machine learning. Minimum description length (MDL) is an established method for model selection, which is however not directly aimed at minimizing generalization error, which is often the primary goal in machine learni…
Paper establishes generalization bounds for representation learning using Minimum Description Length.
Kernel networks' stability edge linked to Fisher Information singularity.
A new method avoids overfitting in network reconstruction by using the minimum description length principle.
We analyze differences between two information-theoretically motivated approaches to statistical inference and model selection: the Minimum Description Length (MDL) principle, and the Minimum Message Length (MML) principle. Based on this analysis, we present two revised versions of MML: a pointwise estimator which give…
Fast, fully-automated histograms for large data sets.
New architectures improve KANs, making them more interpretable and accurate.
Robust low-rank matrix estimation is a topic of increasing interest, with promising applications in a variety of fields, from computer vision to data mining and recommender systems. Recent theoretical results establish the ability of such data models to recover the true underlying low-rank matrix when a large portion o…
This is an up-to-date introduction to and overview of the Minimum Description Length (MDL) Principle, a theory of inductive inference that can be applied to general problems in statistics, machine learning and pattern recognition. While MDL was originally based on data compression ideas, this introduction can be read w…
The paper explores how smaller data sets can lead to better model selection decisions.
Bayesian networks are convenient graphical expressions for high dimensional probability distributions representing complex relationships between a large number of random variables. They have been employed extensively in areas such as bioinformatics, artificial intelligence, diagnosis, and risk management. The recovery …
S2KAN integrates symbolic primitives into neural network activations for improved interpretability.
Non-negative matrix factorization (NMF) is a dimensionality reduction technique which tends to produce a sparse representation of data. Commonly, the error between the actual and recreated matrices is used as an objective function, but this method may not produce the type of representation we desire as it allows for th…
A new complexity measure MDL-COMP for overparameterized models improves generalization performance.
The Minimum Description Length (MDL) principle states that the optimal model for a given data set is that which compresses it best. Due to practial limitations the model can be restricted to a class such as linear regression models, which we address in this study. As in other formulations such as the LASSO and forward …
Study improves neural network performance in sequential learning for image classification.
New model explains how concepts grow based on experience.
This study explores how feature graphs enhance GNNs' performance in modeling interactions.
Score based learning (SBL) is a promising approach for learning Bayesian networks in the discrete domain. However, when employing SBL in the continuous domain, one is either forced to move the problem to the discrete domain or use metrics such as BIC/AIC, and these approaches are often lacking. Discretization can have …
LMs perform poorly in true few-shot learning without held-out examples.
Bayesian methods detect clusters in noisy data more reliably.
Tree Index evaluates cluster quality by creating decision trees from data.
Why do deep neural networks (DNNs) benefit from very high dimensional parameter spaces? Their huge parameter complexities vs stunning performance in practice is all the more intriguing and not explainable using the standard theory of model selection for regular models. In this work, we propose a geometrically flavored …
New methods evaluate data representations by complexity of low-loss predictor learning.
The ropelength problem asks for the minimum-length configuration of a knotted diameter-one tube embedded in Euclidean three-space. The core curve of such a tube is called a tight knot, and its length is a knot invariant measuring complexity. In terms of the core curve, the thickness constraint has two parts: an upper b…
This paper introduces a new notion of dimensionality of probabilistic models from an information-theoretic view point. We call it the "descriptive dimension"(Ddim). We show that Ddim coincides with the number of independent parameters for the parametric class, and can further be extended to real-valued dimensionality w…
Knots are commonly found in molecular chains such as DNA and proteins, and they have been considered to be useful models for structural analysis of these molecules. One interested quantity is the minimum number of monomers necessary to realize a molecular knot. The minimum lattice length $\mbox{Len}(K)$ of a knot i…
We analyze MDL for binary classification, quantifying overfitting and underfitting.
The minimum description length (MDL) principle in supervised learning is studied. One of the most important theories for the MDL principle is Barron and Cover's theory (BC theory), which gives a mathematical justification of the MDL principle. The original BC theory, however, can be applied to supervised learning only …
Investigates the long-only minimum variance portfolio in factor models.
We show that the minimum of asymptotic translation lengths of all point-pushing pseudo-Anosov maps on any one punctured Riemann surface is one.
We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approx…
The minimum number of self-intersection points for members of a free homotopy class of curves on the punctured torus is bounded above in terms of the number L of letters required for a minimal description of the class in terms of the generators of the fundamental group and their inverses: it is less than or equal to (L…
Data discretization is an important step in the process of machine learning, since it is easier for classifiers to deal with discrete attributes rather than continuous attributes. Over the years, several methods of performing discretization such as Boolean Reasoning, Equal Frequency Binning, Entropy have been proposed,…
New method learns convolution-like structures from scratch.
During the past few years Boolean matrix factorization (BMF) has become an important direction in data analysis. The minimum description length principle (MDL) was successfully adapted in BMF for the model order selection. Nevertheless, a BMF algorithm performing good results from the standpoint of standard measures in…
This paper extends the work in [Suzuki, 1996] and presents an efficient depth-first branch-and-bound algorithm for learning Bayesian network structures, based on the minimum description length (MDL) principle, for a given (consistent) variable ordering. The algorithm exhaustively searches through all network structures…
The normalized maximized likelihood (NML) provides the minimax regret solution in universal data compression, gambling, and prediction, and it plays an essential role in the minimum description length (MDL) method of statistical modeling and estimation. Here we show that the normalized maximum likelihood has a Bayes-li…
Let $\mbox{Len}(K)$ be the minimum length of a knot on the cubic lattice (namely the minimum length necessary to construct the knot in the cubic lattice). This paper provides upper bounds for $\mbox{Len}(K)$ of a nontrivial knot in terms of its crossing number as follows: $\mbox{Len}(K) \leq \min \left\{ \fr…