The study examines causal razors and their logical relations, highlighting a dilemma in causal discovery.
problem Selecting a reasonable scoring criterion for causal discovery algorithms.
method Review and logical comparison of numerous causal razors, focusing on parameter minimality in multinomial models.
result Parameter minimality poses a dilemma in selecting a reasonable scoring criterion for causal discovery algorithms.
MDL principle aids in learning neural network-based causal structures.
problem Learning causal relationships from observations with neural networks.
method Prequential minimum description length (MDL) principle.
result Competitive results on synthetic and real-world data, often recovering correct structure.
Geometric Occam's Razor shapes deep learning solutions.
problem Understanding the regularization in over-parameterized neural networks.
method Analyzing the geometric model complexity and Dirichlet energy in neural networks.
result Over-parameterized neural networks are implicitly regularized by geometric model complexity.
We consider the problem of identifying the causal direction between two discrete random variables using observational data. Unlike previous work, we keep the most general functional model but make an assumption on the unobserved exogenous variable: Inspired by Occam's razor, we assume that the exogenous variable is sim…
Transformers prefer simpler explanations in hierarchical tasks.
problem Navigating tasks with varying complexity levels.
method Well-controlled testbeds based on Markov chains and linear regression.
result Transformers favor the least complex sufficient explanation when presented with simpler data.
Bayesian model selection can be misled by ELBO under certain conditions.
problem Misleading model selection when using ELBO for Bayesian inference.
method Analysis of ELBO-based hyperparameter learning in a simple regression model.
result Bayesian model selection can prefer overfit models when ELBO is used, contrary to evidence.
Deep neural networks perform well due to a balance of architecture, training, and structured data.
problem Understanding why overparameterized DNNs perform well.
method Bayesian approach to analyze the interplay between network architecture, training algorithms, and data structure.
result Structured data and an inductive bias towards simple functions explain DNN success.
The article applies Occam's Razor to non-parametric model building, minimizing the number of bits for data encoding.
problem Overlooking the role of model parameters in data encoding leads to inefficient probability density estimators.
method Extends bit counting to model parameters, providing a true measure of complexity for parametric models.
result Minimizing total bit requirement leads to smoother, more efficient probability density estimates and fewer relevant parameters.
Bayesian approach sparsifies neural networks efficiently.
problem Efficiently pruning neural networks to save resources.
method Sparsifiability via the Marginal likelihood (SpaM) framework.
result Prunes neural networks effectively without significant loss in performance.
This work connects symmetries and conserved quantities in machine learning.
problem Improving machine learning models by learning conserved quantities.
method Using Noether's theorem, learn symmetries and conserved quantities directly from data.
result Correctly identifies conserved quantities and improves model performance.
New approach to nonuniform learnability using measure theory.
problem Nonuniform learnability of hypotheses with varying sample sizes.
method Measure theoretic approach to redefine nonuniform learnability, introducing a new algorithm (Generalize Measure Learnability).
result Achieved statistical consistency in learning countable hypothesis classes.
The paper improves deep learning generalization bounds using PAC-Bayes compression.
problem Improving generalization bounds for deep neural networks.
method Quantizing neural network parameters in a linear subspace to develop tight generalization bounds.
result Large models can be compressed significantly, explaining Occam's razor.
The paper explores how smaller data sets can lead to better model selection decisions.
problem Model selection in small data regimes.
method Empirical study of generalization performance with varying training set sizes.
result Training on smaller subsets of data can lead to more reliable model selection decisions.
OccamNet finds interpretable symbolic fits to data efficiently.
problem Complex neural models extrapolate poorly and are hard to interpret.
method Samples functions, biases towards better fits, and uses cross-entropy matching.
result Outperforms state-of-the-art symbolic regression methods on real-world datasets.
Paper proposes a method to estimate scientific parameters in hybrid models without relying on model architecture.
problem Estimating unknown parameters in hybrid models combining machine learning and scientific models.
method Sharpness-aware minimization adapted for hybrid modeling, focusing on model simplicity.
result Demonstrates effectiveness of SAM-based hybrid model learning for scientific parameter estimation.
This study explores how feature graphs enhance GNNs' performance in modeling interactions.
problem Improving GNNs' ability to model feature interactions effectively.
method Investigates feature graphs and their importance in GNNs, using experiments and theoretical support.
result Edges between interacting features are crucial for GNNs, while non-interaction edges can degrade performance.
ResNets minimize circuit size for fitting data in HTMC regime.
problem Finding the simplest algorithm that fits data.
method Defining HTMC and ResNet norms to relate circuit size and function fitting.
result Minimizing ResNet norm is equivalent to finding a circuit with minimal nodes.
Bayesian evidence helps compare models but can overfit.
problem Comparing hypotheses consistent with observations.
method Marginal likelihood, Occam's razor, PAC-Bayes bounds.
result Marginal likelihood can negatively correlate with generalization.
Bayesian inference simplified for machine learning models.
problem Difficulty in specifying general prior belief in machine learning architectures.
method Parsimonious inference using information theory and Kolmogorov complexity.
result Framework quantifies model complexity and prediction information, reducing memorization.
New method measures generalizability of deep neural networks based on decision boundary complexity.
problem Lack of generalization methods for deep neural networks.
method Created Decision Boundary Complexity (DBC) score to measure DNN complexity.
result Simpler decision boundaries lead to better generalizability, supporting Occam's Razor.
NLR models often perform worse than LR for outlying input data in environmental sciences.
problem NLR models often give poor predictions for input data outside the training domain.
method Screened input data for outliers, using linear extrapolation for outliers based on NLR within the non-outlier domain.
result NLROR approach reduces poor extrapolation and tends to outperform NLR and LR for outliers. New estimate reduces overfitting risk in machine learning models.
problem Error rate on test data may not reflect true population error due to adaptive data analysis practices.
method Introduces Rip van Winkle's Razor, a simple estimate of overfit to test data based on information content.
result Shows non-vacuous estimate of deviation in many modern settings.
We exhibit a strong link between frequentist PAC-Bayesian risk bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization risk bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanatio…
Investigates principles of generalization in list learning, refutes sample compression conjecture.
problem Determining applicability of classical principles in list PAC learning.
method Examines uniform convergence and sample compression in list PAC learning.
result Sample compression fails in list PAC learning, refutes conjecture.
New model estimates sparse transport maps for high-dimensional data.
problem Estimating optimal transport maps in high-dimensional spaces.
method Proposes a new model using a family of translation invariant costs and sparsity-inducing norms.
result Sparse transport maps that apply Occam's razor to reduce complexity.
Compressed imitation learning uses simplicity priors for efficient expert behavior copying.
problem Efficiently learn expert behaviors with minimal data.
method Utilizes policy simplicity as a prior for sample-efficient imitation learning.
result Significantly higher scores achieved with limited expert demonstrations.
Bayesian nonparametric models, such as Gaussian processes, provide a compelling framework for automatic statistical modelling: these models have a high degree of flexibility, and automatically calibrated complexity. However, automating human expertise remains elusive; for example, Gaussian processes with standard kerne…
We present an asymptotic analysis of Viterbi Training (VT) and contrast it with a more conventional Maximum Likelihood (ML) approach to parameter estimation in Hidden Markov Models. While ML estimator works by (locally) maximizing the likelihood of the observed data, VT seeks to maximize the probability of the most lik…
Study reveals how model volume affects learning curves in machine learning.
problem Understanding the double descent risk phenomenon in machine learning.
method Investigates the role of model volume using MDL, Occam's Razor, and information geometry.
result Model volume can explain the double descent risk, suggesting better generalization with increased dimensionality.
Why do deep neural networks (DNNs) benefit from very high dimensional parameter spaces? Their huge parameter complexities vs stunning performance in practice is all the more intriguing and not explainable using the standard theory of model selection for regular models. In this work, we propose a geometrically flavored …
New framework learns disentangled causal representations from observed labels.
problem Learning meaningful disentangled causal representations from observed data.
method ICM-VAE framework using flow-based diffeomorphic functions and causal disentanglement prior.
result Induces highly disentangled causal factors and improves robustness.
Reinterprets Granger causality with causal Bayesian networks and Reichenbach's principles.
problem Lack of a rigorous causal foundation in Granger causality.
method Reinterpreting Granger causality through Reichenbach's principles and causal Bayesian networks, implementing as c-GC.
result c-GC provides a more principled framework for causal discovery in observational datasets.
The well known maximum-entropy principle due to Jaynes, which states that given mean parameters, the maximum entropy distribution matching them is in an exponential family, has been very popular in machine learning due to its "Occam's razor" interpretation. Unfortunately, calculating the potentials in the maximum-entro…
New measures for causal entropy and information gain studied.
problem Quantifying causal relationships in machine learning.
method Formal study of causal entropy and information gain.
result Established fundamental properties and relationships.
We quantify causal bias in continuous treatment settings.
problem Identifying and quantifying causal bias in continuous treatment scenarios.
method Developed a novel characterization of causal bias in structural causal models, proving conditions for zero bias and efficient estimation.
result Causal bias can be estimated efficiently under certain structural equation restrictions, allowing for causal regularization of predictive models.
Improved Granger causality method for dynamic time series data.
problem Traditional Granger causality method assumes constant causalities, failing to model dynamic causalities.
method Dynamic window-level Granger causality (DWGC) method with causality indexing.
result Improved DWGC method better detects window-level causalities.
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
This paper presents a light-weight and accurate deep neural model for audiovisual emotion recognition. To design this model, the authors followed a philosophy of simplicity, drastically limiting the number of parameters to learn from the target datasets, always choosing the simplest earning methods: i) transfer learnin…
Paper characterizes and represents pairwise causal background knowledge for improved causal inference.
problem Improving causal inference by handling pairwise causal constraints.
method Graphical characterization, direct causal clause (DCC), unified representation, MPDAG, polynomial-time algorithms.
result Pairwise causal background knowledge uniquely decomposes into MPDAG and DCCs, improving causal effect identification.
A new method clusters heterogeneous subgroups for accurate causal learning.
problem Diverse causal relationships across different time spans, regions, or strategies.
method Nonlinear Causal Kernel Clustering
result Reduction in prediction error through enhanced causal learning.
Framework for Granger causality in extreme events.
problem Identifying causal links from extreme events in time series.
method Causal tail coefficient and novel inference method.
result Framework outperforms state-of-the-art methods in detecting Granger causality in extremes.
Proposes DCNAR for dynamic causal inference from neural time series.
problem Uncertainty and evolution of causal structure in real-world domains.
method Two-stage neural causal modeling integrating discovery and inference.
result Dynamic causal inferences are more stable and meaningful than alternatives.
New algorithms for causal bandits without knowing the graph structure.
problem Causal bandit problems with unknown graph structure.
method Developed novel causal bandit algorithms for causal trees, forests, and general graphs without prior knowledge of the causal graph.
result Regret guarantees significantly improved over standard MAB algorithms under mild conditions.
This review explores causal decision-making to improve decision quality.
problem Effective decision-making requires understanding causal relationships.
method Causal structure learning, causal effect learning, and causal policy learning.
result Challenges in causal decision-making are identified and recent advances are discussed.
New model improves data augmentation for causal tasks.
problem Optimizing causal models robustly under Wasserstein distances.
method Proposes a new G-Causal Normalizing Flow architecture.
result Empirically outperforms standard generative models.
DoWhy-GCM extends causal inference in graphical models for diverse queries.
problem Addressing diverse causal queries in graphical causal models.
method Specify cause-effect relations via a causal graph, fit causal mechanisms, pose causal queries.
result Identification of root causes, attribution of causal influences, diagnosis of causal structures.
ABCI infers causal models and queries simultaneously using Bayesian active learning.
problem Inference of causal models and effects in a two-stage process is inefficient and unnatural.
method Active Bayesian Causal Inference (ABCI) using Gaussian processes for sequentially designing experiments.
result ABCI is more data-efficient and accurate in learning causal queries from fewer samples.
Optimizes causal effects on unknown graphs using Causal Entropy Optimization.
problem Optimizing causal effects in unknown causal graphs.
method Causal Entropy Optimization (CEO) framework that generalizes Causal Bayesian Optimization (CBO). Incorporates causal structure uncertainty in surrogate models and intervention selection.
result CEO achieves faster convergence to global optimum compared to CBO and improves upon sequential structure learning.