Study optimizes step size for Metropolis algorithm in non-identifiable cases.
problem Optimizing step size for Metropolis algorithm in non-identifiable models.
method Analytical derivation of average acceptance rate for non-identifiable cases.
result Developed optimization principle for step size based on average acceptance rate.
Latent feature models (LFM)s are widely employed for extracting latent structures of data. While offering high, parameter estimation is difficult with LFMs because of the combinational nature of latent features, and non-identifiability is a particularly difficult problem when parameter estimation is not unique and ther…
New research shows LLMs can't be explained by statistical generalization alone.
problem Understanding why large language models (LLMs) perform well despite statistical generalization limitations.
method Examined the non-identifiability of AR probabilistic models and their implications for LLMs.
result Non-identifiability of LLMs leads to different behaviors and requires a separate theoretical explanation.
This paper tackles non-identifiability in financial market simulations using multivariate time series data.
problem Non-identifiability issue in social simulation models, leading to indistinguishable simulated time series data.
method Proposes a maximization-based aggregation function to form a new calibration objective function using multiple time series features.
result Significant improvements in alleviating non-identifiability and achieving higher simulation fidelity.
Hypothesis testing in singular models is fundamentally about identifiable vs. non-identifiable parameters.
problem Testing in singular models is inherently problematic due to non-identifiability and degeneracy of Fisher information.
method Formalized the overlap obstruction and showed that hypotheses over non-identifiable parameters are untestable, while those over identifiable parameters reduce to classical testing.
result Hypotheses over non-identifiable parameters are untestable, while those over identifiable parameters reduce to classical testing.
Variational autoencoders often collapse, showing latent variables are non-identifiable.
problem Posterior collapse in variational autoencoders due to non-identifiable latent variables.
method Proves latent variable non-identifiability causes posterior collapse. Proposes latent-identifiable models using Brenier maps and input convex neural networks.
result Latent-identifiable models resolve posterior collapse and provide meaningful representations.
New method warns of counterfactual non-identifiability in DSCMs.
problem Counterfactual inference from observational data is non-identifiable even without unobserved confounding.
method Prove counterfactual identifiability for monotonic generation mechanisms, provide impossibility result for general mechanisms, propose method for estimating worst-case errors.
result Non-identifiability of counterfactual inference from observational data, even in absence of unobserved confounding.
A new method resolves non-identifiability in reward modeling using anchor labels.
problem Non-identifiability in reward modeling from pairwise preferences alone.
method Anchor-guided Variance-aware Reward Modeling (AVRM) framework.
result AVRM resolves non-identifiability and improves reward modeling performance.
IMA addresses non-identifiability in nonlinear ICA by assuming orthogonal Jacobian columns.
problem Non-identifiability in nonlinear ICA.
method IMA assumes orthogonal Jacobian columns and extends to manifold settings.
result IMA circumvents non-identifiability issues and can be beneficial for higher-dimensional observations.
Overparametrized neural networks retain significant epistemic uncertainty even with sufficient data.
problem Epistemic uncertainty in overparametrized neural networks persists despite model identifiability.
method Analysis of non-identifiability and characterization of residual uncertainty in one-hidden-layer ReLU networks.
result Substantial parameter uncertainty remains even when the underlying function is fully identified.
New method uses logical relations to derive bounds and inequality constraints from causal models.
problem Recovering bounds and inequality constraints from unobserved confounding.
method Using rules of probability and restrictions on counterfactuals implied by causal graphical models.
result Powerful method to recover known and novel bounds and constraints.
Analysis of DPPs and k-DPPs via spectral decomposition reveals identifiable parameters and non-identifiability gaps.
problem Identifying parameters of DPPs and k-DPPs through spectral decomposition.
method Spectral decomposition of the covariance matrix, analysis of invariances, and counting arguments.
result Identifiability of parameters changes fundamentally for k-DPPs, with specific invariances and non-identifiability gaps.
Solves parameter non-identifiability in Bayesian LTI system identification.
problem Parameter non-identifiability in standard Bayesian approaches for LTI system identification.
method Embedding canonical forms of LTI systems within the Bayesian framework.
result Unlocking the use of meaningful priors and robust uncertainty estimates.
Bayesian Neural Networks with Latent Variables (BNN+LVs) capture predictive uncertainty by explicitly modeling model uncertainty (via priors on network weights) and environmental stochasticity (via a latent input noise variable). In this work, we first show that BNN+LV suffers from a serious form of non-identifiability…
This work explores how overparametrization and priors affect Bayesian neural network posteriors.
problem Symmetries, non-identifiabilities, and weight-space priors fragment and inflate BNN posteriors.
method We study the interplay between overparametrization and priors in BNN posteriors, deriving key phenomena and validating through experiments.
result Overparametrization induces structured, prior-aligned weight posterior distributions.
Neural networks can learn relationships that traditional models cannot.
problem Identifying factors that differentiate neural networks from traditional models.
method Proving non-identifiability of neural networks compared to smooth parametric models.
result Neural networks can learn nontrivial relationships that traditional models cannot.
New framework estimates treatment effects based on preferences.
problem Estimating treatment effects with flexible outcomes.
method Preference-based Conditional Treatment Effect (CPTE) framework.
result CPTE provides interpretable targets and new identifiability conditions.
A new method for binary ICA using non-stationary sources.
problem Independent component analysis of binary data.
method Linear mixing model in latent space, followed by binary observation model with non-stationary sources.
result Proves non-identifiability with few observed variables but identifies with more variables.
Proposes efficient bounds for causal effect estimation under weak confounding.
problem Estimating causal effects with weakly confounded variables.
method Develops an efficient linear program to derive upper and lower bounds on causal effect under small entropy of unobserved confounders.
result Bounds are consistent and tighter for weakly confounded variables.
Transformers without skip connections collapse token representations to a single direction.
problem Rapid convergence of token representations to a single direction in self-attention-only Transformers.
method Analysis of layer normalization, residual connections, and multi-head attention mechanisms.
result Residual connections prevent rank collapse in real Transformers, while MLPs generate new feature directions.
The Rashomon effect shows many models can perform similarly, explored in this paper.
problem Why do many models perform similarly in machine learning?
method Categorized causes into statistical, structural, and procedural sources.
result Structural multiplicity persists and cannot be resolved without additional assumptions.
Proposes clustering and pruning to simplify causal data fusion models.
problem Combining observational and experimental data to identify causal effects.
method Generalizes pruning and clustering operations for multiple data sources.
result Derives conditions for inferring causal effects from simplified models.
Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
problem Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
method Analyzing the behavior of memory-efficient optimizers like GaLore, which project gradients onto a rank-r subspace recomputed every T steps.
result Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
Nonnegative matrix factorization (NMF) is a popular dimension reduction technique that produces interpretable decomposition of the data into parts. However, this decompostion is not generally identifiable (even up to permutation and scaling). While other studies have provide criteria under which NMF is identifiable, we…
POSCMs extend SCMs for causal modeling with latent contexts.
problem Causal modeling with latent contexts and endogenous mechanisms.
method Kolmogorov-Arnold-Sprecher edge-functional decomposition for explicit parametrization.
result Identifiability of structure and mechanisms under latent context.
We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching. Within a framework of conditional entropy, we propose both adversarial and non-adversarial approaches to learn desirable matched joint distributions for unsupervised and supervised tasks. We…
Unified framework for singular statistical models using observable charts.
problem Non-identifiability and breakdown of classical asymptotic theory in singular models.
method Invariant framework based on observable charts to define local coordinate systems in model space.
result Observable order provides a lower bound on KL divergence vanishing rate in singular models.
The paper proposes a new method to calibrate multiple computer models simultaneously.
problem Calibrating multiple computer models one at a time is inefficient.
method Developed a probabilistic framework using customized neural networks.
result Simultaneous calibration improves predictive accuracy but can be non-identifiable in high dimensions.
We introduce thermodynamic response functions for singular Bayesian models.
problem Singular Bayesian models violate regular asymptotics due to non-identifiability and degenerate Fisher geometry.
method Posterior tempering induces thermodynamic response functions, linking WAIC, WBIC, and singular fluctuation.
result WAIC, WBIC, and singular fluctuation are unified within a thermodynamic response framework.
Single sample estimation for hard-constrained models like SAT and coloring problems.
problem Estimating parameters of Markov Random Fields with hard constraints using a single sample.
method Pseudo-likelihood estimator with coupling techniques.
result Single-sample estimation is not always possible for hard constraints, and existence of an estimator is related to satisfiability.
Unified framework for SGMoE resolves estimation and selection issues.
problem Non-identifiability, coupled differential relations, and tight coupling in softmax-Gated models.
method Unified statistical framework with Voronoi-type loss functions and dendrograms of mixing measures.
result Consistent selection of the number of experts without model sweeps, optimal parameter rates under overfitting.
LLMs can memorize economic data and recall exact values before their training cutoff.
problem Evaluating the trustworthiness of LLMs' economic forecasts during their training period.
method Demonstrated through counterfactual forecasting and analysis of LLMs' recall ability.
result LLMs have memorized economic and financial data, leading to recall-level accuracy before their knowledge cutoff.
The paper discovers a hidden component in data using an autoencoder with a discriminator.
problem Discovering a single independent latent variable in data.
method An autoencoder with a discriminator is used to recover the hidden component.
result The approach can recover the hidden component up to entropy-preserving transformations.
We consider a blind identification problem in which we aim to recover a statistical model of a network without knowledge of the network's edges, but based solely on nodal observations of a certain process. More concretely, we focus on observations that consist of single snapshots taken from multiple trajectories of a d…
Expands experimental design for causal discovery from limited data.
problem Challenges in causal discovery from observational and interventional data.
method Bayesian optimal experimental design incorporating recent advances in causal discovery.
result Active causal discovery of large, nonlinear SCMs with both intervention target and value selection.
New framework extends ICA for non-independent variables, identifying pairwise mean independence.
problem Non-independent variables complicating ICA recovery.
method Algebraic recovery algorithm based on least-squares optimization over the orthogonal group.
result Pairwise mean independence is identifiable, robust to independence constraints.
Method estimates observation functions in state-space models without supervision.
problem Unsupervised learning of non-invertible observation functions in nonlinear state-space models.
method Nonparametric generalized moment method using constrained regression.
result Estimates function space of identifiability from state process.
TRA detects causal direction from bivariate data using geometric shapes.
problem Inferring causal direction from observational data is challenging and unreliable.
method TRA compares rank-based copula-standardized residual clouds to detect causal direction.
result TRA is robust and superior in detecting causal direction across various scenarios.
Estimates multiple related causal graphs with shared causal order.
problem Discovering multiple related Gaussian DAGs with shared causal order.
method Proposes a l1/l2-regularized MLE for joint estimation of K linear structural equation models. result Joint estimator achieves better sample complexity and consistency in causal order recovery.
Interpretable framework evaluates structure learning methods for causal discovery from observational data.
problem Evaluation of structure learning methods under assumption violations in causal discovery.
method Six-dimensional evaluation metric (DOS) tailored for causal discovery.
result Amortized causal discovery delivers results with high proximity to the optimal solution.
Kernel method improves instrumental variable regression rates.
problem Nonparametric instrumental variable regression with weak instruments.
method Kernel-based two-stage least-squares method, strong L2 convergence analysis. result Minimax optimal rates for instrumental regression under standard assumptions.
Bayesian Non-negative Matrix Factorization (NMF) is a promising approach for understanding uncertainty and structure in matrix data. However, a large volume of applied work optimizes traditional non-Bayesian NMF objectives that fail to provide a principled understanding of the non-identifiability inherent in NMF-- an i…
New method learns DAGs from noisy data without identifiability assumptions.
problem Learning DAGs from non-identifiable Gaussian models with heteroscedastic noise.
method Mixed-integer programming framework for medium-sized problems.
result Asymptotically optimal solution with early stopping criterion.
Proposes bounds on bias from low-dimensional representations in CATE estimation.
problem Bias in CATE estimation due to low-dimensional representations.
method Proposes a refutation framework to estimate bounds on representation-induced confounding bias.
result Demonstrates effectiveness of refutation framework in practice.
BCD Nets use variational inference to estimate DAGs with uncertainty.
problem Uncertainty in inferring causal graphs from limited data.
method Variational inference framework for Bayesian DAG estimation.
result BCD Nets outperform maximum-likelihood methods in low data regimes.
SLT explains neural network success by closing theory-practice gap.
problem Failure of classical inference and learning theory in modern neural networks.
method Physics-inspired Singular Learning Theory (SLT) applied to neural networks.
result SLT recovers known and novel scaling laws for neural network phase transitions.
Inversion-free natural gradient method for Riemannian manifolds.
problem Hindered by the need for Euclidean space, Fisher information matrix inversion, and computational cost.
method Intrinsic, inversion-free natural gradient method on Riemannian manifolds, using moving approximation of inverse FIM.
result Almost-sure convergence rates and sub-quadratic storage complexity for large-scale applications.
The study uses pre-trained neural networks to adjust for confounding in non-tabular data.
problem Neglecting non-tabular data sources can lead to biased ATE estimates.
method Leverages latent features from pre-trained neural networks to adjust for confounding.
result Neural networks can achieve fast convergence rates for ATE estimation with latent features.