Invariant Causal Set Covering Machines avoid spurious associations.
problem Learning algorithms for rule-based models are vulnerable to spurious associations.
method Building on invariant causal prediction, propose Invariant Causal Set Covering Machines for conjunctions/disjunctions of binary-valued rules.
result The method can identify causal parents of a variable of interest in polynomial time.
Algorithm removes spurious concepts from neural network representations without harming task performance.
problem Spurious correlations hinder neural network out-of-distribution generalization.
method Iterative algorithm that identifies two orthogonal subspaces in neural network representation.
result Algorithm outperforms existing methods on computer vision and natural language processing benchmarks.
Last layer retraining improves robustness to spurious correlations without high computational costs.
problem Neural networks can rely on spurious features like backgrounds for predictions.
method Simple last layer retraining on large models.
result Last layer retraining matches or outperforms state-of-the-art approaches on spurious correlation benchmarks.
Develops tools to decompose spurious variations in causal models.
problem Understanding and decomposing spurious variations in causal relationships.
method Formal tools for decomposing spurious effects in Markovian and Semi-Markovian models.
result First results on non-parametric decomposition of spurious effects and sufficient conditions for identification.
Online algorithm detects and removes spurious features for strong generalization.
problem Predictive models learn spurious correlations from incomplete data.
method Online continual learning algorithm that detects and removes spurious features.
result Algorithm converges to solutions with strong generalization.
DMs emerge from DenseAMs, transitioning from memorization to generalization.
problem Hindered memory retrieval in DenseAMs due to spurious states.
method Examined diffusion models through the lens of DenseAMs, focusing on their generative process.
result Identified a critical phase in DMs transitioning from memorization to generalization.
The study identifies spurious correlations in high-dimensional regression and quantifies their impact.
problem Spurious correlations in high-dimensional regression models.
method Statistical characterization of spurious correlations, quantifying their amount via ridge regularization.
result The value of regularization strength that minimizes test loss is in an interval where spurious correlations increase.
GRASP removes spurious correlations in fine-tuned models, improving task performance and reducing bias.
problem Fine-tuned models can latch onto spurious correlations, leading to bias and reduced generalization.
method GRASP identifies and removes spurious correlations from model weights without removing latent factors.
result GRASP significantly reduces bias and improves task performance in various fine-tuning tasks.
Despite alarm over the reliance of machine learning systems on so-called spurious patterns, the term lacks coherent meaning in standard statistical frameworks. However, the language of causality offers clarity: spurious associations are due to confounding (e.g., a common cause), but not direct or indirect causal effect…
The paper formalizes criteria for non-spurious and disentangled representations using causal methods.
problem Formalizing criteria for non-spurious and disentangled representations in representation learning.
method Causal perspective, counterfactual quantities, observable consequences of causal assertions.
result Computable metrics for assessing representation learning based on observed data.
Transformer-based method for causal discovery with prior knowledge integration.
problem Complex nonlinear dependencies and spurious correlations in time series data.
method Multi-layer Transformer forecaster with gradient-based causal structure extraction and attention masking for prior knowledge integration.
result Significant improvement in causal discovery and causal lag estimation compared to state-of-the-art methods.
Removing spurious features can hurt model accuracy and disproportionately affect different groups.
problem Interference from spurious features in robust model performance across different groups.
method Characterization and analysis of spurious feature removal in noiseless overparameterized linear regression.
result Removal of spurious features can decrease accuracy and disproportionately affect different groups, even in balanced datasets.
Framework uses human annotations to make models robust to spurious correlations.
problem Machine learning models fail when unmeasured variables change test distributions.
method Human annotations to augment training examples, UV-DRO objective for robustness.
result Improvements of 5-10% on digit recognition task and 1.5-5% on NYPD Police Stops analysis.
SGD quickly learns a spurious XOR feature before the signal feature, revealing learning dynamics.
problem Over-reliance on spurious correlations in neural networks trained by SGD.
method Theoretical analysis of SGD on two-layer ReLU networks trained on XOR data.
result SGD learns the spurious feature first and exponentially fast, dominating the signal feature.
This research examines rare spurious correlations in neural networks and their impact on accuracy and privacy.
problem Rare spurious correlations in neural networks and their privacy risks.
method Introducing spurious patterns correlated with a fixed class to a few training examples, analyzing ℓ2 regularization and Gaussian noise. result Rare spurious correlations can significantly impact neural network accuracy and privacy, and specific mitigation methods can be effective.
We consider the optimization problem associated with training simple ReLU neural networks of the form x↦∑i=1kmax{0,wi⊤x} with respect to the squared loss. We provide a computer-assisted proof that even if the input distribution is standard Gaussian, even if the dime…
Mitigates spurious correlations without bias labels.
problem Spurious correlations bias model performance.
method Introduces a novel training objective and debiasing method DPR.
result DPR achieves state-of-the-art performance.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
Optimization algorithm CoCo improves causal inference from diverse data.
problem Identifying true causal relationships from data with spurious associations.
method CoCo optimizes for causal inference using environments with invariant causal relationships.
result CoCo provides more accurate causal estimates and predictions.
CLIP models robustness to spurious features is re-evaluated using a new dataset.
problem Existing robustness tests of CLIP models may not fully reflect their performance on spurious features.
method Crafted a new dataset (CounterAnimal) to reveal CLIP models' reliance on realistic spurious features.
result CLIP models are robust to spurious features learned from their training data, not ImageNet.
Study identifies and analyzes spurious correlations in data-driven models.
problem Spurious correlations in data-driven models are unreliable and hard to detect.
method Collect and analyze synthetic datasets generated from causal graphs to investigate spurious correlations.
result Patterns connecting spurious correlation hypotheses and model design choices were observed.
Self-training avoids spurious features in domain adaptation.
problem Domain shift with large differences between source and target domains.
method Entropy minimization on unlabeled target data, initialized with a source classifier.
result Entropy minimization avoids using spurious features in large domain shifts.
Current OOD benchmarks overestimate model robustness to spurious correlations.
problem Spurious correlations degrade OOD performance, but benchmarks show the opposite.
method Analyze OOD datasets for spurious correlations and derive conditions for robustness.
result Current OOD benchmarks are misspecified and overestimate model robustness.
Mitigates biases in reward models using variational inference.
problem Spurious correlations in reward models that align large language models with human preferences.
method Formulates data-generating process, identifies non-spurious latent variables, and uses variational inference to recover them.
result Effective mitigation of spurious correlation issues, yielding more robust reward models.
AFR simplifies reducing reliance on spurious features, improving model performance.
problem Reducing reliance on spurious features for out-of-distribution generalization.
method Automatic Feature Reweighting (AFR) updates the model with a weighted loss.
result AFR improves model performance on benchmarks with minimal compute.
This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
problem Understanding the loss landscape of sparse neural networks, especially one-hidden-layer networks.
method Analyzes sparse networks with dense and sparse final layers, focusing on linear and non-linear models.
result Sparse networks can have no spurious valleys under certain conditions, but spurious valleys and minima can exist for wide sparse networks.
Study reveals different types of critical points in shallow neural networks.
problem Optimization challenges in two-layer ReLU networks with symmetry.
method Symmetry analysis, bifurcation theory, and geometric group actions.
result Different types of spurious minima have distinct loss behavior.
Paper proposes mechanism learning to reverse causal inference in ML.
problem Machine learning models learn associational, not causal, relationships.
method Causally weighted Gaussian mixture models (CW-GMMs).
result CW-GMMs can deconfound observational data for reverse causal inference.
Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.
problem Spurious critical points on the boundary of low-rank matrix manifold.
method Riemannian gradient descent with dynamical low-rank approximation and rescaled gradient flow.
result Riemannian gradient descent escapes some spurious critical points on the boundary of the manifold.
Logit correction improves model performance by correcting spurious correlations.
problem Spurious correlations lead to poor model performance during inference.
method Proposes logit correction (LC) loss to mitigate spurious correlations.
result LC loss outperforms state-of-the-art solutions by 5.5% absolute improvement.
Gradient flow in phase retrieval escapes spurious minima with high probability.
problem Understanding gradient-based optimization in high-dimensional non-convex functions.
method Analytical and numerical study of gradient dynamics in phase retrieval.
result Gradient flow avoids spurious minima by drifting along unstable directions.
The paper analyzes how deep models memorize spurious features.
problem Understanding how deep models memorize spurious features in training data.
method Characterizes spurious feature memorization via model stability and feature alignment.
result Memorization of spurious features weakens as generalization capability increases.
New method prevents classifiers from relying on spurious correlations.
problem Group invariant learning fails to prevent classifiers from depending on spurious correlations.
method Statistical independence tests to construct groups and reweight samples by group label proportion.
result New method significantly outperforms existing group invariant learning methods in generalizing to spurious correlation shifts.
Neural networks provide a rich class of high-dimensional, non-convex optimization problems. Despite their non-convexity, gradient-descent methods often successfully optimize these models. This has motivated a recent spur in research attempting to characterize properties of their loss surface that may explain such succe…
The FastICA algorithm is one of the most popular iterative algorithms in the domain of linear independent component analysis. Despite its success, it is observed that FastICA occasionally yields outcomes that do not correspond to any true solutions (known as demixing vectors) of the ICA problem. These outcomes are comm…
Deep models learn spurious features correlated with target, but can still perform well.
problem Spurious correlations in feature learning.
method Empirical risk minimization and specialized group robustness training.
result Feature representations learned by ERM are competitive with specialized methods.
LaCIM avoids spurious correlation by modeling latent causal factors.
problem Avoiding spurious correlation in supervised learning.
method Introducing latent variables for causal prediction and optimizing over latent space.
result Improved interpretability, robustness, and prediction power on OOD scenarios.
We show that for any convex differentiable loss, a deep linear network has no spurious local minima as long as it is true for the two layer case. This reduction greatly simplifies the study on the existence of spurious local minima in deep linear networks. When applied to the quadratic loss, our result immediately impl…
When the linear measurements of an instance of low-rank matrix recovery satisfy a restricted isometry property (RIP)---i.e. they are approximately norm-preserving---the problem is known to contain no spurious local minima, so exact recovery is guaranteed. In this paper, we show that moderate RIP is not enough to elimin…
New rationalization method avoids spurious correlations.
problem Spurious correlations in rationalization criteria.
method Invariant rationalization using game theory constraints.
result Rationales generalize better and align with human judgments.
Study on W2S generalization with spurious correlations, proposing remedies.
problem Understanding and improving W2S generalization with spurious correlations.
method Theoretical analysis and algorithmic remedies for W2S fine-tuning.
result W2S always happens with sufficient pseudolabels when group fractions match, but may fail otherwise.
Large learning rates enhance model robustness and compressibility.
problem Achieving robustness and resource-efficiency in machine learning models.
method Identifying and utilizing large learning rates as a facilitator for robustness and compressibility.
result Large learning rates produce desirable representation properties and compare favorably to other methods.
New findings show Bregman proximal algorithms can get stuck near non-stationary points.
problem Bregman proximal algorithms can get stuck near non-stationary points, misleadingly suggesting convergence.
method Analysis of Bregman proximal algorithms and their behavior near non-stationary points.
result Bregman proximal algorithms can get stuck near spurious stationary points, even in convex problems.
Kernel method approximates Koopman operator eigenfunctions.
problem Complexity of computing Koopman operator spectra.
method Kernel-based approach to construct principal eigenfunctions.
result Principal eigenfunctions match linearization eigenvalues.
Analyzes the Hessian of ReLU networks, proving skewed eigenvalue distribution.
problem Characterizing the Hessian at spurious minima in shallow ReLU models.
method Symmetry breaking and representation theory techniques.
result Proves skewed eigenvalue distribution of Hessian at spurious minima.
DORA analyzes deep neural networks' internal representations to detect spurious correlations.
problem Detecting spurious correlations in deep neural networks' internal representations.
method DORA uses Extreme-Activation (EA) distance measure to assess representation similarities.
result Identifies internal representations capable of detecting spurious correlations.
Mining association rules is an important technique for discovering meaningful patterns in transaction databases. Many different measures of interestingness have been proposed for association rules. However, these measures fail to take the probabilistic properties of the mined data into account. In this paper, we start …
Nonconvex matrix recovery is known to contain no spurious local minima under a restricted isometry property (RIP) with a sufficiently small RIP constant δ. If δ is too large, however, then counterexamples containing spurious local minima are known to exist. In this paper, we introduce a proof technique that is capa…