Convex learning for diverse invariances in semi-inner-product space.
problem Efficiently learning invariant representations for a wide range of invariances.
method Developed a convex representation learning algorithm for generalized invariances modeled as semi-norms, introducing Euclidean embeddings for kernel representers in a semi-inner-product space.
result Accurate invariant representations learned efficiently and effectively, validated by experiments.
The study analyzes neural network predictions of knot invariants and finds that braid representations work best.
problem Understanding and predicting knot invariants using neural networks.
method Investigated different knot representations and invariants, proposed a cosine similarity score.
result Braid representations are best for predicting knot invariants, and some invariants are easier to learn than others.
The paper tackles generalization in machine learning by finding invariant representations of data.
problem Obtaining robust models that generalize well across different training environments.
method The paper introduces the concept of ε-approximate invariance to study the robustness of models to unseen SEMs. result The paper provides finite-sample out-of-distribution generalization guarantees for approximate invariance in linear SEMs.
STAR improves equivariant and invariant representation learning by routing projection heads.
problem Redundant feature learning in equivariant and invariant representation learning.
method Soft Task-Aware Routing (STAR) for projection heads specialization.
result Lower canonical correlations between invariant and equivariant embeddings.
Improves contrastive learning invariance with novel training objectives and feature averaging.
problem Contrastive learning's implicit invariance is insufficient for robust performance.
method Introduces a novel training objective and feature averaging approach to enforce invariance.
result Improved performance and robustness to transformations on downstream tasks.
Paper shows regularization improves robustness in domain generalization.
problem Improving robustness in domain generalization.
method Derives novel theoretical analysis to control representation smoothness and proposes a regularization method.
result Regularization improves robustness in domain generalization.
A new method evaluates invariant performance of IRM-based representations.
problem Impact of data changes on machine learning model performance.
method Proposes a novel method to evaluate invariant performance of IRM-based representations.
result Establishes a robust criterion to assess invariant performance of various representation techniques.
The paper analyzes the tradeoffs between accuracy and invariance in learning representations.
problem Achieving both accuracy and invariance in machine learning models.
method Information theoretic analysis of classification and regression settings.
result Characterization of the accuracy and invariance achievable by any representation of the data.
New findings show invariance alone isn't enough to identify latent causal variables.
problem Lack of theoretical insights for identifying latent causal variables when variables are latent.
method Assessed the connection between invariance and causal representation learning using impossibility results.
result Invariance alone is insufficient to identify latent causal variables.
Obtaining common representations from different modalities is important in that they are interchangeable with each other in a classification problem. For example, we can train a classifier on image features in the common representations and apply it to the testing of the text features in the representations. Existing m…
Method learns representations invariant to task-irrelevant details in reinforcement learning tasks.
problem Learning representations that are invariant to task-irrelevant details in reinforcement learning.
method Uses bisimulation metrics to learn robust latent representations that encode only task-relevant information.
result Demonstrates SOTA performance in modified visual MuJoCo tasks and a first-person driving task.
New algorithm learns invariant representations for robust neural networks.
problem Learning robust neural network representations that are invariant to certain factors.
method Causal perspective and distribution matching approach.
result Empirically, the algorithm achieves state-of-the-art performance on domain generalization.
Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation. Unfortunately, learning representations that exhibit invariance …
Adversarial techniques learn invariant representations across multiple domains.
problem Domain generalization from diverse studies to unseen domains.
method Adversarial censoring techniques for invariant representation learning.
result Limiting behavior of adversarial loss function as the number of domains grows.
Domain adaptation aims at generalizing a high-performance learner on a target domain via utilizing the knowledge distilled from a source domain which has a different but related data distribution. One solution to domain adaptation is to learn domain invariant feature representations while the learned representations sh…
Paper analyzes self-supervised learning using causal methods and proposes a new objective.
problem Lack of theoretical understanding of self-supervised learning success.
method Uses a causal framework to enforce invariance constraints on proxy classifiers.
result ReLIC objective improves generalization guarantees and outperforms existing methods.
We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a data representation such that the optimal classifier, on top of that data representation, matches for all training distributions. Through theo…
Paper develops upper-bounds for target general loss in multiple source DA and DG settings.
problem Complexity and trade-offs in multiple source domain adaptation and domain generalization.
method Defines two types of domain-invariant representations and studies their pros, cons, and trade-offs.
result Developed upper-bounds for target general loss offer insights into domain-invariant representations.
New method identifies stable latent variables across different domains using weak distributional invariances.
problem Learning causal representations for multi-domain datasets.
method Autoencoders incorporating weak distributional invariances.
result Autoencoders can identify stable latent variables across different domains.
The paper proposes a method to create domain-invariant representations using Wasserstein distance.
problem Domain shifts in training data affect machine learning model performance across different domains.
method The method combines classification/regression losses with a GAN-type discriminator to minimize the Wasserstein distance between domains.
result The approach produces the highest minimum classification accuracy and most invariant representation across domains.
Cross-domain sentiment analysis is currently a hot topic in the research and engineering areas. One of the most popular frameworks in this field is the domain-invariant representation learning (DIRL) paradigm, which aims to learn a distribution-invariant feature representation across domains. However, in this work, we …
Representations in the auditory cortex might be based on mechanisms similar to the visual ventral stream; modules for building invariance to transformations and multiple layers for compositionality and selectivity. In this paper we propose the use of such computational modules for extracting invariant and discriminativ…
New method learns robust representations by modeling environment variation.
problem Learning invariant representations across varying environments.
method Explicitly modeling variation across environments and marginalizing it out.
result Proposed method outperforms invariant-learning methods in various settings.
Learning invariant representations is an important problem in machine learning and pattern recognition. In this paper, we present a novel framework of transformation-invariant feature learning by incorporating linear transformations into the feature learning algorithms. For example, we present the transformation-invari…
We combine conditional variational autoencoders (VAE) with adversarial censoring in order to learn invariant representations that are disentangled from nuisance/sensitive variations. In this method, an adversarial network attempts to recover the nuisance variable from the representation, which the VAE is trained to pre…
GALA framework learns invariant graph representations via environment augmentation with minimal assumptions.
problem Learning invariant graph representations from different environments without additional assumptions.
method Developed GALA framework with minimal assumptions of variation sufficiency and consistency. Uses an assistant model to differentiate graph environment changes.
result Extracting maximally invariant subgraphs to proxy predictions identifies underlying invariant subgraphs for successful out-of-distribution generalization.
New neural architectures invariant to sign flips and basis symmetries for graph representation learning.
problem Learning invariant graph representations from eigenvectors.
method SignNet and BasisNet neural architectures that are invariant to sign flips and basis symmetries.
result Proven to be universal, approximating any continuous function of eigenvectors with desired invariances.
Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to solve this issue by pr…
Improves transferability of representations from source to target domains with weights and invariant representations.
problem Label shift between source and target domains in unsupervised domain adaptation.
method Integrates weights and invariant representations to bound the target risk, highlighting the role of inductive bias.
result Empirical evidence shows that weak inductive bias makes adaptation more robust.
Proposes FSM-IRL to learn invariant network representations considering feature and structural shifts.
problem Spatial heterogeneity and temporal dynamics lead to OOD generalization issues in geographic networks.
method Introduces FSM-IRL model that accounts for feature and structural distribution shifts using causal attention and reweighting.
result Demonstrates strong learning capabilities on geographic and social network datasets in OOD scenarios.
Paper presents a world model that learns invariant causal features using contrastive unsupervised learning.
problem Learning invariant causal features in unsupervised settings.
method Contrastive unsupervised learning with intervention invariant auxiliary task.
result Significantly outperforms state-of-the-art methods on out-of-distribution point navigation tasks.
Proposes a method to learn invariant representations for interpretability and fairness.
problem Learning invariant representations to achieve interpretability in algorithmic fairness.
method Adversarially trained model with null-sampling procedure to produce invariant representations in the data domain.
result Shows effectiveness on image and tabular datasets.
Unified approach to causal representation learning using invariance principles.
problem Identifying latent causal variables from high-dimensional observations.
method Guiding identification of causal variables with invariance principles rather than causal hierarchies.
result Unified method that mixes causal and non-causal assumptions improves treatment effect estimation.
Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invari…
Representations of sets are challenging to learn because operations on sets should be permutation-invariant. To this end, we propose a Permutation-Optimisation module that learns how to permute a set end-to-end. The permuted set can be further processed to learn a permutation-invariant representation of that set, avoid…
Deep networks learn hierarchical data by invariant representations.
problem How many examples are needed for deep networks to learn hierarchical data?
method Random Hierarchy Model: synthetic tasks inspired by language and images hierarchy.
result Deep networks learn by invariant representations and require a detectable number of correlations between low-level features and classes.
New method learns disentangled discrete representations using categorical variational autoencoders.
problem Learning disentangled representations from discrete latent spaces.
method Replaced standard Gaussian VAE with a categorical VAE to mitigate rotational invariance.
result Categorical distributions improve learning of disentangled representations.
Transfer learning aims to improve learning in target domain by borrowing knowledge from a related but different source domain. To reduce the distribution shift between source and target domains, recent methods have focused on exploring invariant representations that have similar distributions across domains. However, w…
The paper explores how to make machine learning models robust to domain shifts.
problem Machine learning models are unreliable in domains different from training.
method Introducing a broad formal notion of invariance and causal structures.
result The true underlying causal structure of the data plays a critical role in robustness.
Due to the ability of deep neural nets to learn rich representations, recent advances in unsupervised domain adaptation have focused on learning domain-invariant features that achieve a small error on the source domain. The hope is that the learnt representation, together with the hypothesis learnt from the source doma…
PeL separates sensory interface optimization from decision learning.
problem Optimizing sensory interfaces without task-specific information.
method Formal separation of perception and decision learning, using metrics for stability, informativeness, and geometry.
result Updates preserving invariants are orthogonal to decision gradients.
This paper develops methods for obtaining distribution-free prediction regions for invariant representations.
problem Distributional shifts in machine learning models.
method Invariant risk minimization and weighted conformity scores.
result Proves the effectiveness of adaptive conformal intervals for uncertainty estimation.
Estimates model performance under distribution shift using domain-invariant predictors.
problem Poor performance of models on test distributions different from training distributions.
method Uses domain-invariant predictors as a proxy for unknown target labels.
result Shows that the complexity of latent representations influences target risk.
Paper tackles reinforcement learning generalization through invariant policy optimization.
problem Learning policies that generalize beyond training domains.
method Invariant policy optimization principle and novel learning algorithm IPO.
result Significant improvements in generalization performance on unseen domains.
The paper explores how equivariant models' biases affect latent representations for better performance.
problem The impact of inductive biases on latent representations in equivariant models.
method Demonstrates the importance of accounting for inductive biases in latent representations of equivariant models.
result Effective invariant projections can be used to retain information in latent representations, improving downstream tasks.
A method for fair representation learning through bi-level optimization and implicit differentiation.
problem Ensuring fair predictors invariant across sub-groups.
method Bi-level optimization with inner-loop for invariant predictors, implicit path alignment for efficiency.
result Consistently better trade-off in prediction performance and fairness measurement.
Direct formula found for ADO invariants from homological representations.
problem Computing ADO invariants from quantum group representations.
method Direct homological formula for ADO invariants using partial traces of homological representations.
result Direct formula for ADO invariants without further truncations.
Adversarial representation learning is a promising paradigm for obtaining data representations that are invariant to certain sensitive attributes while retaining the information necessary for predicting target attributes. Existing approaches solve this problem through iterative adversarial minimax optimization and lack…