LORL learns object-centric representations from vision and language.
problem Learning disentangled, object-centric scene representations from vision and language.
method LORL integrates unsupervised object discovery and segmentation with language input to learn object-centric concepts.
result LORL improves unsupervised object discovery methods and aids downstream tasks.
ROOTS learns to represent and render 3D scenes with object-centric models.
problem Learning to represent and render 3D scenes with object-centric compositionality.
method Probabilistic generative model for learning object representations and scene rendering from partial observations.
result The model can infer 3D object representations and render scenes from arbitrary viewpoints.
Graph neural networks detect anomalies in object-centric business processes.
problem Detecting anomalies in graph-like business processes.
method Graph convolutional autoencoder architecture for anomaly detection.
result Promising performance in detecting anomalies at the activity type and attributes level.
Study shows bottlenecks improve image segmentation quality.
problem Robust object discovery in real-world images remains challenging.
method Empirical investigation of reconstruction bottlenecks in GENESIS model.
result Reconstruction bottlenecks determine reconstruction and segmentation quality.
Slot Attention extracts object-centric representations from images.
problem Learning distributed representations that don't capture natural scene composition.
method Slot Attention module interfaces with CNN outputs to produce task-dependent abstract slots.
result Slot Attention enables generalization to unseen compositions.
Object-centric learning improves generalization and robustness in multi-object scenes.
problem Improving generalization and robustness in neural networks for scenes with multiple objects.
method Training state-of-the-art unsupervised models on multi-object datasets and evaluating segmentation metrics and downstream tasks.
result Object-centric representations are useful for downstream tasks and generally robust to most distribution shifts affecting objects, but less so for less structured shifts.
Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet …
KINet learns object interactions without supervision for robotic pushing.
problem Lack of supervised data for object-centric forward prediction.
method End-to-end unsupervised framework using keypoint representation and contrastive estimation.
result Automatically generalizes to unseen scenarios and accurately predicts future states.
Unified model improves object-centric world modeling with new abilities.
problem Integration of recent advances and temporal imagination abilities.
method Generative Structured World Models (G-SWM) framework.
result G-SWM achieves best or comparable performance in temporal generation.
Improved object segmentation and tracking in video using optical flow and initial state conditioning.
problem Challenges in fully unsupervised object-centric learning from video data.
method Weakly-supervised approach using optical flow and initial state conditioning.
result Conditioning the model on simple object location cues significantly improves instance segmentation in realistic synthetic data.
Study explores reinforcement learning in a complex game environment, analyzing rule inference and policy learning.
problem Learning optimal policies in environments with hidden rules.
method Investigated using the Game Of Hidden Rules (GOHR) environment, employing Feature-Centric and Object-Centric state representations with a Transformer-based A2C algorithm.
result Transformer-based A2C models outperform traditional methods in GOHR, demonstrating the effectiveness of representation strategies.
Generative model separates objects, positions, and appearances in images.
problem Generating complete images with accurate object positions and appearances.
method Structured latent representation that separates objects, positions, and appearances.
result Generative model can infer object decompositions and accurate depth orderings.
AXIOM learns games quickly with simple object models.
problem Data inefficiency in reinforcement learning.
method Expanding object-centric models with active inference.
result AXIOM learns games in 10,000 steps with minimal parameters.
Soft geometric bias improves physical dynamics predictions.
problem Learning physical dynamics with exact group equivariance can degrade performance.
method Object-centric world models using geometric algebra neural networks.
result Soft geometric inductive bias leads to better physical fidelity predictions.
Shared workspace improves neural module coordination in deep learning.
problem Pairwise interactions in deep learning models lack global coordination.
method Introduce a shared global workspace with bandwidth limitations among neural modules.
result Capacity limitations encourage specialization and synchronization.
GENESIS-V2 infers unordered object representations without iterative refinement.
problem Unsupervised learning of unordered object representations for complex images.
method Stochastic stick-breaking process for clustering pixel embeddings.
result GENESIS-V2 outperforms recent baselines in unsupervised image segmentation and scene generation.
One of the key challenges of visual perception is to extract abstract models of 3D objects and object categories from visual measurements, which are affected by complex nuisance factors such as viewpoint, occlusion, motion, and deformations. Starting from the recent idea of viewpoint factorization, we propose a new app…
While great progress has been made recently in automatic image manipulation, it has been limited to object centric images like faces or structured scene datasets. In this work, we take a step towards general scene-level image editing by developing an automatic interaction-free object removal model. Our model learns to …
Images seen during test time are often not from the same distribution as images used for learning. This problem, known as domain shift, occurs when training classifiers from object-centric internet image databases and trying to apply them directly to scene understanding tasks. The consequence is often severe performanc…
Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understan…
Generates coherent 3D scenes from monocular videos without supervision.
problem Lack of 3D scene modeling in video generation models.
method Trains a model to generate 3D scenes with moving objects and a background from monocular videos.
result Trained model generates coherent 3D scenes with multiple moving objects and a background.
Finding an embedding space for a linear approximation of a nonlinear dynamical system enables efficient system identification and control synthesis. The Koopman operator theory lays the foundation for identifying the nonlinear-to-linear coordinate transformations with data-driven methods. Recently, researchers have pro…
Group-equivariant subsampling layers improve CNNs' equivariance.
problem Non-translation equivariance in subsampling operations.
method Translation and group-equivariant subsampling/upsampling layers.
result Group-equivariant autoencoders learn equivariant representations.
We propose a framework for the completely unsupervised learning of latent object properties from their interactions: the perception-prediction network (PPN). Consisting of a perception module that extracts representations of latent object properties and a prediction module that uses those extracted properties to simula…
Fastest video anomaly detection via teacher-student distillation.
problem Anomaly detection in video at high speed.
method Adversarial knowledge distillation from object-level teacher models.
result 7-62 times faster than state-of-the-art methods.
Generative models learn from unlabeled videos via object segmentation and scene modeling.
problem Learning generative models from unlabelled videos.
method Decomposed into three subtasks: motion segmentation, background and foreground modeling, and scene sampling.
result Approach allows learning models that generalize beyond occlusions and represent scenes in a modular fashion.
Cosmos models scenes using neural encodings and symbolic attributes for compositional generalization.
problem Modeling scenes with high performance on unseen input scenes composed of known visual elements.
method Neurosymbolic grounding with neurosymbolic scene encodings and attention mechanisms.
result Establishes a new state-of-the-art for compositional generalization in world modeling.
Additive decoders tackle latent variables and image generation.
problem Latent variables identification and out-of-support image generation in representation learning.
method Additive decoders that can identify latent variables up to permutation and block-wise invertible transformations, and generate novel images by recombining observed factors.
result Additive decoders provide a new setting for nonlinear independent component analysis and can generate novel images by recombining observed factors.
We investigate whether a robot arm can learn to pick and throw arbitrary objects into selected boxes quickly and accurately. Throwing has the potential to increase the physical reachability and picking speed of a robot arm. However, precisely throwing arbitrary objects in unstructured settings presents many challenges:…
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before. We present object-centric perception, pred…
ECBMs unify concept-based interpretations in deep learning models.
problem Suboptimal final accuracy and lack of concept interaction and conditional dependencies.
method ECBMs use a set of neural networks to define joint energy, enabling concept correction and conditional dependency quantification.
result ECBMs achieve higher accuracy and richer concept interpretations compared to state-of-the-art methods.
MCD offers a complete model understanding for high-stake decisions.
problem Local model understanding in XAI methods is not sufficient for high-stake decisions.
method MCD extends concept-based methods to ensure global model understanding via multi-dimensional subspaces.
result MCD provides a complete model understanding, ensuring the model reasoning is related to the actual model.
Concept-driven OPE reduces variance in off-policy decision evaluation.
problem High variance in off-policy decision evaluation due to limited sample sizes.
method Integrating human-explainable concepts into OPE to reduce variance.
result Concept-based OPE estimators remain unbiased and reduce variance when concepts are known and predefined.
DCR improves interpretability of concept-based models by using neural networks to build rule structures.
problem Inability of concept-based models to provide transparent decision processes.
method DCR uses neural networks to build syntactic rule structures using concept embeddings and executes these rules on concept truth degrees.
result DCR improves interpretability by up to 25% on challenging benchmarks and discovers meaningful logic rules.
Concept bottleneck models enable concept manipulation for model interpretation.
problem Training models to directly predict labels without intermediate concepts.
method Train models to predict intermediate concepts at training time, then use these concepts to predict labels.
result Concept bottleneck models achieve competitive accuracy with standard models while enabling concept manipulation.
This review covers learning under concept drift, including detection, understanding, and adaptation.
problem Unforeseeable changes in data distribution over time impact machine learning performance.
method Reviews and analyzes methodologies and techniques for concept drift detection, understanding, and adaptation.
result Establishes a framework for learning under concept drift with three main components.
New method discovers concepts in hidden feature layers using sparse subspace clustering.
problem Local attribution methods fail to identify coherent model behavior across samples.
method Sparse Subspace Clustering (SSCC) for concept discovery.
result Empirically validated method for various image classification tasks.
Proposes a simple method to represent and manipulate concepts using polynomials and moment statistics.
problem Lack of a mathematical framework to define and operate on concepts.
method Characterizes concepts as zero sets of polynomials and uses moment statistics for representation; proposes a dictionary-based method to learn hierarchical structures.
result Signature of concepts can be used to discover common structures and recursively produce higher-level concepts.
Geometric framework detects concept frustration between human concepts and machine representations.
problem Aligning human concepts with machine learning representations.
method Geometric framework and similarity measures for detecting concept frustration.
result Concept frustration affects machine learning model performance and reorganizes learned concept representations.
Human explanations of high-level decisions are often expressed in terms of key concepts the decisions are based on. In this paper, we study such concept-based explainability for Deep Neural Networks (DNNs). First, we define the notion of completeness, which quantifies how sufficient a particular set of concepts is in e…
The study identifies latent concepts from diverse observations without assuming specific models.
problem Lack of general theoretical support for concept learning.
method Develops a nonparametric framework for identifying latent concepts from multiple classes of observations.
result Correctness guarantees for concept identification without parametric assumptions.
LCBM model improves image classification without human supervision.
problem Improving interpretability and generalization of unsupervised concept-based models.
method LCBM models concepts as random variables in a Bernoulli latent space, reducing the number of concepts without sacrificing performance.
result LCBM outperforms existing models in generalization and interpretability.
New models handle survival analysis with concept-based learning.
problem Survival analysis tasks involving event times with censored data.
method SurvCBM and SurvRCM models integrating concept-based learning with survival analysis.
result SurvCBM outperforms traditional survival models in numerical experiments.
New methods identify concepts in trained embeddings reliably without human labels.
problem Identifying interpretable concepts in trained embedding spaces without human labels.
method Explicitly connecting concept discovery to PCA and ICA, proposing novel approaches for dependent concepts.
result Proven methods outperform competitors on a variety of experiments, achieving up to 29% better alignment with ground truth.
SPARTAN learns sparse interaction graphs between objects in scenes.
problem Capturing structured interactions between entities in complex settings remains challenging.
method SPARTAN is a Transformer-based world model that learns sparse, context-dependent interaction graphs through sparsity regularization on attention patterns.
result SPARTAN achieves significantly improved few-shot adaptation to dynamics changes and robustness against distractors.
CREAM models enable concept-grounded predictions and interpretability.
problem Designing models that can encode and extend prior knowledge about concept-concept and concept-task relationships.
method Proposes a flexible and efficient framework (CREAMs) that encodes arbitrary C−C and CoY relationships, incorporating a side-channel for incomplete concept sets. result CREAM models achieve competitive task performance while encouraging concept-grounded predictions, avoiding concept leakage and achieving black-box-level performance.
Generative concept representations have three major advantages over discriminative ones: they can represent uncertainty, they support integration of learning and reasoning, and they are good for unsupervised and semi-supervised learning. We discuss probabilistic and generative deep learning, which generative concept re…
New framework quantifies and reduces concept-based models' leakage.
problem Information leakage in concept-based models reduces interpretability.
method Information-theoretic framework with CTL and ICL measures.
result Measures predict model behaviour and identify leakage causes.