Deep networks better approximate functions with compositional structure.
problem Approximating functions with complex structures.
method Design deep networks with compositional structure, leveraging the blessing of compositionality.
result Deep networks can approximate functions better than shallow networks when the function has a compositional structure.
High-dimensional data simplifies problems, contrary to the curse of dimensionality.
problem Exponential difficulty in high-dimensional problems.
method Analysis of high-dimensional datasets and their geometric properties.
result Generic high-dimensional datasets exhibit simple geometric properties.
Paper explores why compositionality is not the right approach for emergent communication.
problem Lack of interaction between biological and computational models of emergent communication.
method Exploring the claim that compositionality is the wrong target for explaining natural language emergence.
result Suggests reflexivity as a better target for explaining natural language emergence.
Capsule networks improve at detecting changes in compositionality with routing.
problem Capsule networks struggle with detecting changes in compositionality.
method Introduced a loss function based on routing entropy to improve compositionality.
result Capsule networks with the new loss function better detect changes in compositionality.
Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to ask whether this compositional structure is reflected in the the inputs' learned representations. W…
Investigates how neural network capacity impacts the learning of compositional languages.
problem The importance of communicative bandwidth in emergent language learning.
method Exploration of how neural network capacity affects the learning of compositional languages.
result There is a specific range of model capacity and channel bandwidth that induces compositional structure.
Reply to Ogburn et al. on their critique of Wang and Blei's work.
problem Critique of Wang and Blei's work on the blessings of multiple causes.
method Discussion and clarification of Wang and Blei's claims and findings.
result Wang and Blei's premise is correct and there are no foundational errors.
Tests neural networks for compositional generalization in language.
problem Understanding how neural networks generalize in a compositional manner.
method Developed five tests bridging linguistic and philosophical compositionality with neural models.
result Revealed strengths and weaknesses of different neural architectures.
We present the multiplicative recurrent neural network as a general model for compositional meaning in language, and evaluate it on the task of fine-grained sentiment analysis. We establish a connection to the previously investigated matrix-space models for compositionality, and show they are special cases of the multi…
Counterexamples show deconfounder fails to control multi-cause confounding.
problem Deconfounding method fails to handle multi-cause confounding.
method Incorrectly inferring independence and joint independence from conditional independence.
result Two simple counterexamples demonstrate deconfounder's failure.
DNNs can learn complex functions efficiently by breaking the curse of dimensionality.
problem Learning complex functions efficiently in high-dimensional spaces.
method Combining compositionality and symmetry learning with generalization bounds.
result DNNs can learn functions with bounded F1-norm efficiently, reducing the curse of dimensionality. The paper develops a framework for abstracting causal models using category theory.
problem Difficulties in changing the variables used to describe a system, especially from fine-grained to coarse-grained.
method Introduces a category of interventional causal models and uses enriched category theory to prove compositionality properties.
result Compositionality of model transformations is established, with bounded errors for each step.
Deep learning models generate languages that lack abstract reasoning.
problem Lack of abstract reasoning in deep learning-generated languages.
method Analyzed emergent language from two multi-agent games with compositional measures.
result Deep learning solutions often fail to generalize to out-of-training examples.
New algorithm tackles multi-agent reinforcement learning issues.
problem Multi-agent reinforcement learning suffers from the curse of many agents.
method Proposes MF-FQI algorithm based on mean embeddings of distributions.
result Establishes a non-asymptotic analysis for MF-FQI algorithm.
Deep learning exploits latent structure to learn high-dimensional tasks.
problem Statistical intractability of high-dimensional tasks in deep learning.
method Study of locality and compositionality in data, tasks, and neural network representations.
result Neural networks improve generalization with more training examples.
Proposes a framework for compositional generalization in language models.
problem Lack of compositional generalization in neural networks compared to humans.
method Introduces Generalized Grammar Rules (GGRs) for transduction tasks, formalizing symmetry-based constraints.
result Framework enables models to generalize compositionally, similar to human learning.
Comments on deconfounder method's pros and cons.
problem Addressing confounding in causal inference.
method Critically reviews deconfounder method and suggests improvements.
result Points out advantages and limitations of deconfounder method.
Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we prop…
An important challenge for human-like AI is compositional semantics. Recent research has attempted to address this by using deep neural networks to learn vector space embeddings of sentences, which then serve as input to other tasks. We present a new dataset for one such task, `natural language inference' (NLI), that c…
Paper proves multiplicative weight updates can train neural networks without learning rate tuning.
problem Vanishing and exploding gradients in gradient descent for compositional functions.
method Proves descent lemma for compositional functions using multiplicative weight updates and derives Madam optimizer.
result Madam optimizer trains state-of-the-art neural networks without learning rate tuning.
Comment refutes the deconfounder method's premise about ignorability.
problem The deconfounder method's premise about ignorability is incorrect.
method The deconfounder method proposes a variable making multiple causes conditionally independent controls for unmeasured multi-cause confounding.
result No fact about observed data alone can be informative about ignorability.
Graphs help identify unobserved confounders in causal inference.
problem Unobserved confounders skew causal inference from observational data.
method Causal graphical view and probabilistic factor models.
result The deconfounder algorithm makes valid inferences of intervention distributions.
KernelBiome tackles microbiome research by improving predictive performance and interpretability.
problem Challenges in analyzing high-throughput sequencing data, especially in microbiome research.
method KernelBiome is a kernel-based nonparametric regression and classification framework for compositional data, incorporating prior knowledge and capturing complex signals.
result Improved predictive performance compared to state-of-the-art machine learning methods, with two novel quantities for interpretability.
Regularization can improve both privacy and performance in machine learning models.
problem Privacy vs. Utility trade-off in machine learning models.
method The study uses logistic regression with ridge regularization and a leave-one-out analysis tool.
result Increasing the number of parameters can improve both privacy and performance when coupled with proper regularization.
New methods detect modular structure in neural networks, revealing surprising effects of dropout.
problem Detecting functional modules in neural networks for learning, compositionality, and generalization.
method Two families of methods: upstream and downstream, to define similarity between units.
result Dropout dramatically increased modularity, and there's little agreement between upstream and downstream methods.
New method to measure compositional generalization on realistic data.
problem Limited compositional generalization in machine learning.
method Maximizing compound divergence while ensuring small atom divergence.
result Machine learning architectures fail to generalize compositionally.
New research shows CFG improves high-dimensional data generation.
problem Characterizing CFG's effect on high-dimensional distributions.
method High-dimensional analysis of CFG's impact on target distributions.
result CFG accurately reproduces the target distribution in high dimensions.
New approach uses neural networks to learn program structure and parameters.
problem Learning programs and their structure efficiently.
method Free category prior over programs, end-to-end learning of structure and parameters.
result Neural networks can serve as primitives in probabilistic programs.
Paper shows how noisy data can improve robust decision-making.
problem The challenge of noisy data in decision-making.
method Distributionally robust optimization (DRO) with a novel ambiguity set construction.
result Noisy data can lead to more robust and equitable decisions.
The interaction between transitivity and sparsity, two common features in empirical networks, implies that there are local regions of large sparse networks that are dense. We call this the blessing of transitivity and it has consequences for both modeling and inference. Extant research suggests that statistical inferen…
ROOTS learns to represent and render 3D scenes with object-centric models.
problem Learning to represent and render 3D scenes with object-centric compositionality.
method Probabilistic generative model for learning object representations and scene rendering from partial observations.
result The model can infer 3D object representations and render scenes from arbitrary viewpoints.
Stacking improves inference for multimodal Bayesian posterior distributions.
problem Difficulty of MCMC in moving between modes and underestimation of posterior uncertainty.
method Parallel runs of MCMC, variational, or mode-based inference, combined using Bayesian stacking.
result Stacking efficiently samples from multimodal posterior distributions and represents uncertainty better than variational inference.
High-dimensional models pose both safety benefits and risks.
problem Emergent problems in safety alignment due to high-dimensional representations.
method Detailed visualizations and lower-dimensional subspace projections.
result Dimensional reduction preserves safety alignment while avoiding linear jailbreaking.
A new GP model for non-Gaussian data with explicit inverse warping.
problem Limited expressiveness and computational complexity of Gaussian processes for non-Gaussian data.
method Compositionally-warped Gaussian processes (CWGP) with explicit inverse warping.
result CWGP provides more accurate predictions and shorter computation times than traditional warped GPs.
Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world ro…
Study shows how transformers learn to combine simple tasks into complex ones.
problem Understanding how transformers learn to perform complex tasks not seen during training.
method Controlled setting involving variable assignment and modular addition; partitioned training data analysis.
result Small transformers can generalize to unseen combinations of variables and numbers.
New methods identify concepts in trained embeddings reliably without human labels.
problem Identifying interpretable concepts in trained embedding spaces without human labels.
method Explicitly connecting concept discovery to PCA and ICA, proposing novel approaches for dependent concepts.
result Proven methods outperform competitors on a variety of experiments, achieving up to 29% better alignment with ground truth.
This paper tackles nonsmooth optimization in machine learning.
problem Nonsmoothness in machine learning optimization problems.
method Identifying specific structures and leveraging them for practical applications.
result Compression, acceleration, and dimension reduction are possible with nonsmooth optimization.
Comment on deconfounder method, highlighting causal identification issues and risks of parametric assumptions.
problem Causal identification issues in settings where deconfounder method is suggested.
method Summarizing arguments against causal identification, cautioning on parametric assumptions, and discussing nonparametric identification results.
result Nonparametric identification results may be more interesting for detecting causal identification from observed data.
Shared workspace improves neural module coordination in deep learning.
problem Pairwise interactions in deep learning models lack global coordination.
method Introduce a shared global workspace with bandwidth limitations among neural modules.
result Capacity limitations encourage specialization and synchronization.
This paper examines how adversarial perturbations affect model performance and equilibrium learning.
problem Adversarial perturbations and covariate shifts impact model performance and equilibrium learning.
method Characterizes the extrapolation region in regression and classification, analyzes dynamics of adversarial learning games.
result Establishes two directional convergence results: a blessing in regression and a curse in classification.
This paper evaluates how well neural models can solve complex tasks by breaking them into simpler ones.
problem Measuring neural models' ability to solve complex tasks by breaking them into simpler subtasks.
method Characterized axes of compositional generalization, introduced a benchmark suite of tasks, and improved Transformer models' attention mechanisms.
result Modified Transformer models generally perform better than natural baselines in solving complex tasks, but challenges remain.
We present efficient differentiable implementations of second-order multi-hop reasoning using a large symbolic knowledge base (KB). We introduce a new operation which can be used to compositionally construct second-order multi-hop templates in a neural model, and evaluate a number of alternative implementations, with d…
The paper explores fairness in multi-component recommender systems.
problem How to ensure fairness in recommender systems composed of multiple models.
method Study of fairness ranking metrics, theoretical analysis, and empirical evaluation.
result Fairness in recommendation systems can be achieved by improving individual components.
Graph neural networks help AI agents learn more complex language.
problem Understanding how AI agents learn and use language.
method Developed graph referential games to compare different AI models.
result Graph neural networks enable AI to learn more complex, compositional language.
This work uses visualizations to make generalization of neural networks more intuitive.
problem Understanding the reasons behind neural networks' ability to generalize to unseen data.
method Visualization methods to explain the geometry of loss landscapes and the role of dimensionality in optimization.
result Visualization helps in understanding how optimizers settle into minima that generalize well.
Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art. We define a space of kernel structures which are built compositionally by adding and multiplying a small number of base kernels. We present a method for searching over this space of structures which mirro…
Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.
problem Collaborative learning in distributed RL settings with limited data sharing.
method Analyzes synchronous and asynchronous federated Q-learning, proposes importance averaging.
result Achieves linear speedup with heterogeneity, robust to local trajectory heterogeneity.