Survey of latent factor models for relational learning to improve their inductive abilities.
problem Understanding and improving latent factor models for multi-relational knowledge graphs.
method Experimental survey of state-of-the-art models, creating synthetic genealogies to assess strengths and weaknesses.
result Proposed new research directions to improve latent factor models.
Study shows different trajectory prediction models generalize better under OoD conditions.
problem Comparing trajectory prediction models' robustness across different datasets.
method Training models on Argoverse 2 and testing on Waymo Open Motion, and vice versa, with various augmentation strategies.
result Smallest model with highest inductive bias performs best in OoD generalization.
Study links neural network inductive bias, feature learning, and generalization on Boolean functions.
problem Understanding how neural networks learn and generalize on Boolean data.
method End-to-end analysis of depth-2 discrete fully connected networks and DNF formulas, using Monte Carlo learning.
result Predictable training dynamics and interpretable features emerge, linking inductive bias and generalization.
Machine learning refactors knowledge to improve learning efficiency.
problem Inductive program synthesis efficiency through knowledge restructuring.
method Introduces Knorf, a system that refactors knowledge bases using constraint optimization.
result Learning from refactored knowledge improves predictive accuracy fourfold and reduces learning time by half.
CLUTRR benchmarks NLU systems' ability to infer logical rules from text.
problem NLU models' failure to generalize systematically and robustly.
method CLUTRR suite requires inference of kinship relations from short stories, evaluating generalization and robustness.
result Graph-based models outperform state-of-the-art NLU models in generalization and robustness.
Transformers capture combinatorial tasks with bounded error and logarithmic sample dependence.
problem Capturing complex combinatorial tasks with bounded error and sample efficiency.
method Formal definition of algorithmic capture, empirical analysis of infinite-width transformers, upper bounds on computational complexity.
result Transformers exhibit an inductive bias favoring simpler algorithmic procedures over higher complexity ones.
Study shows gMPNNs struggle with OOD link prediction in larger test graphs.
problem Inductive out-of-distribution link prediction in larger test graphs.
method Theoretical analysis and development of a gMPNN with structural pairwise embeddings.
result Structural node embeddings from gMPNNs converge to random guessing as test graphs grow.
This research explores inductive biases for deep learning to improve AI's higher-level cognition.
problem Current AI struggles with flexible out-of-distribution and systematic generalization.
method Examines and proposes new inductive biases for deep learning.
result Identifies specific inductive biases for higher-level sequential processing.
Rule induction explains neural network predictions globally.
problem Understanding and explaining the behavior of trained models.
method Calculate feature importance, transform inputs, simplify space, fit rule induction model.
result Rule sets explain neural network predictions with 0.80 macro-averaged F-score.
New approach uses neural networks to learn program structure and parameters.
problem Learning programs and their structure efficiently.
method Free category prior over programs, end-to-end learning of structure and parameters.
result Neural networks can serve as primitives in probabilistic programs.
RPPs improve deep learning models with soft equivariance constraints.
problem Balancing expressiveness and inductive biases in deep learning.
method Introducing Residual Pathway Priors (RPPs) to convert hard constraints into soft priors.
result RPPs enable models to learn structured solutions while retaining flexibility.
Robots learn object dynamics from visuals using GNNs and relational biases.
problem Challenging for robots to reason like humans about physical interactions.
method Graph Neural Networks (GNNs) with relational inductive bias.
result Auto-Predictor outperforms GN-based models and auto-encoder baseline.
Pruning FCNs reveals sub-networks that match CNNs' performance.
problem Understanding the inductive bias of pruning in neural networks.
method Iterative magnitude pruning of a simple FCN followed by analysis of the resulting architecture.
result Pruned FCNs exhibit key features of CNNs, suggesting new architectural biases.
A user-friendly interface constructs effective background knowledge from ER diagrams.
problem Inefficient construction of background knowledge by domain experts in ILP systems.
method Design of a graphical user interface to interact with Entity Relationship diagrams to construct modes for a probabilistic logic learning system.
result Domain experts can construct effective background knowledge on par with experts using the graphical interface.
Graph networks enhance deep learning with relational inductive biases.
problem Generalizing beyond experiences, a key human intelligence trait.
method Integrating relational inductive biases into deep learning architectures.
result Graph networks facilitate learning about entities, relations, and rules.
Theoretical analysis of CNNs' inductive biases and their efficiency in approximating functions.
problem Understanding and optimizing the inductive biases in deep CNNs.
method Theoretical analysis combining multichanneling, downsampling, weight sharing, and locality.
result Deep CNNs with O ( log d ) \mathcal{O}(\log d) O ( log d ) depth can approximate any continuous function, and require O ~ ( log 2 d ) \widetilde{\mathcal{O}}(\log^2d) O ( log 2 d ) samples for sparse functions. Study logical generalization in GNNs using a new benchmark.
problem Understanding how GNNs adapt to new logical tasks.
method Developed GraphLog benchmark suite for logical tasks, evaluated GNNs in supervised, pretraining, and continual learning settings.
result Logical diversity during training affects GNNs' ability to generalize.
Model-free RL agents can learn effective planning without explicit model.
problem Challenging domains with combinatorial complexity.
method Model-free RL with inductive bias in neural networks.
result Model-free RL can exhibit planning characteristics without explicit model.
Overparameterized autoencoders can memorize training examples.
problem Understanding generalization in overparameterized neural networks.
method Analyzing autoencoders of varying depths and types.
result Autoencoders concentrate learned functions around training examples.
New method models negative correlations in knowledge graphs.
problem Lack of negative correlation in probabilistic extensions of order embeddings.
method Box lattice measures for probabilistic modeling of negative correlations.
result Models can now capture negative correlations and disjoint concepts.
New methods to define self-inductance by regularizing divergent integrals.
problem Defining self-inductance for identical loops.
method Regularization of divergent integrals using Neumann/Weber formula.
result Established new methods to calculate self-inductance.
Deep learning's anomalous generalization explained by standard frameworks.
problem Anomalous generalization in deep neural networks.
method Intuitive understanding and rigorous characterization using PAC-Bayes and countable hypothesis bounds.
result Deep learning's anomalous generalization can be explained by soft inductive biases.
Paper explores how knowledge distillation transfers inductive biases between models.
problem Transferring inductive biases between models for tasks with limited data.
method Knowledge distillation applied to models with different inductive biases (LSTMs vs. Transformers, CNNs vs. MLPs).
result Effect of inductive biases is transferred through knowledge distillation, impacting both performance and solution characteristics.
Interpolated-MLPs control inductive bias for better performance in low-compute tasks.
problem Low-compute performance gap between MLPs and CNNs.
method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.
result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.
New method quantifies inductive bias for machine learning tasks.
problem Quantifying the amount of inductive bias in machine learning models.
method Estimates inductive bias by modeling loss distribution of random hypotheses.
result Higher dimensional tasks require greater inductive bias.
Neural networks favor Boolean functions with low entropy.
problem Understanding the inductive bias of neural networks.
method Analyzing a single-layer perceptron with random initial weights.
result There is a strong intrinsic bias towards low entropy Boolean functions.
One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
OTI extends OTP for inductive semi-supervised learning.
problem Inductive semi-supervised learning for out-of-sample data.
method Optimal transport-based approach extended to inductive tasks.
result OTI outperforms state-of-the-art methods in experiments.
The paper proposes a method to transfer knowledge across different settings using causal theory.
problem Learning transfer across similar but different settings.
method Bayesian perspective of causal theory induction, integrating instance-level associative learning and abstract-level structural causal knowledge.
result The proposed model achieved transfer behavior across trials and learning situations, unlike RL algorithms.
Tensor dropout improves CNN robustness and efficiency.
problem Limited generalization and robustness of deep CNNs.
method Tensor layers with low-rank structures and tensor dropout.
result Tensor dropout outperforms other methods in robustness and efficiency.
People can learn complex visual concepts from just a few examples.
problem Understanding how people learn and categorize visual concepts from limited data.
method Bayesian program learning model that searches for the best explanation of observations.
result People's judgments are broadly consistent with a Bayesian program learning model, indicating they can learn rich algorithmic abstractions from sparse input data.
Unified framework combines trace-induced quantum kernels for improved machine learning models.
problem Improving performance of quantum machine learning models using trace-induced kernels.
method Developed a unified framework combining various trace-induced quantum kernels, including global fidelity and local projected kernels, as Lego kernels.
result Local projected kernels can achieve comparable performance to global fidelity kernels with fewer quantum resources.
Strong inductive biases prevent harmless interpolation in overparameterized models.
problem Understanding the conditions under which overparameterized models can interpolate noise without overfitting.
method Theoretical analysis of high-dimensional kernel regression and deep neural networks, focusing on the role of inductive biases.
result The strength of an estimator's inductive bias determines whether interpolation is harmless or requires fitting noise for good generalization.
Deep ResNets favor low bottleneck rank with proper hyperparameters.
problem Understanding the inductive bias of deep neural networks.
method Computed minimum-norm weights of a deep linear ResNet.
result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.
PGNs dynamically infer and use graph structures to improve model generalization.
problem Static graph structures inferred by machine learning practitioners are often suboptimal for tasks.
method PGNs augment graphs with dynamically inferred pointers for improved model generalization.
result PGNs outperform unrestricted GNNs and Deep Sets on dynamic graph connectivity tasks.
This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.
problem Generalization from easy to hard tasks, especially out-of-domain generalization.
method Formalizes inductive generalization, introduces Inductive Learning, and outlines steps to adapt techniques for learning model successors.
result A new learning paradigm (Inductive Learning) that emphasizes induction and universal properties of learning and computation.
Transformers can learn new tasks from diverse pretraining data but struggle with out-of-domain tasks.
problem Transformer models' ability to learn new tasks in-context is limited by their pretraining data coverage.
method Investigation of transformer models trained on ( x , f ( x ) ) (x, f(x)) ( x , f ( x )) pairs, comparing in-context learning capabilities across different task families. result Transformers can identify and learn within task families in their pretraining data but fail with out-of-domain tasks.
We propose a new model of minority game with so-called smart agents such that the standard deviation and the total loss in this model reach the theoretical minimum values in the limit of long time. The smart agents use trail and error method to make a choice but bring global optimization to the system, which suggests t…
We introduce the notion of large scale inductive dimension for asymptotic resemblance spaces. We prove that the large scale inductive dimension and the asymptotic dimensiongrad are equal in the class of r-convex metric spaces. This class contains the class of all geodesic metric spaces and all finitely generated groups…
MLPs can approximate any function in context, challenging the importance of in-context universality.
problem Understanding why transformers are more effective than classical models.
method Proved MLPs with trainable activation functions are universal in context.
result Transformer success is likely due to factors other than in-context universality.
New RL environments help AI learn causal relationships from visual data.
problem Learning causal relationships from visual data for AI agents.
method Designing benchmark RL environments and evaluating representation learning algorithms.
result Explicitly incorporating structure and modularity improves causal induction in model-based RL.
Unsupervised MT struggles with morphologically rich languages.
problem Limitations of unsupervised machine translation on morphologically rich languages.
method Adversarial unsupervised alignment of word embedding spaces for bilingual dictionary induction.
result A simple trick exploiting weak supervision from identical words improves unsupervised bilingual dictionary induction performance.
If p : Y → X p : Y \to X p : Y → X is an unramified covering map between two compact oriented surfaces of genus at least two, then it is proved that the embedding map, corresponding to p p p , from the Teichmüller space T ( X ) {\cal T}(X) T ( X ) , for X X X , to T ( Y ) {\cal T}(Y) T ( Y ) actually extends to an embedding between the Thurston compactification of the tw…
We introduce SADs to reveal how network architecture shapes score-based generative models.
problem Understanding and predicting the inductive biases of score-based generative models.
method Introducing Score Anisotropy Directions (SADs) to analyze network architecture.
result SADs reliably capture model behavior and correlate with performance.
Novel framework for Bayesian reinforcement learning infers value function distributions.
problem Bayesian reinforcement learning's challenges in inferring value function distributions.
method Inferential Induction framework for Bayesian reinforcement learning, developing Bayesian Backwards Induction algorithm.
result Proposed algorithm is competitive with state-of-the-art methods.
Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.
problem The impact of noise on interpolating models with strong inductive biases.
method Analyzing linear and classification models with sparse ground truths, proving fast rates for interpolators.
result Strong inductive biases can lead to faster but noisier interpolators, contrary to intuition.
GraIL predicts relations by reasoning over subgraphs, outperforming embeddings.
problem Relation prediction in knowledge graphs using latent representations is limited.
method Graph neural network with inductive bias to learn entity-independent relational semantics.
result GraIL outperforms existing rule-induction baselines in the inductive setting.
New approach relaxes inductive biases of physics-inspired NNs for better performance.
problem Challenges in applying physics-inspired NNs to real-world systems.
method Examined and relaxed inductive biases of Hamiltonian NNs, improving performance on non-conservative systems.
result Improved performance on practical, non-conservative systems by relaxing inductive biases.