Proposes a topological framework to study modular invariants and related concepts.
problem Exploring modular invariants and related concepts in topological quantum field theory.
method Topological paradigm in alterfold topological quantum field theory.
result Establishes a novel integral identity for modular invariance across multiple Morita contexts.
Children learn concepts without explicit teaching by aligning internal systems.
problem Learning concepts from noisy perceptual experience without explicit teaching.
method Using information in the environment to build and align conceptual systems.
result The more concepts and systems there are, the easier learning becomes.
Combines neural networks and expert rules for concept-based learning.
problem Extending concept-based learning with machine learning models.
method Form constraints for joint probability distribution and represent feasible set as a convex polytope.
result Neural networks can be trained to satisfy expert rules without violating them.
Node Masking improves GNNs' scalability and generalization.
problem Improving GNNs' ability to handle arbitrary graphs.
method Introducing Node Masking to enhance GNNs' performance.
result Node Masking enables GNNs to generalize and scale better.
This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.
problem Generalization from easy to hard tasks, especially out-of-domain generalization.
method Formalizes inductive generalization, introduces Inductive Learning, and outlines steps to adapt techniques for learning model successors.
result A new learning paradigm (Inductive Learning) that emphasizes induction and universal properties of learning and computation.
New approach uses neural networks to learn program structure and parameters.
problem Learning programs and their structure efficiently.
method Free category prior over programs, end-to-end learning of structure and parameters.
result Neural networks can serve as primitives in probabilistic programs.
Paper revises power theory using classical mechanics concepts.
problem Clarifying instantaneous power definitions for circuit elements.
method Defines power using classical mechanics concepts like velocity and momentum.
result General and compact expression for inductance, capacitance, and resistance powers.
Researchers apply concept-based explainability to EEG data.
problem Understanding the internal states of complex EEG transformer models.
method Concept Activation Vectors (CAVs) adapted for EEG data, using externally labeled datasets and anatomically defined concepts.
result Both approaches to concept formation yield valuable insights into EEG model representations.
Enhanced ICM ensemble detects concept drift better with novel betting functions.
problem Addressing Concept Drift in machine learning models.
method Refined ICM approach with improved betting functions and base estimators.
result The ensemble approach outperforms previous methods on benchmark datasets.
RCAV quantifies model sensitivity to semantic concepts, improving interpretability methods.
problem Lack of semantic interpretability in image classification models.
method RCAV calculates concept gradients and ascent steps to assess model sensitivity to semantic concepts.
result RCAV yields more accurate and robust interpretations of model behavior.
Spectral regularization simplifies sequence models by focusing on grammatical simplicity.
problem Sequence modeling challenges in learning tasks.
method Introduces spectral regularization based on Hankel matrices and trace norm, addressing bi-infinite matrices with an unbiased estimator.
result Demonstrates spectral regularization's potential benefits on Tomita grammars.
Recently, Geometric Deep Learning (GDL) has been introduced as a novel and versatile framework for computer-aided disease classification. GDL uses patient meta-information such as age and gender to model patient cohort relations in a graph structure. Concepts from graph signal processing are leveraged to learn the opti…
Unified theory linking node embeddings and graph representations.
problem Clarifying the relationship between node embeddings and graph representations.
method Using invariant theory, the paper establishes a theoretical framework bridging node embeddings and structural graph representations.
result Proves equivalence between node embeddings and structural graph representations, showing they are interchangeable for various tasks.
New approach to disentangled representations using mutual information.
problem Disentangled representations lack sufficient inductive biases.
method Formulate disentanglement through mutual information and conditional independence.
result Violation of mutual information assumption leads to loss of disentanglement.
Transductive learning considers a training set of m labeled samples and a test set of u unlabeled samples, with the goal of best labeling that particular test set. Conversely, inductive learning considers a training set of m labeled samples drawn iid from P(X,Y), with the goal of best labeling any future sample…
Enhances visual explanations with logical rules for complex concepts.
problem Lack of explanatory power for deep learning models, especially for relational concepts.
method Combining LIME for highlighting and Aleph for logic rules.
result Generated relational rules can be explicitly linked to input images.
Paper explores how knowledge distillation transfers inductive biases between models.
problem Transferring inductive biases between models for tasks with limited data.
method Knowledge distillation applied to models with different inductive biases (LSTMs vs. Transformers, CNNs vs. MLPs).
result Effect of inductive biases is transferred through knowledge distillation, impacting both performance and solution characteristics.
GD with large init shows incremental learning in matrix factorization.
problem Understanding GD's behavior with large initial values in matrix factorization.
method Signal-to-noise ratio concepts and inductive arguments.
result Uncovering an incremental learning phenomenon in GD with large initialization.
Interpolated-MLPs control inductive bias for better performance in low-compute tasks.
problem Low-compute performance gap between MLPs and CNNs.
method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.
result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.
New method quantifies inductive bias for machine learning tasks.
problem Quantifying the amount of inductive bias in machine learning models.
method Estimates inductive bias by modeling loss distribution of random hypotheses.
result Higher dimensional tasks require greater inductive bias.
Machine learning has made major advances in categorizing objects in images, yet the best algorithms miss important aspects of how people learn and think about categories. People can learn richer concepts from fewer examples, including causal models that explain how members of a category are formed. Here, we explore the…
Embedding methods which enforce a partial order or lattice structure over the concept space, such as Order Embeddings (OE) (Vendrov et al., 2016), are a natural way to model transitive relational data (e.g. entailment graphs). However, OE learns a deterministic knowledge base, limiting expressiveness of queries and the…
While physics conveys knowledge of nature built from an interplay between observations and theory, it has been considered less importantly in deep neural networks. Especially, there are few works leveraging physics behaviors when the knowledge is given less explicitly. In this work, we propose a novel architecture call…
Neural networks have been successfully applied in applications with a large amount of labeled data. However, the task of rapid generalization on new concepts with small training data while preserving performances on previously learned ones still presents a significant challenge to neural network models. In this work, w…
The paper explores fundamental limits of learning non-hallucinating generative models.
problem Hallucinations in generative models producing invalid outputs.
method Developed a theoretical framework to analyze learnability from a learning-theoretic perspective, incorporating inductive biases.
result Non-hallucinating learning is statistically impossible without additional inductive biases.
One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
OTI extends OTP for inductive semi-supervised learning.
problem Inductive semi-supervised learning for out-of-sample data.
method Optimal transport-based approach extended to inductive tasks.
result OTI outperforms state-of-the-art methods in experiments.
It is well-known that the LIE(Locally Induction Equation) admit soliton-type solutions and same soliton solutions arise from different and apparently irrelevant physical models. By comparing the solitons of LIE and Killing magnetic geodesics, we observe that these solitons are essentially decided by two families of iso…
DGM learns graph structure for better graph neural network performance.
problem Graphs are often unknown or noisy, limiting graph neural network performance.
method DGM learns graph structure from data, improving performance in transductive and inductive settings.
result DGM achieves state-of-the-art results across various domains.
Strong inductive biases prevent harmless interpolation in overparameterized models.
problem Understanding the conditions under which overparameterized models can interpolate noise without overfitting.
method Theoretical analysis of high-dimensional kernel regression and deep neural networks, focusing on the role of inductive biases.
result The strength of an estimator's inductive bias determines whether interpolation is harmless or requires fitting noise for good generalization.
Deep ResNets favor low bottleneck rank with proper hyperparameters.
problem Understanding the inductive bias of deep neural networks.
method Computed minimum-norm weights of a deep linear ResNet.
result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.
We introduce several methods to define the self-inductance of a single loop as the regularization of divergent integrals which we obtain by applying Neumann (or Weber) formula for the mutual inductance of a pair of loops to the case when two loops are identical.
We introduce the notion of large scale inductive dimension for asymptotic resemblance spaces. We prove that the large scale inductive dimension and the asymptotic dimensiongrad are equal in the class of r-convex metric spaces. This class contains the class of all geodesic metric spaces and all finitely generated groups…
If p:Y→X is an unramified covering map between two compact oriented surfaces of genus at least two, then it is proved that the embedding map, corresponding to p, from the Teichmüller space T(X), for X, to T(Y) actually extends to an embedding between the Thurston compactification of the tw…
New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.
problem Current deep learning models lack transparency and high convergence rates.
method Reconsider neural networks as single-species population dynamics with balanced growth and harvesting rates.
result SGD with balanced growth and harvesting rates outperforms adaptive methods in all three requirements.
Novel framework for Bayesian reinforcement learning infers value function distributions.
problem Bayesian reinforcement learning's challenges in inferring value function distributions.
method Inferential Induction framework for Bayesian reinforcement learning, developing Bayesian Backwards Induction algorithm.
result Proposed algorithm is competitive with state-of-the-art methods.
Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.
problem The impact of noise on interpolating models with strong inductive biases.
method Analyzing linear and classification models with sparse ground truths, proving fast rates for interpolators.
result Strong inductive biases can lead to faster but noisier interpolators, contrary to intuition.
Unsupervised machine translation---i.e., not assuming any cross-lingual supervision signal, whether a dictionary, translations, or comparable corpora---seems impossible, but nevertheless, Lample et al. (2018) recently proposed a fully unsupervised machine translation (MT) model. The model relies heavily on an adversari…
GraIL predicts relations by reasoning over subgraphs, outperforming embeddings.
problem Relation prediction in knowledge graphs using latent representations is limited.
method Graph neural network with inductive bias to learn entity-independent relational semantics.
result GraIL outperforms existing rule-induction baselines in the inductive setting.
New approach relaxes inductive biases of physics-inspired NNs for better performance.
problem Challenges in applying physics-inspired NNs to real-world systems.
method Examined and relaxed inductive biases of Hamiltonian NNs, improving performance on non-conservative systems.
result Improved performance on practical, non-conservative systems by relaxing inductive biases.
Study links neural network inductive bias, feature learning, and generalization on Boolean functions.
problem Understanding how neural networks learn and generalize on Boolean data.
method End-to-end analysis of depth-2 discrete fully connected networks and DNF formulas, using Monte Carlo learning.
result Predictable training dynamics and interpretable features emerge, linking inductive bias and generalization.
Novel approach trains LLMs for inductive reasoning using probabilistic programs.
problem Training LLMs for inductive reasoning with sparse, ambiguous data.
method Program-based Posterior Training (PPT) using probabilistic inference.
result Significant improvement in estimation accuracy and alignment with human judgments.
We prove addition and subspace theorems for asymptotic large inductive dimension. We investigate a transfinite extension of this dimension and show that it is trivial.
Extends positive mass theorem to arbitrary dimensions using a new inductive scheme.
problem Overcoming singularities in the Schoen-Yau proof for arbitrary dimensions.
method Inductive scheme combining shielding principle, conformal blow-up, and Cheeger-Naber bound.
result Proof of positive mass theorem in arbitrary dimensions.
I-BERT extends Transformer's self-attention to arbitrary input lengths.
problem Transformer models struggle with inductive generalization to unseen input lengths.
method Replaces positional encodings with a recurrent layer.
result I-BERT achieves state-of-the-art results on algorithmic tasks.
Transformers learn rich in-context dependencies efficiently.
problem Understanding how transformers learn long-range dependencies efficiently.
method Approximation and dynamics analysis of induction head mechanisms.
result Abrupt transition from lazy to rich mechanisms during training.
Novel approach uses inductive biases for semiconductor etching.
problem Significant violations of physics in etching process predictions.
method Introduced deep learning model with inductive biases.
result Fits measurements faster and follows physical behavior.
Factor complexity bφ(n) for a vertex coloring φ of a regular tree is the number of colored n-balls up to color-preserving automorphisms. Sturmian colorings are colorings of minimal unbounded factor complexity bφ(n)=n+2. In this article, we prove an induction algorithm for Sturmian colorings using colored ba…