New method quantifies inductive bias for machine learning tasks.
problem Quantifying the amount of inductive bias in machine learning models.
method Estimates inductive bias by modeling loss distribution of random hypotheses.
result Higher dimensional tasks require greater inductive bias.
Strong inductive biases prevent harmless interpolation in overparameterized models.
problem Understanding the conditions under which overparameterized models can interpolate noise without overfitting.
method Theoretical analysis of high-dimensional kernel regression and deep neural networks, focusing on the role of inductive biases.
result The strength of an estimator's inductive bias determines whether interpolation is harmless or requires fitting noise for good generalization.
Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.
problem The impact of noise on interpolating models with strong inductive biases.
method Analyzing linear and classification models with sparse ground truths, proving fast rates for interpolators.
result Strong inductive biases can lead to faster but noisier interpolators, contrary to intuition.
Established a new inequality for convex bodies in high dimensions.
problem Bounding the product of dual quermassintegrals of convex bodies and their polar sets.
method Induction on dimensions, focusing on convex bodies in \(\mathbb{R}^{n+1}\).
result Proved a generalised Blaschke-Santalò inequality with upper bounds.
Improved neural networks for relational reasoning by projecting high-dimensional data to low-dimensional manifolds.
problem Out-of-distribution generalization in complex relational reasoning tasks.
method Neuroscience-inspired inductive-biased module projecting high-dimensional object representations to low-dimensional manifolds.
result Significantly better out-of-distribution generalization performance on relational reasoning tasks.
The article consists of the Russian and English variants of Ph.D. Thesis in which the answers is given on the following questions: 1. how to construct the spinor formalism for n=6; 2. how to construct the spinor formalism for n=8; 3. how to prolong the Riemannian connection from the tangent bundle into the spinor one w…
IGMC learns inductive matrix completion without side info.
problem Inductive matrix completion without side information.
method Graph Neural Network (GNN) trained on 1-hop subgraphs of the rating matrix.
result Achieves competitive performance with state-of-the-art transductive baselines.
Study reveals biases and generalization patterns in deep image models.
problem Understanding the inductive bias of deep generative models in high dimensions.
method Proposed a framework to empirically investigate bias and generalization using designed training datasets.
result Identified similarities to human psychology in model behavior and patterns.
New method stabilizes machine learning for physics-informed inverse problems.
problem Reconstructing physical quantities from PDE-compliant measurements.
method Physics-informed learning with smooth inductive bias.
result PDE operators stabilize variance and prevent overfitting in fixed dimensions.
NNLMs optimize poorly for word probabilities due to embedding space structure.
problem NNLMs assign suboptimal probabilities to some words.
method Analyzed the inductive bias of NNLMs and the structure of word embeddings.
result Words on the convex hull have bounded probability, affecting others.
Paper explores how knowledge distillation transfers inductive biases between models.
problem Transferring inductive biases between models for tasks with limited data.
method Knowledge distillation applied to models with different inductive biases (LSTMs vs. Transformers, CNNs vs. MLPs).
result Effect of inductive biases is transferred through knowledge distillation, impacting both performance and solution characteristics.
Interpolated-MLPs control inductive bias for better performance in low-compute tasks.
problem Low-compute performance gap between MLPs and CNNs.
method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.
result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.
Method solves high-dimensional nonlinear PDEs using neural networks.
problem Solving high-dimensional fully nonlinear PDEs.
method Backward induction with multi-layer neural networks to estimate solution and its gradient, with Hessian approximated by automatic differentiation.
result Method extends previous work on semi-linear PDEs to fully nonlinear cases, demonstrating accuracy on various examples.
Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches …
We construct a infinite-dimensional manifold structure adapted to analytic Lie pseudogroups of infinite type. More precisely, we prove that any isotropy subgroup of an analytic Lie pseudogroup of infinite type is a regular infinite-dimensional Lie group, modelled on a locally convex strict inductive limit of Banach spa…
On the basis of Liouville theorem the generalization of the Nambu mechanics is considered. Is shown, that Poisson manifolds of n-dimensional multi-symplectic phase space have inducting by (n-1) Hamiltonian k-vector fields, each of which requires of (k)-hamiltonians.
One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
OTI extends OTP for inductive semi-supervised learning.
problem Inductive semi-supervised learning for out-of-sample data.
method Optimal transport-based approach extended to inductive tasks.
result OTI outperforms state-of-the-art methods in experiments.
Deep convolutional networks can be understood through kernel methods, providing insights into their inductive bias.
problem Understanding the functional space and inductive bias of deep convolutional networks.
method Using kernel methods to analyze simple hierarchical kernels with convolution and pooling layers.
result The RKHS consists of additive models of interaction terms between patches, and pooling layers encourage spatial similarities.
Deep ResNets favor low bottleneck rank with proper hyperparameters.
problem Understanding the inductive bias of deep neural networks.
method Computed minimum-norm weights of a deep linear ResNet.
result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.
This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.
problem Generalization from easy to hard tasks, especially out-of-domain generalization.
method Formalizes inductive generalization, introduces Inductive Learning, and outlines steps to adapt techniques for learning model successors.
result A new learning paradigm (Inductive Learning) that emphasizes induction and universal properties of learning and computation.
We introduce several methods to define the self-inductance of a single loop as the regularization of divergent integrals which we obtain by applying Neumann (or Weber) formula for the mutual inductance of a pair of loops to the case when two loops are identical.
We introduce the notion of large scale inductive dimension for asymptotic resemblance spaces. We prove that the large scale inductive dimension and the asymptotic dimensiongrad are equal in the class of r-convex metric spaces. This class contains the class of all geodesic metric spaces and all finitely generated groups…
CADE learns dual node representations for better generalization.
problem Transductive graph embeddings cannot generalize to unseen nodes or across different graphs.
method CADE combines real-time neighborhoods with neighbor-attentioned representation, preserving known node memory.
result CADE outperforms state-of-the-art methods in generalization and context-awareness.
2-simplicial Transformer enhances logical reasoning in reinforcement learning.
problem Logical reasoning in reinforcement learning.
method Introduces 2-simplicial Transformer with higher-dimensional attention and tensor product updates. result Shows effectiveness of 2-simplicial Transformer for logical reasoning. If p:Y→X is an unramified covering map between two compact oriented surfaces of genus at least two, then it is proved that the embedding map, corresponding to p, from the Teichmüller space T(X), for X, to T(Y) actually extends to an embedding between the Thurston compactification of the tw…
Novel framework for Bayesian reinforcement learning infers value function distributions.
problem Bayesian reinforcement learning's challenges in inferring value function distributions.
method Inferential Induction framework for Bayesian reinforcement learning, developing Bayesian Backwards Induction algorithm.
result Proposed algorithm is competitive with state-of-the-art methods.
Quantum kernels offer potential speed-ups but require encoding problem-specific knowledge.
problem Generalization difficulty in high-dimensional feature spaces.
method Analysis of spectral properties of quantum kernels and their RKHS.
result Quantum advantage is expected if RKHS is low-dimensional and contains hard-to-compute functions.
Unsupervised machine translation---i.e., not assuming any cross-lingual supervision signal, whether a dictionary, translations, or comparable corpora---seems impossible, but nevertheless, Lample et al. (2018) recently proposed a fully unsupervised machine translation (MT) model. The model relies heavily on an adversari…
Matrix SMD converges to unique solution minimizing Bregman divergence.
problem High-dimensional multi-output classification and matrix completion problems.
method Stochastic Mirror Descent with matrix parameters and matrix mirror functions.
result Matrix SMD converges exponentially to the unique solution minimizing Bregman divergence.
GraIL predicts relations by reasoning over subgraphs, outperforming embeddings.
problem Relation prediction in knowledge graphs using latent representations is limited.
method Graph neural network with inductive bias to learn entity-independent relational semantics.
result GraIL outperforms existing rule-induction baselines in the inductive setting.
Theoretical study on how model architecture affects contrastive learning performance.
problem Understanding the role of model architecture in self-supervised learning.
method Theoretical analysis of contrastive learning, focusing on model capacity and clustering structures.
result Contrastive representations have lower dimensionality than the number of clusters in the data distribution.
New approach relaxes inductive biases of physics-inspired NNs for better performance.
problem Challenges in applying physics-inspired NNs to real-world systems.
method Examined and relaxed inductive biases of Hamiltonian NNs, improving performance on non-conservative systems.
result Improved performance on practical, non-conservative systems by relaxing inductive biases.
Counterexamples show no simple generalization of Hasimoto transform for higher-dimensional Euler fluids.
problem No straightforward generalization of Hasimoto transform for higher-dimensional Euler fluids.
method Derivation of evolution equations for mean curvature and torsion form for membranes.
result Existence of counterexamples implies no simple generalization of Hasimoto transform.
Product models of low dimensional experts are a powerful way to avoid the curse of dimensionality. We present the ``under-complete product of experts' (UPoE), where each expert models a one dimensional projection of the data. The UPoE is fully tractable and may be interpreted as a parametric probabilistic model for pro…
Forward regression is a statistical model selection and estimation procedure which inductively selects covariates that add predictive power into a working statistical regression model. Once a model is selected, unknown regression parameters are estimated by least squares. This paper analyzes forward regression in high-…
Study links neural network inductive bias, feature learning, and generalization on Boolean functions.
problem Understanding how neural networks learn and generalize on Boolean data.
method End-to-end analysis of depth-2 discrete fully connected networks and DNF formulas, using Monte Carlo learning.
result Predictable training dynamics and interpretable features emerge, linking inductive bias and generalization.
We study the topological and differentiable singularities of the configuration space C(Γ) of a mechanical linkage Γin d-dimensional Euclidean space, defining an inductive sufficient condition to determine when a configuration is singular. We show that this condition holds for generic singularities, provide a mechanical…
New measure quantifies task difficulty for machine learning models.
problem Quantifying the inherent difficulty of machine learning tasks.
method Inductive bias complexity measure.
result Tasks requiring generalization over many dimensions are more difficult.
Extends program induction for probabilistic programming.
problem Automatic probabilistic program synthesis for diverse data types.
method Further steps to extend previous work on program induction.
result Generalization over various data types (text, image, video).
Novel approach trains LLMs for inductive reasoning using probabilistic programs.
problem Training LLMs for inductive reasoning with sparse, ambiguous data.
method Program-based Posterior Training (PPT) using probabilistic inference.
result Significant improvement in estimation accuracy and alignment with human judgments.
Extends positive mass theorem to arbitrary dimensions using a new inductive scheme.
problem Overcoming singularities in the Schoen-Yau proof for arbitrary dimensions.
method Inductive scheme combining shielding principle, conformal blow-up, and Cheeger-Naber bound.
result Proof of positive mass theorem in arbitrary dimensions.
We prove addition and subspace theorems for asymptotic large inductive dimension. We investigate a transfinite extension of this dimension and show that it is trivial.
I-BERT extends Transformer's self-attention to arbitrary input lengths.
problem Transformer models struggle with inductive generalization to unseen input lengths.
method Replaces positional encodings with a recurrent layer.
result I-BERT achieves state-of-the-art results on algorithmic tasks.
In this paper we investigate the higher dimensional divergence functions of mapping class groups of surfaces and of CAT(0)--groups. We show that, for mapping class groups of surfaces, these functions exhibit phase transitions at the rank (as measured by thrice the genus plus the number of punctures minus 3). We also pr…
The paper presents a new method to represent directed graphs using pseudo-Riemannian manifolds.
problem Representing directed graphs in a compact and meaningful way.
method Combines pseudo-Riemannian metric structure, non-trivial global topology, and a unique likelihood function.
result Low-dimensional cylindrical Minkowski and anti-de Sitter spacetimes produce equal or better graph representations than curved Riemannian manifolds.
Transformers learn rich in-context dependencies efficiently.
problem Understanding how transformers learn long-range dependencies efficiently.
method Approximation and dynamics analysis of induction head mechanisms.
result Abrupt transition from lazy to rich mechanisms during training.
Novel approach uses inductive biases for semiconductor etching.
problem Significant violations of physics in etching process predictions.
method Introduced deep learning model with inductive biases.
result Fits measurements faster and follows physical behavior.