OMIC improves matrix completion with orthonormal side information and nuclear-norm regularization.
problem Matrix completion with improved interpretability and adaptability.
method OMIC combines orthonormal side information and nuclear-norm regularization, optimized by a converging algorithm.
result OMIC outperforms state-of-the-art methods in synthetic and real-world datasets.
OPT framework improves neural network generalization by learning an orthogonal transformation.
problem Improving neural network generalization.
method Orthogonal over-parameterized training (OPT) framework that minimizes hyperspherical energy.
result OPT framework provably minimizes hyperspherical energy and improves empirical generalization.
We prove the existence of a family of embedded doubly periodic minimal surfaces of (quotient) genus g with orthogonal ends that generalizes the classical doubly periodic surface of Scherk and the genus-one Scherk surface of Karcher. The proof of the family of immersed surfaces is by induction on genus, while the proo…
The method of double extension, introduced by A.~Medina and Ph.~Revoy, is a procedure which decomposes a Lie algebra with an invariant symmetric form into elementary pieces. Such decompositions were developed for other algebras, for instance for Lie superalgebras and associative algebras, Filippov n-algebras and Jord…
Paper explores how knowledge distillation transfers inductive biases between models.
problem Transferring inductive biases between models for tasks with limited data.
method Knowledge distillation applied to models with different inductive biases (LSTMs vs. Transformers, CNNs vs. MLPs).
result Effect of inductive biases is transferred through knowledge distillation, impacting both performance and solution characteristics.
Interpolated-MLPs control inductive bias for better performance in low-compute tasks.
problem Low-compute performance gap between MLPs and CNNs.
method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.
result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.
New method quantifies inductive bias for machine learning tasks.
problem Quantifying the amount of inductive bias in machine learning models.
method Estimates inductive bias by modeling loss distribution of random hypotheses.
result Higher dimensional tasks require greater inductive bias.
The BPS decomposition theorem splits cohomology of symmetric stacks into invariant parts.
problem Decomposing the cohomology of smooth symmetric stacks into invariant parts.
method Using cohomological Hall induction and intersection cohomology of moduli spaces.
result Establishes the BPS decomposition theorem for various symplectic stacks.
Convolutional nets require fewer samples than fully-connected nets for image classification.
problem Understanding why convolutional nets are more sample-efficient than fully-connected nets.
method Construction of a natural distribution and target function to demonstrate a sample complexity gap.
result Convolutional nets require O(1) samples for a single target function, while fully-connected nets require Ω(d2) samples. The study examines how shallow neural nets converge to training samples or manifold points during diffusion.
problem Understanding when and how shallow neural nets converge to training samples or manifold points during diffusion.
method Analysis of shallow ReLU neural network denoisers trained with minimal ℓ2 norm, comparing score flow and diffusion flow. result Probability flow converges to training points, sums of training points, or manifold points, depending on the diffusion time scheduler.
One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
OTI extends OTP for inductive semi-supervised learning.
problem Inductive semi-supervised learning for out-of-sample data.
method Optimal transport-based approach extended to inductive tasks.
result OTI outperforms state-of-the-art methods in experiments.
New ODD metrics defined on manifolds with degeneracy conditions.
problem Defining metrics on manifolds with degeneracy conditions.
method Introducing ODD metrics that degenerate on submanifolds while maintaining compatibility.
result ODD metrics satisfy basic properties and induce metric space structures.
Strong inductive biases prevent harmless interpolation in overparameterized models.
problem Understanding the conditions under which overparameterized models can interpolate noise without overfitting.
method Theoretical analysis of high-dimensional kernel regression and deep neural networks, focusing on the role of inductive biases.
result The strength of an estimator's inductive bias determines whether interpolation is harmless or requires fitting noise for good generalization.
Deep ResNets favor low bottleneck rank with proper hyperparameters.
problem Understanding the inductive bias of deep neural networks.
method Computed minimum-norm weights of a deep linear ResNet.
result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.
This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.
problem Generalization from easy to hard tasks, especially out-of-domain generalization.
method Formalizes inductive generalization, introduces Inductive Learning, and outlines steps to adapt techniques for learning model successors.
result A new learning paradigm (Inductive Learning) that emphasizes induction and universal properties of learning and computation.
We introduce several methods to define the self-inductance of a single loop as the regularization of divergent integrals which we obtain by applying Neumann (or Weber) formula for the mutual inductance of a pair of loops to the case when two loops are identical.
We introduce the notion of large scale inductive dimension for asymptotic resemblance spaces. We prove that the large scale inductive dimension and the asymptotic dimensiongrad are equal in the class of r-convex metric spaces. This class contains the class of all geodesic metric spaces and all finitely generated groups…
If p:Y→X is an unramified covering map between two compact oriented surfaces of genus at least two, then it is proved that the embedding map, corresponding to p, from the Teichmüller space T(X), for X, to T(Y) actually extends to an embedding between the Thurston compactification of the tw…
The dominant paradigm for relation prediction in knowledge graphs involves learning and operating on latent representations (i.e., embeddings) of entities and relations. However, these embedding-based methods do not explicitly capture the compositional logical rules underlying the knowledge graph, and they are limited …
Novel framework for Bayesian reinforcement learning infers value function distributions.
problem Bayesian reinforcement learning's challenges in inferring value function distributions.
method Inferential Induction framework for Bayesian reinforcement learning, developing Bayesian Backwards Induction algorithm.
result Proposed algorithm is competitive with state-of-the-art methods.
Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.
problem The impact of noise on interpolating models with strong inductive biases.
method Analyzing linear and classification models with sparse ground truths, proving fast rates for interpolators.
result Strong inductive biases can lead to faster but noisier interpolators, contrary to intuition.
Unsupervised machine translation---i.e., not assuming any cross-lingual supervision signal, whether a dictionary, translations, or comparable corpora---seems impossible, but nevertheless, Lample et al. (2018) recently proposed a fully unsupervised machine translation (MT) model. The model relies heavily on an adversari…
Free Random Projection enhances reinforcement learning by naturally incorporating hierarchical structure.
problem Improving reinforcement learning algorithms for better generalization and adaptability.
method Introduces Free Random Projection, a method that uses free probability theory to create random orthogonal matrices encoding hierarchical structure.
result Empirically shows consistent improvement in generalization over standard methods on multi-environment benchmarks.
New approach relaxes inductive biases of physics-inspired NNs for better performance.
problem Challenges in applying physics-inspired NNs to real-world systems.
method Examined and relaxed inductive biases of Hamiltonian NNs, improving performance on non-conservative systems.
result Improved performance on practical, non-conservative systems by relaxing inductive biases.
The recent advent of automated neural network architecture search led to several methods that outperform state-of-the-art human-designed architectures. However, these approaches are computationally expensive, in extreme cases consuming GPU years. We propose two novel methods which aim to expedite this optimization prob…
Study links neural network inductive bias, feature learning, and generalization on Boolean functions.
problem Understanding how neural networks learn and generalize on Boolean data.
method End-to-end analysis of depth-2 discrete fully connected networks and DNF formulas, using Monte Carlo learning.
result Predictable training dynamics and interpretable features emerge, linking inductive bias and generalization.
Novel approach trains LLMs for inductive reasoning using probabilistic programs.
problem Training LLMs for inductive reasoning with sparse, ambiguous data.
method Program-based Posterior Training (PPT) using probabilistic inference.
result Significant improvement in estimation accuracy and alignment with human judgments.
We prove addition and subspace theorems for asymptotic large inductive dimension. We investigate a transfinite extension of this dimension and show that it is trivial.
Extends positive mass theorem to arbitrary dimensions using a new inductive scheme.
problem Overcoming singularities in the Schoen-Yau proof for arbitrary dimensions.
method Inductive scheme combining shielding principle, conformal blow-up, and Cheeger-Naber bound.
result Proof of positive mass theorem in arbitrary dimensions.
I-BERT extends Transformer's self-attention to arbitrary input lengths.
problem Transformer models struggle with inductive generalization to unseen input lengths.
method Replaces positional encodings with a recurrent layer.
result I-BERT achieves state-of-the-art results on algorithmic tasks.
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language underst…
Transformers learn rich in-context dependencies efficiently.
problem Understanding how transformers learn long-range dependencies efficiently.
method Approximation and dynamics analysis of induction head mechanisms.
result Abrupt transition from lazy to rich mechanisms during training.
Novel approach uses inductive biases for semiconductor etching.
problem Significant violations of physics in etching process predictions.
method Introduced deep learning model with inductive biases.
result Fits measurements faster and follows physical behavior.
Factor complexity bφ(n) for a vertex coloring φ of a regular tree is the number of colored n-balls up to color-preserving automorphisms. Sturmian colorings are colorings of minimal unbounded factor complexity bφ(n)=n+2. In this article, we prove an induction algorithm for Sturmian colorings using colored ba…
Integrates inductive biases into VAEs using intermediary latent variables.
problem Ineffective mechanisms for incorporating inductive biases into VAEs.
method InteL-VAEs use an intermediary latent space to control encoding, with a parametric function to enforce desired properties.
result InteL-VAEs lead to better generative models and representations.
This paper explains the theoretical inductive bias of Isolation Forest.
problem Lack of theoretical foundation explaining Isolation Forest's success.
method Formulated the growth process of iForest as a random walk, derived expected depth function using transition probabilities.
result Established a theoretical understanding of iForest's effectiveness and parameter adaptability.
R2N learns interpretable rules and literals from numerical features.
problem Lack of expressive vocabulary in rule-based decision models.
method Relational Rule Network (R2N) learns literals and rules end-to-end.
result Learned literals improve prediction accuracy and rule conciseness.
Machine learning refactors knowledge to improve learning efficiency.
problem Inductive program synthesis efficiency through knowledge restructuring.
method Introduces Knorf, a system that refactors knowledge bases using constraint optimization.
result Learning from refactored knowledge improves predictive accuracy fourfold and reduces learning time by half.
We define and study the Burnside quotient Green ring of a Mackey functor. Some refinements of Dress induction theory are presented, together with applications to computation results for K-theory and L-theory of finite and infinite groups.
The study extends Jacobi-orthogonality to indefinite scalar product spaces.
problem Generalizing Jacobi-orthogonality to indefinite scalar product spaces.
method Comparing principles, investigating tensor relations, proving properties.
result Every quasi-Clifford tensor is Jacobi-orthogonal; certain tensors are Jacobi-dual or Osserman.
New characterization of Osserman tensors using Jacobi-orthogonality.
problem Characterizing Osserman tensors.
method Introducing Jacobi-orthogonality as a new potential characterization.
result Jacobi-orthogonal tensors are Osserman, and all known Osserman tensors are Jacobi-orthogonal.
Transformers learn to use induction heads or shortcuts based on data diversity.
problem How data diversity influences the behavior of transformers.
method Gradient-based training of a single-layer transformer on a minimal task.
result Data diversity steers transformers toward induction heads or shortcuts.
We apply Murasugi-Tristram inequality to real algebraic curves of odd degree on RP2 with a deep nest, i.e. a nest of the depth k−1 where 2k+1 is the degree. For such curves, the ingredients of the Murasugi-Tristram inequality can be computed (or estimated) inductively using the computations for iterated torus li…
We propose a Poisson-Lie analog of the symplectic induction procedure, using an appropriate Poisson generalization of the reduction of symplectic manifolds with symmetry. Having as basic tools the equivariant momentum maps of Poisson actions, the double group of a Poisson-Lie group and the reduction of Poisson manifold…
We propose an inductive matrix completion model without using side information. By factorizing the (rating) matrix into the product of low-dimensional latent embeddings of rows (users) and columns (items), a majority of existing matrix completion methods are transductive, since the learned embeddings cannot generalize …
Study isotropy groups for complex orthogonal and skew-symmetric matrices.
problem Understanding isotropy subgroups of orthogonal similarity transformations.
method Analysis of group structure of nonsingular block matrices.
result Group structure of isotropy subgroups related to block Toeplitz matrices.
Proposes a topological framework to study modular invariants and related concepts.
problem Exploring modular invariants and related concepts in topological quantum field theory.
method Topological paradigm in alterfold topological quantum field theory.
result Establishes a novel integral identity for modular invariance across multiple Morita contexts.