Theoretical analysis of CNNs' inductive biases and their efficiency in approximating functions.
problem Understanding and optimizing the inductive biases in deep CNNs.
method Theoretical analysis combining multichanneling, downsampling, weight sharing, and locality.
result Deep CNNs with O ( log d ) \mathcal{O}(\log d) O ( log d ) depth can approximate any continuous function, and require O ~ ( log 2 d ) \widetilde{\mathcal{O}}(\log^2d) O ( log 2 d ) samples for sparse functions. Counterexamples show no simple generalization of Hasimoto transform for higher-dimensional Euler fluids.
problem No straightforward generalization of Hasimoto transform for higher-dimensional Euler fluids.
method Derivation of evolution equations for mean curvature and torsion form for membranes.
result Existence of counterexamples implies no simple generalization of Hasimoto transform.
New diagnostic tool for assessing approximate Bayesian inference.
problem Assessing the trustworthiness of approximate Bayesian inference.
method Reframe the problem in terms of incompatible conditional distributions and use Gibbs priors.
result The diagnostic tool can discover the inductive bias in various Bayesian models and approximations.
Transformers learn rich in-context dependencies efficiently.
problem Understanding how transformers learn long-range dependencies efficiently.
method Approximation and dynamics analysis of induction head mechanisms.
result Abrupt transition from lazy to rich mechanisms during training.
Novel framework for Bayesian reinforcement learning infers value function distributions.
problem Bayesian reinforcement learning's challenges in inferring value function distributions.
method Inferential Induction framework for Bayesian reinforcement learning, developing Bayesian Backwards Induction algorithm.
result Proposed algorithm is competitive with state-of-the-art methods.
Local nonparametric meta-learning improves meta-generalization across tasks.
problem Meta-learning struggles with global inductive biases and out-of-distribution tasks.
method Proposes a local, nonparametric meta-learning algorithm using meta-trained local learning rules.
result Improved meta-generalization and state-of-the-art results in robotics benchmarks.
New method quantifies inductive bias for machine learning tasks.
problem Quantifying the amount of inductive bias in machine learning models.
method Estimates inductive bias by modeling loss distribution of random hypotheses.
result Higher dimensional tasks require greater inductive bias.
The local induction equation, or the binormal flow on space curves is a well-known model of deformation of space curves as it describes the dynamics of vortex filaments, and the complex curvature is governed by the nonlinear Schrödinger equation. In this paper, we present its discrete analogue, namely, a model of defor…
A well known result of Da Rios and Levi-Civita says that a closed planar curve is elastic if and only if it is stationary under the localized induction (or smoke ring) equation, where stationary means that the evolution under the localized induction equation is by rigid motions. We prove an analogous result for surface…
GraIL predicts relations by reasoning over subgraphs, outperforming embeddings.
problem Relation prediction in knowledge graphs using latent representations is limited.
method Graph neural network with inductive bias to learn entity-independent relational semantics.
result GraIL outperforms existing rule-induction baselines in the inductive setting.
DRUM discovers interpretable rules from knowledge graphs for unseen entities.
problem Inductive link prediction on unseen entities and lack of interpretability.
method Differentiable approach using bidirectional RNNs for low-rank tensor approximation.
result DRUM outperforms existing methods in inductive link prediction.
New method for embedding large networks without attributes, achieving state-of-the-art performance.
problem Learning embeddings from large-scale networks without domain-dependent attributes.
method Use predefined local encodings based on node degree frequencies at different distances.
result Inductive network embeddings generalize well across unseen or distant regions in the network.
Deep convolutional networks can be understood through kernel methods, providing insights into their inductive bias.
problem Understanding the functional space and inductive bias of deep convolutional networks.
method Using kernel methods to analyze simple hierarchical kernels with convolution and pooling layers.
result The RKHS consists of additive models of interaction terms between patches, and pooling layers encourage spatial similarities.
Novel approach trains LLMs for inductive reasoning using probabilistic programs.
problem Training LLMs for inductive reasoning with sparse, ambiguous data.
method Program-based Posterior Training (PPT) using probabilistic inference.
result Significant improvement in estimation accuracy and alignment with human judgments.
We present a Hamiltonian framework for higher-dimensional vortex filaments (or membranes) and vortex sheets as singular 2-forms with support of codimensions 2 and 1, respectively, i.e. singular elements of the dual to the Lie algebra of divergence-free vector fields. It turns out that the localized induction approximat…
IGMC learns inductive matrix completion without side info.
problem Inductive matrix completion without side information.
method Graph Neural Network (GNN) trained on 1-hop subgraphs of the rating matrix.
result Achieves competitive performance with state-of-the-art transductive baselines.
Study generalizes matrix completion with side info in low noise settings.
problem Matrix completion with side information in low noise conditions.
method Inductive matrix completion with i.i.d. subgaussian noise, uniform sampling, and side information.
result Generalization bounds with noise scaling, convergence to zero, and logarithmic dependence on matrix size.
Study the inductive bias of neural networks using neural tangent kernels.
problem Understanding the generalization properties of over-parameterized neural networks.
method Analysis of the neural tangent kernel and its corresponding function space (RKHS).
result Stability properties of functions with finite norm, including stability to image deformations in convolutional networks.
New method relaxes spatial invariance in locally connected layers, improving accuracy.
problem Improving classification accuracy with locally connected layers.
method Designing a low-rank locally connected layer with varying spatially varying combining weights.
result Relaxing spatial invariance improves classification accuracy over convolution and locally connected layers.
NLM combines neural networks and logic programming for complex reasoning.
problem Complex reasoning tasks involving logic and properties.
method Neural-symbolic architecture combining neural networks and logic programming.
result NLM achieves perfect generalization on various tasks.
New model generates images by reversing heat equation, revealing disentanglement.
problem Image generation without considering image structure.
method Stochastically reverses the heat equation to generate images, using variational approximation.
result Emergent disentanglement of overall colour and shape in images.
A new method uses algebraic insights to create approximately equivariant networks without complex architectures.
problem Designing equivariant neural networks with complex architectures and high computational cost.
method Imposes the group's regular representation as an inductive bias via an auxiliary loss, adding no learnable parameters.
result Matches or outperforms specialized models in several cases, even for infinite groups.
Algorithm extracts non-monotonic rules from statistical models using HUIM.
problem Extracting non-monotonic rules from statistical learning models.
method Reduces problem to HUIM, uses TreeExplainer for feature importance.
result Significant improvement in classification metrics and training time.
Our interest in this paper is in the construction of symbolic explanations for predictions made by a deep neural network. We will focus attention on deep relational machines (DRMs, first proposed by H. Lodhi). A DRM is a deep network in which the input layer consists of Boolean-valued functions (features) that are defi…
ConViT combines CNN and ViT strengths, improving image classification.
problem Combining the strengths of CNNs and ViTs while avoiding their limitations.
method Introducing GPSA, a form of positional self-attention with a soft convolutional inductive bias.
result ConViT outperforms DeiT on ImageNet while offering improved sample efficiency.
This paper improves sample efficiency in noisy inductive matrix completion with side-information.
problem Improving sample efficiency in noisy inductive matrix completion with side-information.
method Nonconvex projected gradient descent algorithm with spectral initialization.
result Achieves linear convergence and stable recovery at a sample complexity governed by the effective side-information dimension.
Proposes a neural framework to select subsets efficiently across different models.
problem Lack of generalizability in subset selection methods for unseen architectures.
method Introduces a trainable subset selection framework, SubSelNet, that uses attention-based neural gadgets and subset samplers.
result SubSelNet generalizes across architectures and outperforms existing methods.
Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches …
New methods for CI testing under model misspecification.
problem Challenges in CI testing with misspecified models.
method Proposes new approximations and upper bounds for testing errors of regression-based CI tests.
result Introduces the Rao-Blackwellized Predictor Test (RBPT) robust against misspecified inductive biases.
DNF-Net tackles tabular data challenges with neural architecture.
problem Handling tabular data efficiently using neural networks.
method DNF-Net uses a neural architecture with inductive bias corresponding to logical Boolean formulas in disjunctive normal form over affine soft-threshold decision terms.
result DNF-Net significantly outperforms fully connected networks on tabular data.
GraphReach improves GNN performance by incorporating node positions.
problem Existing GNNs fail to capture node positions, leading to inaccurate predictions.
method GraphReach uses reachability estimations from anchor nodes to capture global node positions.
result GraphReach achieves up to 40% relative improvement in accuracy compared to state-of-the-art GNNs.
New method TLC improves transductive learning bounds.
problem Sharp generalization bounds for transductive learning.
method Transductive Local Complexity (TLC) framework.
result Nearly sharp bounds consistent with inductive results.
Stacked conformal prediction simplifies model validation.
problem Validating stacked predictive models efficiently.
method Meta-learner at the top of a stacked ensemble for approximate marginal validity.
result The method achieves approximate marginal validity without a separate calibration sample.
Paper analyzes Bezier simplex fitting risks and optimal sampling.
problem Analyzing risks and optimal sampling for Bezier simplex fitting.
method Two fitting methods: inductive skeleton and all-at-once.
result Optimal subsample ratio for inductive skeleton fitting reduces risk.
RPPs improve deep learning models with soft equivariance constraints.
problem Balancing expressiveness and inductive biases in deep learning.
method Introducing Residual Pathway Priors (RPPs) to convert hard constraints into soft priors.
result RPPs enable models to learn structured solutions while retaining flexibility.
We propose a new framework for modeling stochastic local volatility, with potential applications to modeling derivatives on interest rates, commodities, credit, equity, FX etc., as well as hybrid derivatives. Our model extends the linearity-generating unspanned volatility term structure model by Carr et al. (2011) by a…
Paper explores how knowledge distillation transfers inductive biases between models.
problem Transferring inductive biases between models for tasks with limited data.
method Knowledge distillation applied to models with different inductive biases (LSTMs vs. Transformers, CNNs vs. MLPs).
result Effect of inductive biases is transferred through knowledge distillation, impacting both performance and solution characteristics.
Interpolated-MLPs control inductive bias for better performance in low-compute tasks.
problem Low-compute performance gap between MLPs and CNNs.
method Introduced Interpolated MLP (I-MLP) approach to control inductive bias incrementally.
result Continuous logarithmic relationship between inductive bias and performance in low-compute tasks.
Given the Euclidean space R 2 n + 2 \R^{2n+2} R 2 n + 2 endowed with a constant symplectic structure and the standard flat connection, and given a polynomial of degree 2 on that space, Baguis and Cahen have defined a reduction procedure which yields a symplectic manifold endowed with a Ricci-type connection. We observe that any symplect…
Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.
problem Understanding the impact of model parametrization on optimization and generalization in deep learning.
method Characterization of natural gradient flow in deep linear networks and nonlinear neural networks.
result Natural gradient descent fails to generalize in some cases, while gradient descent with the right architecture performs well.
Linearized neural networks provide a fast and interpretable way to adapt models to new settings.
problem Difficulty in understanding and adapting inductive biases of trained neural networks.
method Linearization of neural networks and embedding these biases into Gaussian processes through a kernel designed from the Jacobian.
result Domain adaptation becomes interpretable posterior inference with analytic and scalable computational speed-ups.
Method solves high-dimensional nonlinear PDEs using neural networks.
problem Solving high-dimensional fully nonlinear PDEs.
method Backward induction with multi-layer neural networks to estimate solution and its gradient, with Hessian approximated by automatic differentiation.
result Method extends previous work on semi-linear PDEs to fully nonlinear cases, demonstrating accuracy on various examples.
One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
Unified framework combines trace-induced quantum kernels for improved machine learning models.
problem Improving performance of quantum machine learning models using trace-induced kernels.
method Developed a unified framework combining various trace-induced quantum kernels, including global fidelity and local projected kernels, as Lego kernels.
result Local projected kernels can achieve comparable performance to global fidelity kernels with fewer quantum resources.
Recent work in model-agnostic explanations of black-box machine learning has demonstrated that interpretability of complex models does not have to come at the cost of accuracy or model flexibility. However, it is not clear what kind of explanations, such as linear models, decision trees, and rule lists, are the appropr…
OTI extends OTP for inductive semi-supervised learning.
problem Inductive semi-supervised learning for out-of-sample data.
method Optimal transport-based approach extended to inductive tasks.
result OTI outperforms state-of-the-art methods in experiments.
TabPFN learns to approximate functions on tabular data.
problem TabPFN tackles function approximation on tabular data.
method Treated as a black-box function approximator generator, observed behavior on varied datasets.
result Observed behavior that is both brilliant and baffling.
Pruning FCNs reveals sub-networks that match CNNs' performance.
problem Understanding the inductive bias of pruning in neural networks.
method Iterative magnitude pruning of a simple FCN followed by analysis of the resulting architecture.
result Pruned FCNs exhibit key features of CNNs, suggesting new architectural biases.