arXiv research

Paper explores how knowledge distillation transfers inductive biases between models.

PAC-Net prunes deep models for better transfer learning.

This work presents an approach to automatically induction for non-greedy decision trees constructed from neural network architecture. This construction can be used to transfer weights when growing or pruning a decision tree, allowing non-greedy decision tree algorithms to automatically learn and adapt to the ideal arch…

Improves transferability of representations from source to target domains with weights and invariant representations.

An important goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning. In particular…

IGNNK uses GNN for spatiotemporal kriging, improving scalability and transferability.

New model detects Alzheimer's and severity from speech, cognitive, and language data.

Learning transferable knowledge across similar but different settings is a fundamental component of generalized intelligence. In this paper, we approach the transfer learning challenge from a causal theory perspective. Our agent is endowed with two basic yet general theories for transfer learning: (i) a task shares a c…

We propose an inductive matrix completion model without using side information. By factorizing the (rating) matrix into the product of low-dimensional latent embeddings of rows (users) and columns (items), a majority of existing matrix completion methods are transductive, since the learned embeddings cannot generalize …

Linearized neural networks provide a fast and interpretable way to adapt models to new settings.

The focus in machine learning has branched beyond training classifiers on a single task to investigating how previously acquired knowledge in a source domain can be leveraged to facilitate learning in a related target domain, known as inductive transfer learning. Three active lines of research have independently explor…

Transfer learning borrows knowledge from a source domain to facilitate learning in a target domain. Two primary issues to be addressed in transfer learning are what and how to transfer. For a pair of domains, adopting different transfer learning algorithms results in different knowledge transferred between them. To dis…

Novel approach trains LLMs for inductive reasoning using probabilistic programs.

Paper introduces a new method for improving reinforcement learning performance using transfer learning.

Optimal transport metric transfers deep network representations efficiently.

Learning-to-learn or meta-learning leverages data-driven inductive bias to increase the efficiency of learning on a novel task. This approach encounters difficulty when transfer is not advantageous, for instance, when tasks are considerably dissimilar or change over time. We use the connection between gradient-based me…

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce tech…

The successful application of general reinforcement learning algorithms to real-world robotics applications is often limited by their high data requirements. We introduce Regularized Hierarchical Policy Optimization (RHPO) to improve data-efficiency for domains with multiple dominant tasks and ultimately reduce require…

As reinforcement learning agents are tasked with solving more challenging and diverse tasks, the ability to incorporate prior knowledge into the learning system and to exploit reusable structure in solution space is likely to become increasingly important. The KL-regularized expected reward objective constitutes one po…

Meta-learning improves few-shot land cover classification across diverse regions.

New research suggests privileged information doesn't improve model performance.

Agent learns to trade currency pairs with improved risk management.

This paper extends transfer learning for linear regression to uncertain domain information.

Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision. The typical way of conducting transfer learning with deep neural networks is to fine-tune a model pre-trained on the source task using data from the target task. In this paper, we propose an…

This work extends GNNs to handle multiple graphs with non-commuting operators, proving transferability.

MuJAM learns traffic signal control policies that generalize to unseen intersections and traffic conditions.

In this work we present ISA, a novel approach for learning and exploiting subgoals in reinforcement learning (RL). Our method relies on inducing an automaton whose transitions are subgoals expressed as propositional formulas over a set of observable events. A state-of-the-art inductive logic programming system is used …

Universal Language Model for Fine-tuning [arXiv:1801.06146] (ULMFiT) is one of the first NLP methods for efficient inductive transfer learning. Unsupervised pretraining results in improvements on many NLP tasks for English. In this paper, we describe a new method that uses subword tokenization to adapt ULMFiT to langua…

IG-RL learns adaptive traffic signals for any network, outperforming existing methods.

Learning meaningful and compact representations with disentangled semantic aspects is considered to be of key importance in representation learning. Since real-world data is notoriously costly to collect, many recent state-of-the-art disentanglement models have heavily relied on synthetic toy data-sets. In this paper, …

The recent advent of automated neural network architecture search led to several methods that outperform state-of-the-art human-designed architectures. However, these approaches are computationally expensive, in extreme cases consuming GPU years. We propose two novel methods which aim to expedite this optimization prob…

Given the return series for a set of instruments, a \emph{trading strategy} is a switching function that transfers wealth from one instrument to another at specified times. We present efficient algorithms for constructing (ex-post) trading strategies that are optimal with respect to the total return, the Sterling ratio…

Metalearned neural circuit performs inference over open classes.

The paper bridges spectral and spatial graph convolutions, improving model capacity and transferability.

A new model decouples global and local image representations without supervision.

Paper refines cross-lingual word embeddings using Manhattan norm.

Interpolated-MLPs control inductive bias for better performance in low-compute tasks.

New method quantifies inductive bias for machine learning tasks.

One-layer transformers can't solve induction heads task efficiently.

OTI extends OTP for inductive semi-supervised learning.

Strong inductive biases prevent harmless interpolation in overparameterized models.

Deep ResNets favor low bottleneck rank with proper hyperparameters.

This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.

We introduce several methods to define the self-inductance of a single loop as the regularization of divergent integrals which we obtain by applying Neumann (or Weber) formula for the mutual inductance of a pair of loops to the case when two loops are identical.

We introduce the notion of large scale inductive dimension for asymptotic resemblance spaces. We prove that the large scale inductive dimension and the asymptotic dimensiongrad are equal in the class of r-convex metric spaces. This class contains the class of all geodesic metric spaces and all finitely generated groups…

If $p : Y \to X$ is an unramified covering map between two compact oriented surfaces of genus at least two, then it is proved that the embedding map, corresponding to $p$, from the Teichmüller space ${\cal T}(X)$, for $X$, to ${\cal T}(Y)$ actually extends to an embedding between the Thurston compactification of the tw…

The dominant paradigm for relation prediction in knowledge graphs involves learning and operating on latent representations (i.e., embeddings) of entities and relations. However, these embedding-based methods do not explicitly capture the compositional logical rules underlying the knowledge graph, and they are limited …

Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.