FAST improves fast and stable task adaptation in DNNs.
problem Catastrophic forgetting in fine-tuned pretrained models.
method Introducing FAST, an easy-to-implement fine-tuning algorithm.
result FAST learns target tasks faster and retains source knowledge longer.
DANCE improves prediction set efficiency for deep learning models.
problem Inefficient, overly conservative prediction sets for pre-trained models.
method DANCE combines adaptive kernel regression and nearest-neighbor approach.
result DANCE produces more efficient and robust prediction sets.
A new method prunes neural networks faster and more efficiently.
problem Reducing training time and memory usage for neural networks.
method Set-based Task-Adaptive Meta Pruning (STAMP) that meta-learns a pruning mask.
result Significantly improved compression rates and faster training speed.
Handling previously unseen tasks after given only a few training examples continues to be a tough challenge in machine learning. We propose TapNets, neural networks augmented with task-adaptive projection for improved few-shot learning. Here, employing a meta-learning strategy with episode-based training, a network and…
While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, some issues remain to be tackled to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number o…
Task-adaptive clustering improves few-shot learning with unlabeled data.
problem Handling unseen tasks with limited labeled data.
method Task-conditioned clustering in a new projection space.
result State-of-the-art semi-supervised few-shot classification performance.
Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNe…
A new meta-learning BO approach that bypasses surrogate models and directly learns task utility.
problem Scalability issues and sensitivity to task similarity in existing meta-learning BO methods.
method Directly learns the utility of queries across tasks, models task uncertainty, and includes an auxiliary model for robust adaptation.
result Demonstrates strong anytime performance and outperforms state-of-the-art methods in various benchmarks.
ACL improves DRL by adapting task difficulty to agent's capabilities.
problem Improving sample efficiency and performance in DRL.
method Adapting task difficulty based on agent's capacity.
result Enhanced sample efficiency and asymptotic performance.
AvaGrad optimizes vision tasks by decoupling learning rate and adaptability.
problem Improving optimization methods for vision tasks.
method Derives AvaGrad, a new optimizer that decouples learning rate and adaptability.
result AvaGrad outperforms SGD on vision tasks when adaptability is properly tuned.
We show that several popular few-shot learning benchmarks can be solved with varying degrees of success without using support set Labels at Test-time (LT). To this end, we introduce a new baseline called Centroid Networks, a modification of Prototypical Networks in which the support set labels are hidden from the metho…
We propose a new architecture called Memory-Augmented Encoder-Solver (MAES) that enables transfer learning to solve complex working memory tasks adapted from cognitive psychology. It uses dual recurrent neural network controllers, inside the encoder and solver, respectively, that interface with a shared memory module a…
How can deep learning systems flexibly reuse their knowledge? Toward this goal, we propose a new class of challenges, and a class of architectures that can solve them. The challenges are meta-mappings, which involve systematically transforming task behaviors to adapt to new tasks zero-shot. The key to achieving these c…
Adversarial weighting improves regression task adaptation.
problem Improving regression performance across domains with covariate shift.
method Adversarial network algorithm for instance weighting and task learning.
result The method enhances regression accuracy on target domains.
Online learning with streaming data in a distributed and collaborative manner can be useful in a wide range of applications. This topic has been receiving considerable attention in recent years with emphasis on both single-task and multitask scenarios. In single-task adaptation, agents cooperate to track an objective o…
TAAN model learns optimal network architecture for MTL tasks.
problem Improving generalization performance of MTL by finding flexible and accurate shared architecture.
method TAAN model with flexible activation functions and functional regularization.
result TAAN and regularization methods improve MTL performance.
Improves meta-learning efficiency with mixed-mode differentiation.
problem Efficiently calculating complex derivatives in meta-learning.
method Mixed-Flow Meta-Gradients (MixFlow-MG) for scalable differentiation.
result Significant memory and time improvements in meta-learning tasks.
Deep neural networks are traditionally trained using human-designed stochastic optimization algorithms, such as SGD and Adam. Recently, the approach of learning to optimize network parameters has emerged as a promising research topic. However, these learned black-box optimizers sometimes do not fully utilize the experi…
New method improves meta-learning performance by task-specific initialization.
problem Difficulties in generalizing and achieving theoretical guarantees in conditional meta-learning.
method Structured prediction approach for task-specific initialization.
result TASML improves performance of existing meta-learning models.
Algorithm improves sample efficiency for dynamic task adaptation.
problem Sample inefficiency in imitation learning for changing task distributions.
method Assigns importance weights to past demonstrations for meta adaptation.
result Robot adapts to unseen environments with few demonstrations.
New method optimizes MRI sampling patterns for faster scans.
problem Accelerate MRI scans without sacrificing image quality.
method Joint learning of adaptive sampling patterns and model-based recovery.
result Improved MR image quality compared to other methods.
Few-shot learning is a challenging problem where the goal is to achieve generalization from only few examples. Model-agnostic meta-learning (MAML) tackles the problem by formulating prior knowledge as a common initialization across tasks, which is then used to quickly adapt to unseen tasks. However, forcibly sharing an…
Plan2Explore learns new tasks efficiently through self-supervised planning.
problem Challenges in reinforcement learning, especially task-specific learning and sample efficiency.
method Self-supervised exploration and fast adaptation to new tasks through efficient planning.
result Plan2Explore outperforms prior methods in learning new tasks without supervision.
New method encodes function preferences into neural nets for better generalization.
problem Challenges in encoding explicit function preferences in neural network training.
method Function-space empirical Bayes (FSEB) regularization.
result FSEB leads to near-perfect semantic shift detection and improved generalization.
This paper tackles continuous domain adaptation with a new approach.
problem Learning in non-stationary environments, especially domain drift.
method Variational domain-agnostic feature replay, composed of inference, generative, and solver modules.
result Demonstrates the effectiveness of the proposed approach for practical usage.
AdaPTS adapts univariate FMs for multivariate time series forecasting.
problem Challenges in managing feature dependencies and uncertainty quantification in multivariate time series forecasting.
method Adapters that transform multivariate inputs into a latent space and apply univariate FMs independently to each dimension.
result AdaPTS enhances forecasting accuracy and uncertainty quantification compared to baseline methods.
The paper quantifies uncertainty in aggregated machine learning metrics.
problem Uncertainty in summarizing model performance across multiple tasks.
method Statistical methodologies including bootstrapping and Bayesian modeling.
result Insights into model performance dominance for specific tasks.
While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that the number of instances per task and class is fixed. Due to such restriction, they learn to equally utilize the meta-knowledge across all the tasks…
Meta-learning has received a tremendous recent attention as a possible approach for mimicking human intelligence, i.e., acquiring new knowledge and skills with little or even no demonstration. Most of the existing meta-learning methods are proposed to tackle few-shot learning problems such as image and text, in rather …
Novel regularization for Vision Transformers improves model generalization and sparsity.
problem Improving generalization and sparsity in Vision Transformers.
method Likelihood-guided variational Ising-based regularization.
result Improved generalization and sparsity in Vision Transformers.
Paper presents a Bayesian-decision-theory framework for long-tailed classification.
problem Heavy imbalance and asymmetric misprediction costs in long-tailed datasets.
method Bayesian-decision-theory perspective, unifying re-balancing and ensemble methods.
result Improves accuracy for all classes, especially tails, with provably optimal decisions.
Adaptive kernels from neural networks improve model performance.
problem Improving neural network performance through adaptive kernels.
method Deriving adaptive kernels from infinite-width neural networks using feature learning and gradient flow training.
result Adaptive kernels achieve lower test loss compared to traditional kernels.
MAGDiff detects data shifts in neural networks without retraining.
problem Neural networks' sensitivity to data distribution shifts.
method Extracts MAGDiff representations from neural networks to detect shifts.
result MAGDiff representations improve data set shift detection.
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
A tensor model for meta-learning adapts to task-specific features.
problem Learning shared representations for diverse tasks without task-specific observable information.
method Modeling meta-parameters as an order-3 tensor, estimating through tensor regression and method of moments.
result Tensor-based approach improves meta-learning performance with fewer samples.
Gradient-based meta-RL fails with incorrect task distributions, leading to instability and poor performance.
problem Gradient-based meta-RL's sensitivity to task distributions causes instability and poor performance.
method Proposes meta Active Domain Randomization (meta-ADR) to learn task distributions for gradient-based meta-RL.
result Meta-ADR improves stability and generalization of MAML on simulated locomotion and navigation tasks.
While domain adaptation has been actively researched in recent years, most theoretical results and algorithms focus on the single-source-single-target adaptation setting. Naive application of such algorithms on multiple source domain adaptation problem may lead to suboptimal solutions. As a step toward bridging the gap…
M-L2O adapts fast to new tasks by self-adapting during test-time.
problem L2O optimizers struggle with out-of-distribution tasks.
method Meta-training an L2O optimizer to adapt quickly to new tasks.
result M-L2O converges significantly faster than vanilla L2O with only 5 steps of adaptation.
New method learns robust meta-representations for fast task adaptation.
problem Transferability of meta-representations in fast adaptation.
method Decoupled encoder-decoder approach with contrastive objective.
result Meta-representations improve downstream performance and noise robustness.
MetaNAS improves few-shot learning by optimizing neural architectures with meta-learning.
problem Few-shot learning challenges due to limited data and compute time.
method MetaNAS integrates NAS with gradient-based meta-learning to adapt neural architectures to new tasks efficiently.
result MetaNAS achieves state-of-the-art results on few-shot classification benchmarks.
Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better i…
The paper tackles learning from similar but not identical linear representations, improving performance over single-task learning.
problem Understanding how to learn from tasks with similar but not exactly the same linear representations, especially when dealing with outlier tasks.
method Proposes adaptive and robust penalized empirical risk minimization and spectral methods.
result Both methods outperform single-task learning when representations are similar and perform at least as well otherwise, with minimax optimality demonstrated.
Framework adapts to new tasks based on prior knowledge.
problem Models struggle to adapt to novel tasks without direct experience.
method Learned task representations and meta-mappings to transform them.
result Meta-mapping achieves 80-90% performance on novel tasks.
The study examines how language models learn to represent the world, identifying conditions for ecological veridicality.
problem Understanding when language models learn to represent the world accurately and how this learning process can fail.
method Analyzes the Bayes-optimal next-token cross-entropy decomposition and the role of training ecology in shaping model representations.
result The minimum-complexity zero-excess solution is the quotient partition by training equivalence, and this solution is not preserved in in-context learning or per-task adaptation.
New asynchronous SGD algorithms achieve optimal performance in distributed learning.
problem Asynchronous training introduces staleness, complicating optimization analysis.
method Developed rigorous framework for asynchronous first order stochastic optimization.
result Asynchronous SGD can achieve optimal time complexity, matching synchronous methods.
Deep neural networks have shown promising results for various clinical prediction tasks. However, training deep networks such as those based on Recurrent Neural Networks (RNNs) requires large labeled data, significant hyper-parameter tuning effort and expertise, and high computational resources. In this work, we invest…
This paper bridges MTL and meta-learning, showing their shared structure and efficiency.
problem Improving generalization and adaptation in multi-task and few-shot learning.
method Theoretical analysis and empirical investigation of MTL and gradient-based meta-learning.
result MTL and GBML share similar optimization formulations and predictions over unseen tasks.