arXiv research

Gradient-based meta-RL fails with incorrect task distributions, leading to instability and poor performance.

Proposes VSGD optimizer combining probabilistic and gradient-based methods.

Gradient learning optimises MCMC proposal distributions.

Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these …

Improved privacy and utility in machine learning with adaptive differential privacy.

We study the problem of estimating from data, a sparse approximation to the inverse covariance matrix. Estimating a sparsity constrained inverse covariance matrix is a key component in Gaussian graphical model learning, but one that is numerically very challenging. We address this challenge by developing a new adaptive…

Adaptive model learns from time series data with changing distributions.

In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. We experimentally demonstrate that as meta-training progresses, the meta-test solutions, obtained after adapting the meta-train solution of the model, to new tasks via few…

SAP learns efficient task-specific parameter subspaces for few-shot learning.

In modern data analysis, random sampling is an efficient and widely-used strategy to overcome the computational difficulties brought by large sample size. In previous studies, researchers conducted random sampling which is according to the input data but independent on the response variable, however the response variab…

Adaptive vehicle trajectory prediction for safer autonomous driving.

Improved DP algorithms for non-convex optimization with tighter generalization bounds.

Meta learning improves with contextualizers that adapt to examples.

Stochastic-gradient-based optimization has been a core enabling methodology in applications to large-scale problems in machine learning and related areas. Despite the progress, the gap between theory and practice remains significant, with theoreticians pursuing mathematical optimality at a cost of obtaining specialized…

Meta learning works well with overparameterized models, a phenomenon called 'benign overfitting'.

Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.

Proposes an automatic cyclical scheduling for gradient-based discrete sampling.

This paper surveys gradient-based multi-objective deep learning methods.

Paper introduces a privacy-preserving line search method for optimization.

LAWN normalizes logits to improve deep network adaptability and generalization.

RLMH improves adaptive MCMC by optimizing contrastive divergence reward.

Enhances gradient-based discrete samplers with parallel tempering for multimodal distributions.

Improves meta-learning efficiency with mixed-mode differentiation.

Paper investigates robustness to interference as a new training signal for meta-learning.

Combines gradient-based and competitive learning for unsupervised feature extraction.

Gradient-based meta-learning methods leverage gradient descent to learn the commonalities among various tasks. While previous such methods have been successful in meta-learning tasks, they resort to simple gradient descent during meta-testing. Our primary contribution is the {\em MT-net}, which enables the meta-learner…

Throughout the past five years, the susceptibility of neural networks to minimal adversarial perturbations has moved from a peculiar phenomenon to a core issue in Deep Learning. Despite much attention, however, progress towards more robust models is significantly impaired by the difficulty of evaluating the robustness …

TTT improves model adaptation to test data, especially for nonlinear models.

Researchers developed a differentially private method for computing Wasserstein distances.

Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model b…

MetaNAS improves few-shot learning by optimizing neural architectures with meta-learning.

Characterizes corridors in loss surfaces for gradient-based optimization.

Meta-Graph learns to predict missing edges quickly from few samples.

GBML with deep nets converges globally and generalizes well.

New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.

Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies. This paper provide…

Cross-regularization adapts model complexity during training.

Learning to infer Bayesian posterior from a few-shot dataset is an important step towards robust meta-learning due to the model uncertainty inherent in the problem. In this paper, we propose a novel Bayesian model-agnostic meta-learning method. The proposed method combines scalable gradient-based meta-learning with non…

We study local complexity measures for stochastic convex optimization problems, providing a local minimax theory analogous to that of Hájek and Le Cam for classical statistical problems. We give complementary optimality results, developing fully online methods that adaptively achieve optimal convergence guarantees. Our…

Paper presents a new method for Bayesian deep learning that scales to Atari games.

Unified approach for Bayesian optimal experiment design using stochastic gradients.

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …

WassersteinGrad improves weather forecasting explanations by addressing geometric misalignment issues.

We build a theoretical framework for designing and understanding practical meta-learning methods that integrates sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms. Our approach enables the task-similarity to be learned adapti…

New algorithm solves min-max optimization problems in a decentralized manner.

Adaptive HMC improves sampling efficiency by optimizing mass matrix.

Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters …

We introduce a generic scheme to solve nonconvex optimization problems using gradient-based algorithms originally designed for minimizing convex functions. Even though these methods may originally require convexity to operate, the proposed approach allows one to use them on weakly convex objectives, which covers a larg…