A new type of neural network variable called 'fast weights' improves sequence models by storing recent past memories.
problem Sequence models benefit from attention to the past, but current neural networks lack a mechanism to store recent past information efficiently.
method Introduce 'fast weights' that change faster than neural activities but slower than standard weights, allowing for temporary memory storage of recent past.
result Fast weights enable efficient implementation of past attention in sequence models without needing to store neural activity patterns.
FAWMF adapts weights for implicit feedback recommendation efficiently.
problem Challenges in treating unobserved data as negative in implicit feedback recommendation.
method FAWMF uses a variational auto-encoder with a parameterized neural network to adaptively assign personalized data confidence weights, and fBGD for efficient learning.
result FAWMF and fBGD outperform existing methods in real-world datasets.
A fast matrix factorization method for sparse data with non-uniform missing data weights.
problem Sparse and imbalanced data in real-world learning systems.
method Non-uniform weighting of missing data, efficient learning method with truncated SVD and eALS.
result Improved performance in downstream applications compared to uniform weighting.
The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.
problem Estimating weights and quality functions in reinforcement learning with function approximation.
method The paper uses minimax methods for estimating marginal importance weights and q-functions.
result The minimax approach enables fast rates of convergence for weights and quality functions, achieving first-order efficiency.
The paper analyzes SBL pruning criteria under weakened assumptions.
problem Sparse Bayesian learning hyperparameter divergence and pruning.
method Analyzing marginal likelihood function under weakened Gaussian assumptions.
result Conditions for finite vs infinite hyperparameters lead to F-SBL pruning.
Unified neural and associative memory for one-shot learning.
problem One-shot learning with limited data.
method Jointly learns data representation and task binding via slow and fast weights.
result State-of-the-art results on one-shot learning benchmarks.
Sparse Meta Networks adapt deep neural networks incrementally for fast learning.
problem Training deep neural networks is slow and impractical for complex, changing environments.
method Sparse Meta Networks use a memory layer to learn online sequential adaptation, accumulating fast-weights incrementally.
result Sparse Meta Networks achieve strong performance in various sequential adaptation scenarios.
Gradient descent converges to perfect classification in neural nets for non-separable data.
problem Classifying linearly non-separable data using neural networks.
method Analysis of gradient descent dynamics in neural networks with sufficient but not large number of neurons.
result Gradient descent converges to global minima with perfect classification in the landscape of minimization problems.
We propose weight-sharing regularization for neural networks.
problem Training fully connected networks with shuffled pixel data.
method Introduce a regularization penalty on weights and design parallel algorithms.
result Weight-sharing regularization enables learning convolution-like filters.
Finite-time extinction and smoothing effects in fractional fast diffusion on manifolds.
problem Finite-time extinction and smoothing effects in fractional fast diffusion equations.
method Nonlinear semigroups techniques, weighted L p L^p L p spaces, fractional Green function. result Sharp extinction rates and pointwise lower bounds for solutions.
Study improves kernel quadrature for infinitely wide models with faster approximation and estimation rates.
problem Efficiently approximating and estimating expectations in infinitely wide models.
method Developed general kernel quadrature (GKQ) for parameter distributions, achieving faster rates.
result Achieved a fast approximation rate of O ( e − p ) O(e^{-p}) O ( e − p ) and a fast estimation rate of O ~ ( 1 / n ) \widetilde{O}(1/n) O ( 1/ n ) . Power weighted shortest paths improve clustering of high-dimensional data.
problem Clustering high-dimensional Euclidean data with disjoint low-dimensional manifolds.
method Use of power weighted shortest path distance functions and a fast algorithm.
result Higher clustering accuracy achieved through power weighted shortest paths.
New normalization technique balances positive and negative weights for faster convergence.
problem Balancing positive and negative weights for faster convergence.
method Transformation of layer weights instead of outputs, balancing positive and negative contributions.
result Balanced normalization leads to faster convergence on standard benchmarks.
A fast method for approximate TDE in ultrasound elastography.
problem Challenging and critical step in ultrasound elastography.
method PCA-GLUE, exploiting PCA to find TDE principal components.
result PCA-GLUE is more than ten times faster than Dynamic Programming.
Decentralized algorithm reduces regret and converges to Nash equilibrium in online congestion games.
problem Online congestion games with exponential action sets and strict Nash equilibria.
method CongestEXP algorithm using exponential weights method.
result CongestEXP achieves O ( k F T ) O(kF\sqrt{T}) O ( k F T ) regret bound and almost exponential convergence to strict Nash equilibrium. We consider the problem of learning the inhomogeneous intensity of a counting process, under a sparse segmentation assumption. We introduce a weighted total-variation penalization, using data-driven weights that correctly scale the penalization along the observation interval. We prove that this leads to a sharp tuning …
We describe k k k -MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expec…
VSML unifies meta learning concepts and enables simple backpropagation.
problem Improving and unifying meta learning concepts for neural networks.
method Unified approach using variable shared meta learning and simple weight-sharing.
result Simple backpropagation can be implemented and meta learned without gradient calculation.
We compress large neural networks for quick adaptation to specific contexts.
problem How to quickly adapt a pretrained large neural network to specific contexts.
method Propose a Bayesian hypernetwork framework to compress the network and encourage sparsity.
result Generated compressed networks are significantly smaller than baseline methods.
Fast detection of changepoints in linear regression models.
problem Early detection of changes in linear regression models.
method Heavy-weighted CUSUM statistics and composite statistics.
result Effective detection of changepoints regardless of their location.
Study reveals conditions for neural networks to forget learned features.
problem Understanding feature unlearning in neural networks.
method Infinite-width limit analysis with stochastic gradient descent, fast-slow dynamics.
result Conditions for feature unlearning are determined by the strength of nonlinear terms and initial weights.
Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen consid…
Develops a robust, fast, and widely-applicable neural architecture search method.
problem Inability of current NAS methods to be easily applied to new problems.
method Adaptive stochastic natural gradient method for simultaneous optimization of weights and architecture.
result Near state-of-the-art performances with low computational budgets.
New theory maps neural network weights to optimize faster and scale.
problem Optimizing neural networks for speed and scalability.
method Constructing a duality map using layer-wise operator norms.
result Derived GPU-friendly algorithms for various layers.
We describe a new optimization scheme for finding high-quality correlation clusterings in planar graphs that uses weighted perfect matching as a subroutine. Our method provides lower-bounds on the energy of the optimal correlation clustering that are typically fast to compute and tight in practice. We demonstrate our a…
In-Place TTT enhances LLMs with dynamic parameter updates at inference time.
problem Static training limits LLMs from adapting to new information.
method In-Place TTT updates a subset of model parameters (fast weights) at inference time.
result In-Place TTT enables 4B-parameter models to outperform on tasks with up to 128k contexts.
Net-Trim simplifies neural networks by pruning layers efficiently.
problem Simplifying trained deep neural networks for efficiency.
method Net-Trim is a convex post-processing technique that prunes neural network layers.
result Net-Trim can find a network with a limited number of nonzero terms from a small number of samples.
Wide residual networks achieve low error rates with single-bit weights.
problem Deploying deep neural networks on resource-constrained hardware with low memory.
method Binarizing weights using sign function and scaling factors, applying warm-restart learning rate schedule.
result Achieved error rates of 3.9% on CIFAR-10, 18.5% on CIFAR-100, and 26.0% on ImageNet with 1-bit-per-weight.
Paper proposes a fast stochastic algorithm for neural network quantization with error bounds.
problem Error analysis for quantized neural networks with non-convex loss functions and nonlinear activations.
method Greedy path-following mechanism combined with stochastic quantizer.
result Established full-network error bounds for quantized neural networks.
New method learns to weight unlabeled data in semi-supervised learning.
problem Equal weighting of all unlabeled data in semi-supervised learning.
method Adjust weights for each unlabeled example using influence function.
result Technique outperforms state-of-the-art methods on image and language classification tasks.
We investigate the m m m -relative entropy, which stems from the Bregman divergence, on weighted Riemannian and Finsler manifolds. We prove that the displacement K K K -convexity of the m m m -relative entropy is equivalent to the combination of the nonnegativity of the weighted Ricci curvature and the K K K -convexity of the weig…
New method combines simulated annealing and Levy distribution for fast matrix factorization.
problem High complexity and difficulty in parallelizing matrix factorization for large matrices.
method Combining simulated annealing with Levy distribution for matrix factorization.
result Achieves good solutions in acceptable time with low computations.
The paper analyzes prediction error in nonstationary settings using weighted risk minimization.
problem Prediction under distribution drift and nonstationary conditions.
method General decomposition of excess risk into learning and drift terms, proving oracle inequalities under mixing conditions.
result Oracle inequalities for the learning error, providing bounds that hold uniformly over arbitrary weight classes.
Gradient descent with random weights in linear regression analyzed for various noise types.
problem Analyzing the impact of random noise on gradient descent in linear regression.
method Gradient descent with randomly weighted data points, various weighting distributions, geometric moment contraction.
result Characterization of implicit regularization and non-asymptotic convergence bounds.
A new sampling strategy for random Fourier features reduces computation time and improves prediction performance.
problem Efficient generation of random Fourier features for kernel approximation.
method Surrogate leverage weighted sampling guided by kernel alignment, avoiding matrix inversion.
result Time complexity reduced from O(ns^2+s^3) to O(ns^2), comparable or slightly better prediction performance.
A-BLINK speeds up Gaussian process covariance estimation.
problem Slow covariance matrix inversion in Gaussian processes.
method Two pre-trained neural networks learn Kriging weights and spatial variance.
result Significant computational speedups and posterior inference.
Item neighbourhood methods for collaborative filtering learn a weighted graph over the set of items, where each item is connected to those it is most similar to. The prediction of a user's rating on an item is then given by that rating of neighbouring items, weighted by their similarity. This paper presents a new neigh…
Two randomized algorithms improve hypergraph learning accuracy and efficiency.
problem Efficiently learning and tagging images in hypergraphs.
method Block randomized SVD and conjugate gradient method.
result Both methods achieve high accuracy and reduce computational requirements.
We provide a fast L2-embedding for arbitrary accuracy with applications to regression and L1 tasks.
problem Efficiently embedding high-dimensional data while maintaining accuracy.
method Oblivious L2-embedding with dimension independent of accuracy.
result Achieves arbitrary accuracy with constant embedding dimension.
A novel weighted distance improves fuzzy c-means clustering accuracy.
problem Improving fuzzy c-means clustering performance with weighted distances.
method Proposed Canberra Weighted Distance to enhance FCM algorithm.
result Experimental results show superior performance of the proposed method.
A new method for fast XMLC using IR vector space model.
problem Handling many labels in extreme multi-label classification.
method Sparse Weighted Nearest-Neighbor Method derived from SOTA linear classifiers.
result Equivalent performance to SOTA models on large datasets.
RCRC model extracts features quickly and efficiently for reinforcement learning tasks.
problem High computational costs in training reinforcement learning models.
method Convolutional Reservoir Computing (RCRC) model using random fixed-weight CNN and reservoir computing.
result RCRC model achieves state-of-the-art scores in reinforcement learning tasks.
A method to reduce memory usage in deep learning models by adding inducing weights.
problem Memory inefficiency in Bayesian neural networks and deep ensembles.
method Augmenting the weight matrix with inducing weights and using Matheron's conditional Gaussian sampling rule.
result Reduces parameter size to 24.3% of a single neural network while maintaining competitive performance.
Weighted Monte Carlo prices exotic options calibrating the probabilities of previously generated paths by a regular Monte Carlo to fit a set of option premiums. When only vanilla call and put options and forward prices are considered, the Martingale condition might not be preserved. This paper shows that this is indeed…
FIDDLE uses deep learning to estimate ATE from complex data.
problem Estimating ATE from high-dimensional, correlated covariates with sparse nonlinear effects.
method Factor-augmented deep learning for propensity and outcome models.
result FIDDLE consistently estimates ATE under model misspecification and is semiparametrically efficient.
A method refines weights to estimate smooth manifolds from noisy data.
problem Estimating smooth manifolds from noisy data.
method Structure-adaptive procedure that iteratively refines weights using structural information.
result The method achieves nearly optimal performance even with large noise.
WildWood improves Random Forest predictions using bootstrap out-of-bag samples.
problem Improving Random Forest predictions for supervised learning.
method Uses bootstrap out-of-bag samples to compute improved predictions by aggregating all possible subtrees with exponential weights.
result WildWood produces faster and more competitive predictions compared to other ensemble methods.
A new machine learning method solves high-dimensional Kolmogorov PDEs efficiently.
problem Solving high-dimensional Kolmogorov PDEs and SDEs.
method Stochastic weighted minimization and stochastic gradient descent with Malliavin weights.
result Accurate approximation of high-dimensional Kolmogorov PDEs and SDEs without curse of dimensionality.