We describe -MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expec…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Exponential weight updates”
Efficiently optimizes boolean functions using multilinear polynomials and exponential weight updates.
problem Optimizing boolean functions over the boolean hypercube with high computational cost.
method Proposes a computationally efficient algorithm using multilinear polynomials and exponential weight updates.
result Improves computational time up to several orders of magnitude compared to state-of-the-art algorithms.
A new accelerated method with simpler momentum update rules.
problem Optimizing parameters in machine learning models.
method Proposes a novel accelerated stochastic gradient method with simpler momentum update rules.
result The method outperforms Sgdm and Adam in practical problems.
Paper proposes a new activation function to reduce overfitting and large weight update issues.
problem Overfitting and large weight update problems in neural networks.
method Introduces a new activation function called Thresholded Exponential Rectified Linear Units (TERELU).
result TERELU shows better performance in reducing overfitting and large weight update issues compared to other activation functions.
A new method for exponentially weighted moving models using approximations.
problem Efficiently updating moving averages for time series data.
method Approximates EWMM using a fixed window and quadratic term, solving non-growing problems.
result Approximation produces estimates similar to exact EWMM.
A new method for robust training under label noise using weighted gradient descent.
problem Overfitting to noisy examples in machine learning.
method Exponentiated gradient reweighting for flexible handling of noisy data.
result Improved generalization in noisy classification and PCA problems.
Generalized Exponentiated Gradient Algorithms and Their Application to On-Line Portfolio Selectioncs.LG
EGAB algorithms improve online portfolio selection.
problem Online portfolio selection problem.
method Generalized exponentiated gradient (EG) updates with Alpha-Beta divergence regularization.
result EGAB algorithms enhance portfolio performance, especially with transaction costs.
A new method normalizes flow mixtures for better inference across different data types.
problem Inference failure across diverse posterior geometries in normalizing flows.
method Introduces a two-stage framework with a stable global weighting mechanism based on sEMA.
result Achieves consistent NLL improvements and stable weight trajectories over baselines.
Gradient descent on normalized networks reveals sparsity preferences.
problem Understanding the inductive bias of gradient descent on normalized neural nets.
method Analysis of gradient descent on weight-normalized smooth homogeneous neural nets, focusing on SWN and EWN.
result EWN causes weights to be updated in a way that prefers asymptotic relative sparsity.
Stochastic mirror descent (SMD) is a fairly new family of algorithms that has recently found a wide range of applications in optimization, machine learning, and control. It can be considered a generalization of the classical stochastic gradient algorithm (SGD), where instead of updating the weight vector along the nega…
Paper proposes efficient weight updates for edge nodes with minimal communication.
problem Inefficient and resource-intensive full weight updates for edge nodes.
method Deep partial updating, selecting a subset of weights to update.
result Achieves similar performance with fewer weight updates.
Improved MMWU algorithm achieves instance-optimal regret bound for matrix LEA.
problem Matrix Learning from Expert Advice problem.
method Developed a general potential-based framework for matrix LEA, using a new Jensen's trace inequality.
result Achieved instance-optimal regret bound of .
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
Multi-layer neural networks have lead to remarkable performance on many kinds of benchmark tasks in text, speech and image processing. Nonlinear parameter estimation in hierarchical models is known to be subject to overfitting and misspecification. One approach to these estimation and related problems (local minima, co…
We show that deep networks can be trained using Hebbian updates yielding similar performance to ordinary back-propagation on challenging image datasets. To overcome the unrealistic symmetry in connections between layers, implicit in back-propagation, the feedback weights are separate from the feedforward weights. The f…
In this paper, we present a novel approach for fine-tuning a decoder-side neural network in the context of image compression, such that the weight-updates are better compressible. At encoder side, we fine-tune a pre-trained artifact removal network on target data by using a compression objective applied on the weight-u…
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For …
Improved diffusion models for image synthesis with better training dynamics.
problem Uneven and ineffective training in diffusion models.
method Redesigned network layers to preserve activation, weight, and update magnitudes.
result Significantly better networks at equal computational complexity, improving FID to 1.81.
We derive scaling laws for optimizing neural networks in hardware.
problem Optimizing the large parameter space of neural networks in hardware.
method Analytical derivation of scaling laws for Coordinate Descent optimization.
result Convergence is exponential and scales linearly with the number of neurons.
RSO uses random weight perturbations to train deep networks without gradients.
problem Training deep neural networks efficiently and without gradient information.
method RSO is a gradient-free Markov Chain Monte Carlo approach that updates weights based on mini-batch loss reduction.
result RSO achieves high accuracy (99.1% on MNIST) with significantly fewer updates than traditional methods.
Paper proves multiplicative weight updates can train neural networks without learning rate tuning.
problem Vanishing and exploding gradients in gradient descent for compositional functions.
method Proves descent lemma for compositional functions using multiplicative weight updates and derives Madam optimizer.
result Madam optimizer trains state-of-the-art neural networks without learning rate tuning.
We show that a simple randomized sketch of the matrix multiplicative weight (MMW) update enjoys (in expectation) the same regret bounds as MMW, up to a small constant factor. Unlike MMW, where every step requires full matrix exponentiation, our steps require only a single product of the form , which the Lanczos …
Extends online learning to metric spaces using exponential weights.
problem Online learning in metric spaces.
method Exponentially weighted average forecaster, barycenters, Jensen's inequality, measure contraction property.
result Results in a statistical learning framework.
Online learning makes sequence of decisions with partial data arrival where next movement of data is unknown. In this paper, we have presented a new technique as multiple times weight updating that update the weight iteratively forsame instance. The proposed technique analyzed with popular state-of-art algorithms from …
We present an online approach to portfolio selection. The motivation is within the context of algorithmic trading, which demands fast and recursive updates of portfolio allocations, as new data arrives. In particular, we look at two online algorithms: Robust-Exponentially Weighted Least Squares (R-EWRLS) and a regulari…
Bio-inspired neural networks use predictive coding for efficient weight updates.
problem Training artificial neural networks efficiently and biologically plausibly.
method Predictive Coding (PC) updates weights locally using only local information.
result PC provides theoretical advantages like automatic gradient scaling.
Exponential Weighting and Random-Matrix-Theory-Based Filtering of Financial Covariance Matrices for Portfolio Optimizationcond-mat.stat-mech
We introduce a covariance matrix estimator that both takes into account the heteroskedasticity of financial returns (by using an exponentially weighted moving average) and reduces the effective dimensionality of the estimation (and hence measurement noise) via techniques borrowed from random matrix theory. We calculate…
A new update rule for deep reinforcement learning reduces learning variance and variance in reference signals.
problem Learning variance and incorrect reference signals in deep reinforcement learning.
method t-soft update method inspired by student-t distribution, which reduces extreme updates and accelerates similar updates.
result The t-soft update method outperforms conventional methods in terms of return and variance in PyBullet robotics simulations.
In this paper, a Bayesian inference technique based on Taylor series approximation of the logarithm of the likelihood function is presented. The proposed approximation is devised for the case, where the prior distribution belongs to the exponential family of distributions. The logarithm of the likelihood function is li…
This paper optimizes deep learning training by efficiently sharding weight updates across replicas.
problem Redundant weight update computation on all replicas in data-parallel training.
method Automatic sharding of weight updates using static analysis and transformations on the training graph.
result Substantial speedups achieved on large-scale models using Cloud TPUs.
Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameter…
Adversarial training is a technique for training robust machine learning models. To encourage robustness, it iteratively computes adversarial examples for the model, and then re-trains on these examples via some update rule. This work analyzes the performance of adversarial training on linearly separable data, and prov…
Introduces Exponentially Weighted Signature for better path representation.
problem Uniform treatment of historical information in signatures.
method Generalizes EFM signature to bounded linear operators, enabling contextualised temporal weighting.
result EWS is the unique solution to a linear controlled differential equation and generalizes state-space models.
New learning rule simplifies Bayesian updates for deep learning.
problem Bayesian learning rule's complexity and manifold constraints.
method Lie-group approach to simplify Bayesian updates.
result New algorithm learns sparse features in deep learning.
Fewer data weight updates lead to faster convergence in machine learning models.
problem Improving robustness of machine learning models through data mixing.
method Analyzing convergence behavior of data mixing with a finite number of inner steps.
result The optimal number of inner steps scales with the budget and type of gradients used.
Massively parallel RWS improves inference in complex models.
problem Exponential sample requirement for effective importance weighting.
method Draws samples of all latent variables and individually reasons through all combinations.
result Significant improvements over standard RWS.
MELO predicts electricity loads by adapting to shifts without external indicators.
problem Adapting to non-stationary prediction challenges in online settings.
method MELO combines multiple forgetting factors and aggregation rules to adaptively predict.
result MELO reduces RMSE by 34.7% compared to base predictors and external covariates.
Sparse Inverse Covariance Estimation (SICE) is useful in many practical data analyses. Recovering the connectivity, non-connectivity graph of covariates is classified amongst the most important data mining and learning problems. In this paper, we introduce a novel SICE approach using adaptive thresholding. Our method i…
Researchers extend asymptotic analysis to Bergman projections with Gevrey weights.
problem Analyzing Bergman projections with Gevrey weights.
method Extending direct approach to semiclassical asymptotics to Gevrey weights using Fourier integral operators.
result Gevrey symbol amplitude of asymptotic Bergman projection with Gevrey weights and Gevrey-type growth rate.
New algorithm for online optimization over symmetric cones, unifying previous methods.
problem Online convex optimization over symmetric cones.
method Symmetric-Cone Multiplicative Weights Update (SCMWU) algorithm.
result SCMWU is a no-regret algorithm.
Spectral Adaptive Conformal Prediction for Structured Non-Exchangeable Data
problem Improving prediction intervals for non-exchangeable time-indexed datasets
method Spectral adaptive conformal prediction
result Improves on fixed spectral weighting while monitoring uncertainty changes
The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.
problem Reward modeling in policy optimization and its impact on downstream value.
method Two-stage neural reward model: first learns hidden direction, then fits readout layer.
result For any feature-learning temperature above a dimension-free threshold, a constant fraction of neurons recover the hidden direction.
Sparse random networks reduce communication in federated learning.
problem Large communication cost in federated learning.
method Freeze random weights, train stochastic binary mask to sparsify.
result Improves accuracy, reduces communication, speeds convergence.
Paper explores exact recovery of communities in weighted graphs using Gaussian and exponential distributions.
problem Exact recovery of communities in weighted graphs with Gaussian and exponential distributions.
method Introduces a new semi-metric to describe conditions for exact recovery and analyzes conditions for both complete and incomplete graphs.
result Necessary and sufficient conditions for exact recovery are asymptotically tight and applicable to both complete and incomplete graphs.
This paper studies recursive ensembles driven by Fibonacci updates, improving learning dynamics.
problem Improving learning dynamics in recursive ensemble learning.
method Develops second-order recursive architectures with Fibonacci-type update flows.
result Establishes global convergence conditions and generalization bounds for recursive ensembles.
This paper considers the actor-critic contextual bandit for the mobile health (mHealth) intervention. The state-of-the-art decision-making methods in mHealth generally assume that the noise in the dynamic system follows the Gaussian distribution. Those methods use the least-square-based algorithm to estimate the expect…
QBVI uses natural gradients for efficient Bayesian learning.
problem Efficient Bayesian learning in complex models.
method Natural gradient updates in a black-box framework for exponential-family distributions.
result QBVI framework is effective for a wide range of Bayesian inference problems.
A new associative memory uses Sinkhorn divergence for efficient pattern retrieval.
problem Efficiently retrieving patterns from large datasets of weighted point clouds.
method Derived retrieval dynamics as a SHK gradient flow, discretized for a deterministic algorithm.
result Proved basin invariance, geometric convergence, and robust recovery from perturbations.