arXiv research

COMMOD debiases models with minimal and interpretable changes.

We introduce MinimalRNN, a new recurrent neural network architecture that achieves comparable performance as the popular gated RNNs with a simplified structure. It employs minimal updates within RNN, which not only leads to efficient learning and testing but more importantly better interpretability and trainability. We…

Decision trees are ubiquitous in machine learning for their ease of use and interpretability. Yet, these models are not typically employed in reinforcement learning as they cannot be updated online via stochastic gradient descent. We overcome this limitation by allowing for a gradient update over the entire tree that i…

Bayesian uncertainty quantification is flawed, according to new research.

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we …

Model forgets examples; this research predicts which ones to replay.

PrIU optimizes machine learning model updates after data cleaning.

Shampoo achieves higher token efficiency than Muon in language models.

Proposes a probabilistic digital twin for dynamical systems using sparse Bayesian learning.

We describe $k$-MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expec…

Improved backpropagation with consequentialism weight updates for neural networks.

When predictive models are used to support complex and important decisions, the ability to explain a model's reasoning can increase trust, expose hidden biases, and reduce vulnerability to adversarial attacks. However, attempts at interpreting models are often ad hoc and application-specific, and the concept of interpr…

In this paper, we propose a probabilistic optimization method, named probabilistic incremental proximal gradient (PIPG) method, by developing a probabilistic interpretation of the incremental proximal gradient algorithm. We explicitly model the update rules of the incremental proximal gradient method and develop a syst…

S2MAM improves semi-supervised learning by selecting relevant variables and updating similarity metrics.

In this paper we revisit the weighted likelihood bootstrap, a method that generates samples from an approximate Bayesian posterior of a parametric model. We show that the same method can be derived, without approximation, under a Bayesian nonparametric model with the parameter of interest defined as minimising an expec…

Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better i…

Fed-ensemble improves FL by averaging predictions from multiple models.

In this paper, we provide a novel construction of the linear-sized spectral sparsifiers of Batson, Spielman and Srivastava [BSS14]. While previous constructions required $Ω(n^4)$ running time [BSS14, Zou12], our sparsification routine can be implemented in almost-quadratic running time $O(n^{2+\varepsilon})$. The funda…

Additive models form a widely popular class of regression models which represent the relation between covariates and response variables as the sum of low-dimensional transfer functions. Besides flexibility and accuracy, a key benefit of these models is their interpretability: the transfer functions provide visual means…

The paper analyzes and proposes a new stopping criterion for recursive Bayesian classification.

Microstructure of market dynamics is studied through analysis of tick price data. Linear trend is introduced as a tool for such analysis. Trend arbitrage inequality is developed and tested. The inequality sets limiting relationship between trend, bid-ask spread, market reaction and average update frequency of price inf…

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an esti…

In this paper we study the problem of convergence and generalization error bound of stochastic momentum for deep learning from the perspective of regularization. To do so, we first interpret momentum as solving an $\ell_2$-regularized minimization problem to learn the offsets between arbitrary two successive model para…

Stochastic mirror descent (SMD) is a fairly new family of algorithms that has recently found a wide range of applications in optimization, machine learning, and control. It can be considered a generalization of the classical stochastic gradient algorithm (SGD), where instead of updating the weight vector along the nega…

Federated learning distributes model training among a multitude of agents, who, guided by privacy concerns, perform training using their local data but share only model parameter updates, for iterative aggregation at the server. In this work, we explore the threat of model poisoning attacks on federated learning initia…

Expectation-Maximization (EM) is a prominent approach for parameter estimation of hidden (aka latent) variable models. Given the full batch of data, EM forms an upper-bound of the negative log-likelihood of the model at each iteration and updates to the minimizer of this upper-bound. We first provide a "model level" in…

Improved neural network ensembles using Stein Variational Newton updates.

A method interprets black-box models using an ensemble of gradient boosting machines.

We show that unconverged stochastic gradient descent can be interpreted as a procedure that samples from a nonparametric variational approximate posterior distribution. This distribution is implicitly defined as the transformation of an initial distribution by a sequence of optimization updates. By tracking the change …

We show how to convert ICL in linearized transformers into model weights.

Disease progression models are instrumental in predicting individual-level health trajectories and understanding disease dynamics. Existing models are capable of providing either accurate predictions of patients prognoses or clinically interpretable representations of disease pathophysiology, but not both. In this pape…

Interpretable ML models predict recidivism as well as non-interpretable methods and are more fair.

CPR models complex decision processes by breaking them into context-specific policies, improving interpretability and accuracy.

NSA-Flow optimizes matrix representations for interpretability in complex data.

Muon replaces matrix gradient with polar factor, optimizing flat spectrum updates

Paper interprets contrastive learning dynamics using message passing.

Dynamic CBDT improves treatment effect estimation in clinical data.

New model predicts video sequences with latent dynamics.

Chance-constrained ActInf allows for small violations of constraints to drive goal-directed behavior.

A new gradient boosting method improves interpretability of probabilistic models.

A new model detects anomalies in time series data efficiently.

We propose a general model explanation system (MES) for "explaining" the output of black box classifiers. This paper describes extensions to Turner (2015), which is referred to frequently in the text. We use the motivating example of a classifier trained to detect fraud in a credit card transaction history. The key asp…

We present DRLViz, a visual analytics interface to interpret the internal memory of an agent (e.g. a robot) trained using deep reinforcement learning. This memory is composed of large temporal vectors updated when the agent moves in an environment and is not trivial to understand due to the number of dimensions, depend…

Improves inference-time alignment for diffusion models without updating weights.

A new TS-SA method alleviates non-stationarity in TS algorithms for bandits.

Over 50 years of work on group actions on $4$-manifolds, from the 1960's to the present, from knotted fixed point sets to Seiberg-Witten invariants, is surveyed. Locally linear actions are emphasized, but differentiable and purely topological actions are also discussed. The presentation is organized around some of the …

Mixability of a loss is known to characterise when constant regret bounds are achievable in games of prediction with expert advice through the use of Vovk's aggregating algorithm. We provide a new interpretation of mixability via convex analysis that highlights the role of the Kullback-Leibler divergence in its definit…

The paper develops a stationary-distribution theory for Random Forest ensemble size selection.