A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
DSelect-k improves MoE models for multi-task learning with better performance and smoother training.
New insights into the top-K sparse softmax gating function for deep learning.
Theory explains why DARTS favors deep architectures over shallow ones.
We propose a practical method for norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known …
There is a pressing need to build an architecture that could subsume these networks under a unified framework that achieves both higher performance and less overhead. To this end, two fundamental issues are yet to be addressed. The first one is how to implement the back propagation when neuronal activations are discret…
Entropy regularization improves sparse model discovery in federated learning.
A new method for differentiable structured sparsity improves neural network performance and sparsity.
FleXOR trains fractional quantization for neural networks, improving accuracy and size.
New method uses temperature to control sparse MoE convergence rates.
A new method prunes neural networks efficiently without losing effectiveness.
Proposes -CCA for sparse CCA with improved representation learning.
We introduce the Genetic-Gated Networks (G2Ns), simple neural networks that combine a gate vector composed of binary genetic genes in the hidden layer(s) of networks. Our method can take both advantages of gradient-free optimization and gradient-based optimization methods, of which the former is effective for problems …
Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. It aims to use gates to control information flow (e.g., whether to skip some information or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this g…
Study compares L1 and VG sparsity priors in inverse problems.
HS-MoE selects sparse experts using adaptive priors and data-adaptive gating.
AdaEnsemble learns adaptive feature interactions for CTR prediction.
Gating is a key feature in modern neural networks including LSTMs, GRUs and sparsely-gated deep neural networks. The backbone of such gated networks is a mixture-of-experts layer, where several experts make regression decisions and gating controls how to weigh the decisions in an input-dependent manner. Despite having …
The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice…
Binary autoencoder with sparse hidden layer preserves information and zero reconstruction error.
Methods from convex optimization are widely used as building blocks for deep learning algorithms. However, the reasons for their empirical success are unclear, since modern convolutional networks (convnets), incorporating rectifier units and max-pooling, are neither smooth nor convex. Standard guarantees therefore do n…
A new hashing method improves accuracy by learning an orthogonal transform.
Transformers for binary decisions are sensitive to evidence order, leading to unreliable outcomes.
The paper analyzes convergence rates of softmax gating in MoE models.
Proposes a deep generative model for robust forecasting on sparse multivariate time series.
Quantum algorithm improves sparse vector recovery from noisy measurements.
Model compression techniques, such as pruning and quantization, are becoming increasingly important to reduce the memory footprints and the amount of computations. Despite model size reduction, achieving performance enhancement on devices is, however, still challenging mainly due to the irregular representations of spa…
New algorithm learns sparse GLMs for binary outcomes efficiently.
Temporal Functional Circuits explain KAN forecasts with interpretable edge functions.
Inspired by the advances in biological science, the study of sparse binary projection models has attracted considerable recent research attention. The models project dense input samples into a higher-dimensional space and output sparse binary data representations after the Winner-Take-All competition, subject to the co…
Study reveals heavy-tailed behavior in training ReLU gates.
Boosts share routing for multi-task learning with flexible sparse connections.
This paper develops the FastRNN and FastGRNN algorithms to address the twin RNN limitations of inaccurate training and inefficient prediction. Previous approaches have improved accuracy at the expense of prediction costs making them infeasible for resource-constrained and real-time applications. Unitary RNNs have incre…
GRU-D detects age-specific missing patterns in vital signs.
A new framework for sparse regression models with slow variations.
Efficiently poisons offline RLHF models by flipping preference labels.
Neural plasticity is an important functionality of human brain, in which number of neurons and synapses can shrink or expand in response to stimuli throughout the span of life. We model this dynamic learning process as an -norm regularized binary optimization problem, in which each unit of a neural network (e.g., …
A fast binary embedding method preserves Euclidean distances in high-dimensional data.
Mixtures-of-Experts models and their maximum likelihood estimation (MLE) via the EM algorithm have been thoroughly studied in the statistics and machine learning literature. They are subject of a growing investigation in the context of modeling with high-dimensional predictors with regularized MLE. We examine MoE with …
We consider network sparsification as an -norm regularized binary optimization problem, where each unit of a neural network (e.g., weight, neuron, or channel, etc.) is attached with a stochastic binary gate, whose parameters are jointly optimized with original network parameters. The Augment-Reinforce-Merge (ARM),…
We consider the problem of robust compressed sensing whose objective is to recover a high-dimensional sparse signal from compressed measurements corrupted by outliers. A new sparse Bayesian learning method is developed for robust compressed sensing. The basic idea of the proposed method is to identify and remove the ou…
Proposes a new deep learning model for uncertainty quantification and propagation.
This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.
In this paper we propose a mixture model, SparseMix, for clustering of sparse high dimensional binary data, which connects model-based with centroid-based clustering. Every group is described by a representative and a probability distribution modeling dispersion from this representative. In contrast to classical mixtur…
Novel approximation hierarchy for sparse quadratic programs.
Recurrent neural networks (RNNs) have recently achieved remarkable successes in a number of applications. However, the huge sizes and computational burden of these models make it difficult for their deployment on edge devices. A practically effective approach is to reduce the overall storage and computation costs of RN…
The pseudo-likelihood method is one of the most popular algorithms for learning sparse binary pairwise Markov networks. In this paper, we formulate the regularized pseudo-likelihood problem as a sparse multiple logistic regression problem. In this way, many insights and optimization procedures for sparse logistic…
Pruning is an efficient model compression technique to remove redundancy in the connectivity of deep neural networks (DNNs). Computations using sparse matrices obtained by pruning parameters, however, exhibit vastly different parallelism depending on the index representation scheme. As a result, fine-grained pruning ha…
This paper presents Sparse Partitioning, a Bayesian method for identifying predictors that either individually or in combination with others affect a response variable. The method is designed for regression problems involving binary or tertiary predictors and allows the number of predictors to exceed the size of the sa…