Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
In order to quantize the gate parameters of the LSTM (Long Short-Term Memory) neural network model with almost no recognition performance degraded, a new quantization method named Quantization Loss Re-Learn Method is proposed in this paper. The method does lossy quantization on gate parameters during training iteration…
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
problem Identifiability issues and complex interactions in Gaussian mixture of experts.
method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.
result Connects convergence rate of MLE to a solvability problem of polynomial equations.
Gating is a key feature in modern neural networks including LSTMs, GRUs and sparsely-gated deep neural networks. The backbone of such gated networks is a mixture-of-experts layer, where several experts make regression decisions and gating controls how to weigh the decisions in an input-dependent manner. Despite having …
New insights into the top-K sparse softmax gating function for deep learning.
problem Understanding the theoretical effects of the top-K sparse softmax gating function on density and parameter estimations.
method Using a Gaussian mixture of experts, novel loss functions, and theoretical analysis.
result The convergence rates of density and parameter estimations are parametric under certain conditions, but slow under over-specified models.
Analyzes convergence rates for Gaussian-gated MoE model.
problem Theoretical understanding of Gaussian-gated MoE model is incomplete.
method Maximum likelihood estimation with novel Voronoi loss functions.
result MLE has distinct behaviors under different settings of Gaussian gating function parameters.
The paper establishes convergence rates for MoE models in classification problems.
problem Understanding the behavior of MoE models in classification settings.
method Established convergence rates for density and parameter estimation in softmax gating multinomial logistic MoE models.
result Parameter estimation rates are significantly improved with a novel modified softmax gating function.
Spatiotemporal sequence prediction is an important problem in deep learning. We study next-frame(s) video prediction using a deep-learning-based predictive coding framework that uses convolutional, long short-term memory (convLSTM) modules. We introduce a novel reduced-gate convolutional LSTM(rgcLSTM) architecture that…
The paper evaluates three variants of the Gated Recurrent Unit (GRU) in recurrent neural networks (RNN) by reducing parameters in the update and reset gates. We evaluate the three variant GRU models on MNIST and IMDB datasets and show that these GRU-RNN variant models perform as well as the original GRU RNN model while…
The paper analyzes convergence rates of softmax gating in MoE models.
problem The effectiveness and scalability of machine learning models using MoE.
method Convergence analysis of parameter and expert estimation under MoE with softmax gating and its variants.
result Theoretical results show polynomially many data points are needed for strong identifiability conditions, while exponential points are required for linear experts.
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
Recurrent neural networks with various types of hidden units have been used to solve a diverse range of problems involving sequence data. Two of the most recent proposals, gated recurrent units (GRU) and minimal gated units (MGU), have shown comparable promising results on example public datasets. In this paper, we int…
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
Graph processes model a number of important problems such as identifying the epicenter of an earthquake or predicting weather. In this paper, we propose a Graph Convolutional Recurrent Neural Network (GCRNN) architecture specifically tailored to deal with these problems. GCRNNs use convolutional filter banks to keep th…
Improved HMoE models using Laplace gating function enhance expert specialization and performance.
problem Improving performance of hierarchical mixture of experts models.
method Used Laplace gating function instead of Softmax in hierarchical mixture of experts models.
result Laplace gating function accelerates expert convergence and enhances specialization.
Bayesian models combine experts with a flexible gating mechanism for complex data.
problem Theoretical properties of Bayesian mixture-of-experts models with softmax gating remain unexplored.
method Investigated asymptotic behavior of posterior distribution for density estimation, parameter estimation, and model selection.
result Established posterior contraction rates for density estimation and parameter estimation, providing insights for practical model design.
Unified framework for SGMoE resolves estimation and selection issues.
problem Non-identifiability, coupled differential relations, and tight coupling in softmax-Gated models.
method Unified statistical framework with Voronoi-type loss functions and dendrograms of mixing measures.
result Consistent selection of the number of experts without model sweeps, optimal parameter rates under overfitting.
Gated recurrent neural networks have achieved remarkable results in the analysis of sequential data. Inside these networks, gates are used to control the flow of information, allowing to model even very long-term dependencies in the data. In this paper, we investigate whether the original gate equation (a linear projec…
We propose a practical method for L0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known …
The study tightens risk bounds for mixtures of experts using local differential privacy.
problem Improving risk bounds for mixtures of experts.
method Imposing local differential privacy (LDP) on the gating mechanism of mixtures of experts.
result Theoretical bounds exhibit logarithmic dependence on the number of experts and tighter than existing bounds.
The standard LSTM recurrent neural networks while very powerful in long-range dependency sequence applications have highly complex structure and relatively large (adaptive) parameters. In this work, we present empirical comparison between the standard LSTM recurrent neural network architecture and three new parameter-r…
Paper proposes a novel method to accurately determine the number of experts in Gaussian-gated Gaussian MoE models.
problem Challenges in model selection for MoE models, especially with covariates.
method Introduces a novel extension using dendrograms of mixing measures to estimate the true number of mixture components.
result Achieves optimal convergence rates for parameter estimation and accurately approximates the regression function.
Model accurately gates ocean microbes from high-frequency flow cytometry data.
problem Gating of high-frequency flow cytometry data for ocean microbes is challenging.
method Trend filtered mixture of experts with smooth parameter variation.
result Model accurately matches human-annotated gating and corrects errors.
SG-NTF completes HDI tensors with spectral mapping and spatio-temporal gating.
problem High-dimensional and incomplete tensor completion.
method Spectra-Guided Neural Tucker Factorization (SG-NTF) with Spatio-Temporal Co-Gating (STCG).
result Maintains competitive completion accuracy with parameter efficiency.
Recurrent neural networks can be difficult to train on long sequence data due to the well-known vanishing gradient problem. Some architectures incorporate methods to reduce RNN state updates, therefore allowing the network to preserve memory over long temporal intervals. To address these problems of convergence, this p…
A new method selects features for clustering without labels.
problem Identifying meaningful features in large datasets.
method Differentiable unsupervised feature selection using a gated Laplacian.
result The method improves clustering performance in noisy data.
Perceptual capabilities of artificial systems have come a long way since the advent of deep learning. These methods have proven to be effective, however they are not as efficient as their biological counterparts. Visual attention is a set of mechanisms that are employed in biological visual systems to ease computationa…
This paper develops the FastRNN and FastGRNN algorithms to address the twin RNN limitations of inaccurate training and inefficient prediction. Previous approaches have improved accuracy at the expense of prediction costs making them infeasible for resource-constrained and real-time applications. Unitary RNNs have incre…
The paper studies estimation rates for MoE models with a new prompt.
problem Estimating parameters in a softmax-contaminated MoE model.
method Analytic notion of distinguishability, minimax optimal estimation rates.
result Estimation rates are minimax optimal under distinguishability, but slower otherwise.
Entropy regularization improves sparse model discovery in federated learning.
problem Sparse model discovery in federated learning with limited data.
method Entropy regularization of gate distributions for probabilistic sparse model exploration.
result Entropy regularization leads to better sparse model recovery and performance.
New algorithm trains ReLU gates provably in linear time.
problem Training ReLU gates in realizable settings with mild conditions.
method Iterative stochastic algorithm with moment assumptions.
result First recovery of true labels under data-poisoning attacks.
Toward enabling next-generation robots capable of socially intelligent interaction with humans, we present a computationalmodel of interactions in a social environment of multiple agents and multiple groups. The Multiagent Group Perception and Interaction (MGpi) network is a deep neural network that predi…
Federated Learning with L0 constraint improves sparsity and performance.
problem Inherent sparsity in data and models leads to dense models with poor generalizability.
method L0 constraint on model density achieved through probabilistic gates and federated stochastic gradient descent.
result Achieves target sparsity (rho) in FL with minimal loss in statistical performance.
Architecture optimization, which is a technique for finding an efficient neural network that meets certain requirements, generally reduces to a set of multiple-choice selection problems among alternative sub-structures or parameters. The discrete nature of the selection problem, however, makes this optimization difficu…
Sophisticated gated recurrent neural network architectures like LSTMs and GRUs have been shown to be highly effective in a myriad of applications. We develop an un-gated unit, the statistical recurrent unit (SRU), that is able to learn long term dependencies in data by only keeping moving averages of statistics. The SR…
For any positive integer k, there exist neural networks with Θ(k3) layers, Θ(1) nodes per layer, and Θ(1) distinct parameters which can not be approximated by networks with O(k) layers unless they are exponentially large --- they must possess Ω(2k) nodes. This result is proved here for a class o…
New method uses temperature to control sparse MoE convergence rates.
problem Sparse MoE convergence rates are slow due to temperature interactions.
method Proposes a novel activation gate to improve convergence rates.
result Improved convergence rates to polynomial rates via novel gate.
Study stability of selective SSMs with discontinuous gating.
problem Challenges in stability analysis of selective SSMs with discontinuous gating.
method Passivity and Input-to-State Stability (ISS) analysis of continuous-time selective SSMs.
result Derivation of sufficient conditions for global ISS with respect to the port input.
Gating is a key technique used for integrating information from multiple sources by long short-term memory (LSTM) models and has recently also been applied to other models such as the highway network. Although gating is powerful, it is rather expensive in terms of both computation and storage as each gating unit uses a…
DSelect-k improves MoE models for multi-task learning with better performance and smoother training.
problem Smoothness and convergence issues in sparse gate selection for MoE models.
method Developed DSelect-k, a differentiable and sparse gate for MoE models.
result DSelect-k achieves statistically significant improvements in prediction and expert selection over Top-k.
A new method prunes neural networks efficiently without losing effectiveness.
problem Efficient pruning of neural networks without sacrificing performance.
method Deterministic approximation of binary gates and L0 regularization. result Pruning neural networks significantly without loss in effectiveness.
New non-semisimple Ising anyons enable robust universal quantum computation.
problem Limitation of semisimple theories in universal topological quantum computation.
method Developed non-semisimple Ising anyon model with new anyon types indexed by α. result Robust universality of braiding persists over an open interval of α. We improve MoE models for classification with rigorous guarantees and practical methods.
problem Limited guarantees for stable maximum-likelihood training and model selection in softmax-gated MoE models.
method Derived a batch MM algorithm with closed-form updates, proved finite-sample rates, and developed a dendrogram selector.
result Achieved near-parametric optimal rates for parameter recovery and improved accuracy over baselines.
This work interprets GELU and related activations via a first-order loss function.
problem Understanding and optimizing activation functions in neural networks.
method Complementary interpretation using the Gaussian first-order loss function.
result Calibrated or learned uniform-threshold gates are competitive and often outperform GELU, ReLU, and SiLU/Swish.
New method measures treatment effects across different groups.
problem Understanding treatment effects across subgroups while accounting for covariates.
method Proposes BGATE, a new parameter for balanced group average treatment effect.
result Demonstrates usefulness of BGATE in estimating treatment heterogeneity.
Study examines Lasso performance in high-dimensional MoE models.
problem Estimating MoE models in high-dimensional settings with Lasso.
method Investigates SGMoE models with Lasso regularization under mild assumptions.
result Provides non-asymptotic bounds for Lasso regularization parameter.
New interpretation of RNN forget gate improves learnability for long-term sequential data.
problem Improving learnability of recurrent neural networks for long-term temporal dependencies.
method Generalized theory of gated RNNs, focusing on gradient behavior over time.
result Existing RNNs satisfy the gradient condition for initial training, suggesting validity of forget gate interpretation.
APINNs improve physics-informed neural networks through flexible domain decomposition.
problem Improving physics-informed neural networks (PINNs) for solving partial differential equations (PDEs).
method Introduces a trainable gate network for soft domain decomposition, allowing flexible parameter sharing and improved generalization.
result APINNs significantly improve PINNs and XPINNs, demonstrating better performance on various types of PDEs.