LSTMs are explained as a weighted sum of context-independent functions.
problem Vanishing gradients in simple RNNs.
method Decouple LSTM gates from simple RNNs, computing an element-wise weighted sum of context-independent functions.
result Gating mechanism alone performs as well as an LSTM in most settings, suggesting more representational power.
Ranking is a key aspect of many applications, such as information retrieval, question answering, ad placement and recommender systems. Learning to rank has the goal of estimating a ranking model automatically from training data. In practical settings, the task often reduces to estimating a rank functional of an object …
Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SGMCMC algorithm. Furthermore, we extend recent SG-MCMC methods with two key co…
Bilinear MLPs offer a new way to interpret deep learning models without complex nonlinearities.
problem Lack of mechanistic understanding in how MLPs compute.
method Introduced bilinear MLPs without element-wise nonlinearities, analyzed their weights using tensor and eigendecomposition.
result Bilinear MLPs provide interpretable weight structures and enable adversarial attacks and overfitting analysis.
Tensor algebra compiler calculates derivatives for machine learning.
problem Deriving gradients for tensor functions in machine learning.
method Developed an algorithm to compute derivatives of tensor functions with arbitrary indexing schemes.
result Automatic differentiation for tensor algebra expressions.
We consider the problem of Graphical lasso with an additional ℓ∞ element-wise norm constraint on the precision matrix. This problem has applications in high-dimensional covariance decomposition such as in \citep{Janzamin-12}. We propose an ADMM algorithm to solve this problem. We also use a continuation st…
Paper extends trigonometric summation formula with weights.
problem Trigonometric summation formula by Grigor'yan, Lin and Yau.
method Weighted trigonometric summation formula derivation.
result Extension of trigonometric summation formula.
Modified Newton step for online learning reduces matrix size for large datasets.
problem Handling large multi-class datasets efficiently in online learning.
method Element-wise multiplication to reduce matrix size of second order matrices.
result Proposed method achieves similar mistake rates to popular methods but with faster computations.
Paper develops a new weighted low-rank matrix approximation technique.
problem Matrix completion with missing data.
method Element-wise weighted generalization of low-rank matrix approximation.
result Proposes an algorithm and acceleration techniques for solving the weighted problem.
AReLU uses attention-based rectification to improve neural network performance.
problem Improving neural network performance through better activation functions.
method Integrates attention mechanism with rectified linear unit (ReLU) to learn and scale feature maps.
result AReLU significantly boosts performance of most network architectures with minimal changes.
Polynomial sketch approximates functions of low-rank matrices efficiently.
problem Approximating element-wise functions of low-rank matrices without full access.
method Combining polynomial approximation and tensor sketch for monomials.
result Efficient algorithm with lower complexity than full matrix access.
A new optimization algorithm improves convergence in unconstrained problems.
problem Unconstrained optimization problems.
method Element-wise relaxed scalar auxiliary variable (E-RSAV) algorithm.
result Improved convergence and alignment of modified and original energy.
Paper introduces new approximations for lognormal sums, matching comonotonicity and moments.
problem Approximating sums of lognormal random variables accurately.
method Introduces new approximations based on weighted distribution theory, emphasizing comonotonicity and moment matching.
result Approximations perform better than classical methods, especially in the right tail of the distribution.
We present an autoencoder that leverages learned representations to better measure similarities in data space. By combining a variational autoencoder with a generative adversarial network we can use learned feature representations in the GAN discriminator as basis for the VAE reconstruction objective. Thereby, we repla…
Jones polynomials compute weighted sums of Lefschetz numbers.
problem Computing Lefschetz numbers for braids.
method Colored Jones polynomials of braid closures.
result Jones polynomials compute abelianized Lefschetz numbers.
A fast matrix factorization method for sparse data with non-uniform missing data weights.
problem Sparse and imbalanced data in real-world learning systems.
method Non-uniform weighting of missing data, efficient learning method with truncated SVD and eALS.
result Improved performance in downstream applications compared to uniform weighting.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
Shampoo achieves higher token efficiency than Muon in language models.
problem Understanding the relationship and relative data efficiency of Shampoo and Muon compared to Adam and Signum.
method Extensive experiments on language models, demonstrating Shampoo's higher efficiency and decomposing its updates.
result Shampoo's benefits are attributed to its application to weight matrices, challenging interpretations based on variance adaptation and whitening.
For any matrix A in R^(m x n) of rank ρ, we present a probability distribution over the entries of A (the element-wise leverage scores of equation (2)) that reveals the most influential entries in the matrix. From a theoretical perspective, we prove that sampling at most s = O ((m + n) ρ^2 ln (m + n)) entries of the ma…
The paper develops concentration inequalities for structured random data, extending beyond independent terms.
problem Developing concentration inequalities for structured weighted sums of random data, including tensors and matrix-valued data.
method The paper develops Hoeffding and Bernstein bounds for structured weighted sums under exchangeability, extending beyond the classical framework of independent terms.
result The paper develops a sharper concentration bound for combinatorial sums of matrix arrays.
Sharp bounds for Dirichlet sums lead to improved Bayesian algorithm analysis.
problem Improving Bayesian algorithm performance through precise deviation bounds.
method Novel integral representation of Dirichlet sum density, Gaussian approximation, complex analysis.
result Significantly sharpened regret bounds for Multinomial Thompson Sampling.
Optimizing full likelihoods adapts loss scales and shapes for robust modeling.
problem Rigid loss functions limit model adaptability and robustness.
method Optimize full likelihoods with adjustable parameters.
result Adaptive tuning of loss scales and shapes improves model robustness.
ZeroS improves Transformers by adding negative weights, matching or beating softmax attention.
problem Limited performance of linear attention methods, especially in long context sequences.
method Proposes Zero-Sum Linear Attention (ZeroS) that removes the zero-order term and reweights zero-sum softmax residuals.
result ZeroS matches or exceeds standard softmax attention across various benchmarks, theoretically expanding representable functions.
Develops a Bayesian non-parametric approach for signal separation with varying components.
problem Signal separation with varying components across different input locations.
method Augments Gaussian Process Latent Variable Models with weighted sums of pure component signals and incorporates priors for linear weights.
result Framework allows for non-linear variations in signals and incorporates useful priors for linear weights.
This letter presents an improved version of diffusion least mean ppower (LMP) algorithm for distributed estimation. Instead of sum of mean square errors, a weighted sum of mean square error is defined as the cost function for global and local cost functions of a network of sensors. The weight coefficients are updated b…
G-FIGS uses instance weights to create interpretable models from diverse data.
problem Generalizing to diverse data distributions while maintaining interpretability.
method Estimates group membership probabilities, uses as instance weights in FIGS to grow decision trees.
result Achieves state-of-the-art prediction performance and maintains interpretability.
NESTA accelerates neural networks by compressing Hamming weights.
problem Efficiently computing convolution layers in deep neural networks.
method NESTA reformats convolutions into 3imes3 batches and uses Hamming Weight Compressors to process each batch, approximating partial sums and adding residuals. result Significantly speeds up convolution computations with reduced energy consumption.
Adapting deep learning for object detection to detect mixed image tampering.
problem Detecting mixed image tampering without prior knowledge of the method.
method Adapting deep learning for object detection to learn from a large database of various image types, using a Multi-stream Faster RCNN network with fused features from ELA and BAG error maps.
result Improved accuracy in detecting mixed image tampering.
Weibull framework diagnoses transformer weight distributions, revealing distinct patterns across modules.
problem Diagnosing weight distributions in transformer models for better understanding of training dynamics.
method Applied Weibull distribution to diagnose weight magnitude distributions in transformer models, fitting each weight matrix independently.
result Distinct patterns of weight distributions across different transformer modules were identified.
This paper proposes a boosting-based solution addressing metric learning problems for high-dimensional data. Distance measures have been used as natural measures of (dis)similarity and served as the foundation of various learning methods. The efficiency of distance-based learning methods heavily depends on the chosen d…
New method preserves privacy by aggregating feature-vectors with weighted sums, ensuring label differential privacy.
problem Ensuring privacy in training data aggregation for sensitive labels.
method Learning from bag aggregates (LBA) with weighted Gaussian sums, preserving label differential privacy (label-DP).
result Weighted LBA using iid Gaussian weights with m randomly sampled disjoint k-sized bags provides (ε,δ)-label-DP. We study a data model in which the data matrix D can be expressed as D = L + S + C, where L is a low rank matrix, S an element-wise sparse matrix and C a matrix whose non-zero columns are outlying data points. To date, robust PCA algorithms have solely considered models with either S or C, but not both. As such, existi…
This is the fourth article of our series. Here, we study weighted norm inequalities for the Riesz transform of the Laplace-Beltrami operator on Riemannian manifolds and of subelliptic sum of squares on Lie groups, under the doubling volume property and Gaussian upper bounds.
New invariants for RNA foldings and stuck links defined.
problem Defining invariants for RNA foldings and stuck links.
method Assigning Boltzmann weights at classical and stuck crossings.
result Explicit computations of new invariants provided.
DeepHoyer introduces differentiable, scale-invariant sparsity measures for neural networks.
problem Efficiently sparsifying neural networks with scale-invariant sparsity measures.
method Developed DeepHoyer, a set of differentiable, scale-invariant sparsity-inducing regularizers based on the Hoyer measure.
result DeepHoyer produces sparser neural networks than previous methods, maintaining similar accuracy.
New RDPC dissimilarity measure improves time series clustering.
problem Improving time series clustering methods for diverse data.
method Combining weighted Pearson correlation with largest element-wise differences.
result RDPC outperforms existing methods in complex datasets.
Paper proposes DR-NMF for robust NMF with unknown noise model.
problem Robust NMF for unknown noise model.
method Multi-objective NMF with weighted-sum approach, DR-NMF via Frank-Wolfe algorithm.
result DR-NMF is robust to unknown noise model.
Develops algorithms for multi-way similarity clustering in hypergraphs.
problem Challenges of spectral clustering in multi-way similarity settings.
method Hypergraph Spectral Clustering (HSC) and Hypergraph Spectral Clustering with Local Refinement (HSCLR).
result Achieves optimal performance under the weighted stochastic block model.
In a seminal paper Abadie, Diamond, and Hainmueller [2010] (ADH), see also Abadie and Gardeazabal [2003], Abadie et al. [2014], develop the synthetic control procedure for estimating the effect of a treatment, in the presence of a single treated unit and a number of control units, with pre-treatment outcomes observed f…
Paper proposes a new time series prediction method using weighted past data and optimization.
problem Predicting time series data with improved accuracy considering both deterministic and stochastic assumptions.
method The approach uses a weighted sum of past data, solving a constrained linear optimization problem to minimize an outer bound of prediction error.
result The method can outperform existing non-parametric methods in short-term forecasts.
Study symmetry groups and curves from sums of exponentials.
problem Understanding the geometry and symmetry of curves from sums of exponentials.
method Analysis of symmetry groups, winding numbers, and parametrization of the unit circle.
result Unified method for constructing curves with specific properties.
Sum-of-norms clustering recovers mixtures of Gaussians even with infinite samples.
problem Recovering a mixture of Gaussians from a large number of samples.
method Sum-of-norms clustering with equal weights, convex optimization.
result Sum-of-norms clustering can recover mixtures of Gaussians even as the number of samples tends to infinity.
This paper certifies cluster assignments from sum-of-norms clustering algorithms.
problem Certifying the correct cluster assignments from approximate solutions of sum-of-norms clustering.
method Presented a clustering test that identifies and certifies the correct cluster assignment from an approximate solution.
result The correct cluster assignment is guaranteed to be certified by a primal-dual path following algorithm after sufficient iterations.
Let G be a torus and M a G-Hamiltonian manifold with Kostant line bundle L and proper moment map. Let P be the weight lattice of G. We consider a parameter k and the multiplicity m(λ,k) of the quantized representation associated to M and the k-th power of L . We prove that the weighted sum ∑m(λ,k)f(λ/k) of the…
Developed a Particle-Gibbs sampler for Bayesian feature allocation models.
problem Intractable exact inference in Bayesian feature allocation models.
method Particle-Gibbs sampler for feature allocation matrix updates.
result PG sampler improves performance of feature allocation models.
Weil-Petersson volumes vary continuously with weighted points on a projective line.
problem Continuity of Weil-Petersson volumes in moduli space with weighted points.
method Localization and geometric computation methods.
result CM volume converges to geometric volume as weights approach Calabi-Yau geometry.
We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL dive…
Efficiently estimates covariance for sub-Weibull vectors with sub-Gaussian rate.
problem Outliers in high-dimensional covariance estimation.
method Cross-Fitted Norm-Truncated Estimator for Sub-Weibull distributions.
result Achieves optimal sub-Gaussian rate with O(Nd2) operations.