ATOMO reduces communication in distributed learning by sparsifying gradients.
problem Communication overheads in distributed model training.
method ATOMO framework for atomic sparsification of stochastic gradients.
result Sparsifying singular value decomposition can lead to faster distributed training.
Paper tackles anomaly detection in e-commerce using Bayesian semi-supervised tensor decomposition.
problem Detecting anomalies in seller-reviewer data in e-commerce.
method Bayesian semi-supervised tensor decomposition with Polya-Gamma data augmentation and partial natural gradient learning.
result Semi-supervised approach outperforms state-of-the-art unsupervised baselines.
Efficiently decomposes large tensors using stochastic gradients.
problem Efficiently decomposing large tensors for multiway data analysis.
method Stochastic gradients computed via MTTKRP kernel for efficient computation.
result Advantages and scalability demonstrated for large-scale problems.
Wavelet boosts gradient boosting, improving accuracy, especially in imbalanced data.
problem Improving gradient boosting performance, especially in imbalanced data.
method Wavelet decomposition of trees in gradient boosting.
result Wavelet-based gradient boosting outperforms existing methods, especially in imbalanced data.
VecHGrad solves complex tensor decomposition problems more accurately and efficiently.
problem Complex tensor decomposition with multiple matrices and diagonal tensors.
method VecHGrad algorithm using gradient, Hessian-vector product, and adaptive line search.
result VecHGrad converges faster and more accurately than existing methods.
The paper shows how gradient flow on over-parametrized tensor decomposition behaves like deflation.
problem Understanding the training dynamics of gradient flow on tensor decomposition.
method Empirical observation and mathematical proof of gradient flow dynamics for orthogonally decomposable tensors.
result Gradient flow dynamics for orthogonally decomposable tensors follows a tensor deflation process, recovering all tensor components.
Gradient descent can find better tensor decompositions than lazy training in over-parameterized settings.
problem Finding better tensor decompositions in over-parameterized settings.
method Gradient descent on over-parameterized tensor decomposition problems.
result Gradient descent can find an approximate tensor decomposition with rank m=O∗(r2.5llogd), while lazy training requires m=Ω(dl−1). We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allo…
The space of probability densities is an infinite-dimensional Riemannian manifold, with Riemannian metrics in two flavors: Wasserstein and Fisher--Rao. The former is pivotal in optimal mass transport (OMT), whereas the latter occurs in information geometry---the differential geometric approach to statistics. The Rieman…
Develops methods to analyze feature-outcome associations in subpopulations.
problem Challenges in understanding feature-outcome associations in high-dimensional data.
method Geometric decomposition framework using gradient flow and co-monotonicity decomposition.
result Identifies context-dependent patterns and improves statistical power and interpretability.
New recommendations improve Gaussian process accuracy and stability.
problem Numerical instabilities and poor test likelihoods in iterative Gaussian process learning.
method Investigated CG tolerance, preconditioner rank, and Lanczos decomposition rank. Recommended small CG tolerance and large root decomposition size.
result L-BFGS-B optimizer achieves convergence with fewer gradient updates, improving Gaussian process accuracy.
NeCPD improves online tensor decomposition using SGD with Hessian analysis and NAG.
problem Efficiently decompose multi-way tensors in online data processing.
method NeCPD solver based on SGD with Hessian analysis and NAG.
result NeCPD provides more accurate results than existing methods.
Paper characterizes optimization landscape of Tucker decomposition.
problem Finding exact Tucker decomposition is a nonconvex optimization problem.
method Characterized the optimization landscape and provided a local search algorithm.
result All local minima are globally optimal if tensor has an exact Tucker decomposition.
New matrix approximation method using RBF components for better memory efficiency.
problem Efficiently approximate any real matrix without being symmetric or positive definite.
method Formulate as an optimization problem with gradient descent methods.
result Significantly reduces memory usage for various matrix types.
Gradient flow solves optimal mass transport for covariance matrices.
problem Optimal mass transport for covariance matrices.
method Gradient flow on fiber bundle structure.
result Global convergence to polar decomposition.
Paper detects review abuse using tensor decomposition.
problem Detecting review abuse by sellers and reviewers.
method Semi-supervised binary multi-target tensor decomposition.
result The model achieves higher precision and recall.
Paper reconstructs training data from a single gradient query.
problem Privacy threats in federated learning due to model gradients.
method Provable attack using tensor decomposition.
result Training samples can be fully reconstructed from a single gradient query.
Graph neural networks are explained through energy gradient flow and framelet decomposition.
problem Understanding and improving graph neural networks.
method Viewing framelet-based models as gradient flows of energy, proposing a generalized energy via framelet decomposition.
result The proposed model leads to more flexible dynamics, enhancing graph neural networks.
Develops SymGCP for tensor decompositions with general symmetry.
problem Handling symmetry in tensor decompositions for better model accuracy.
method Introduces SymGCP, a generalized CP decomposition that accounts for any subset of tensor modes' symmetry.
result SymGCP enables efficient and scalable tensor decomposition with improved model robustness and accuracy.
Study explains Graph Convolutional Networks decisions.
problem Difficulty in understanding Graph Network decisions.
method Gradient-based and decomposition-based techniques.
result Sets groundwork for future explainability development.
Unified framework for analyzing gradient flows of measures with exponential decay of entropy.
problem Analyzing exponential decay of entropy functionals in gradient flows of measures.
method Characterization of global exponential decay behaviors using Hellinger-Kantorovich geometry, shape-mass decomposition, and Polyak-Łojasiewicz-type inequalities.
result Unified theoretical framework for gradient flows with complete analysis of exponential decay behaviors.
Analyzes explaining nonlinear model predictions.
problem Understanding the contribution of inputs to outputs in nonlinear models.
method Merges integrated gradient and deep Taylor decomposition methods.
result Provides a natural reference point for model at use.
CPOPT-Net predicts sparse client actions in banking using tensor decomposition and neural networks.
problem Predicting sparse client activities in the banking environment with evolving regulations.
method Combines CP tensor decomposition and neural networks for time series predictions.
result CPOPT-Net achieves accurate predictions of clients' financial activities.
Physics-inspired methods optimize SVD compression of LLMs.
problem Efficiently compressing large language models (LLMs) using SVD.
method FermiGrad for globally optimal rank selection and PivGa for lossless compression.
result Global optimization of SVD ranks and lossless compression of low-rank factors.
NOHD optimizes multi-agent systems by decomposing dynamics into irrotational and solenoidal components.
problem Non-stationarity and conflicting interests in multi-agent learning problems.
method NOHD (Newton Optimization on Helmholtz Decomposition) decomposes system dynamics into irrotational and solenoidal components.
result NOHD ensures quadratic convergence in purely irrotational and solenoidal systems and attracts to stable fixed points in general multi-agent systems.
A graph theory approach defines curl and decomposes vector fields.
problem Defining curl for vector fields on graphs and decomposing them.
method Definition of curl as orthogonal complement of circulation-free fields, proving analogues of vector field theorems.
result Helmholtz-Hodge decomposition on graphs: gradient, curl, and harmonic fields.
New splitting theorems in a semi-Riemannian manifold which admits an irrotational vector field (not necessarily a gradient) with some suitable properties are obtained. According to the extras hypothesis assumed on the vector field, we can get twisted, warped or direct decompositions. Some applications to Lorentzian man…
Study finds Deep Taylor Decomposition is unreliable for explaining neural networks.
problem Reliability of Deep Taylor Decomposition for explaining neural networks.
method Investigated the theoretical foundations of Deep Taylor Decomposition (DTD) and found it under-constrained.
result DTD is unreliable because its theoretical foundations are under-constrained and roots do not align with input.
Paper tackles tensor decomposition for unaligned observations using RKHS and novel loss functions.
problem Tackles tensor decomposition for unaligned observations.
method Uses functions in RKHS to represent mode with unaligned observations, introduces versatile loss function, proposes optimization algorithm and stochastic gradient method.
result Demonstrates improved tensor decomposition efficiency and effectiveness with synthetic and real data.
Study shows trained neural networks can overfit without bias or variance issues.
problem Understanding overfitting in trained two-layer ReLU networks.
method Analysis of gradient flow in the neural tangent kernel regime, decomposition of excess risk.
result Trained networks can overfit benignly without bias or variance issues.
Efficient algorithm for Hadamard decomposition of matrices.
problem Decomposing matrices into low-rank factors efficiently.
method Alternating optimization with SVD-inspired initialization and momentum.
result Significantly improved performance compared to existing methods.
ICCNLS models complex relationships as convex and concave components.
problem Complex input-output relationships with affine ambiguity.
method Sub-gradient constrained affine functions, global orthogonality constraints, L1, L2, and elastic net regularisation.
result Improved predictive accuracy and model simplicity compared to conventional methods.
Investigates nearly Kähler and parallel G2 manifolds using Hitchin functionals.
problem Stability analysis of nearly Kähler and parallel G2 manifolds.
method Gradient flow of Hitchin functionals, spectral decomposition of Hessians, Hitchin index.
result Hitchin index provides a lower bound for the Einstein co-index.
Gradient flow solves multi-index regression for high-dimensional Gaussian data.
problem Learning multi-index functions from high-dimensional Gaussian data.
method Two-timescale algorithm with non-parametric link function learning.
result Global convergence of Grassmannian population gradient flow dynamics.
Momentum SGD fails to track nonstationary optima due to drift amplification.
problem Tracking nonstationary optima in stochastic optimization.
method Theoretical analysis of SGD and momentum variants under strong convexity and smoothness.
result Momentum incurs a drift-amplification penalty that diverges as the momentum parameter approaches 1, leading to systematic lag.
Residual networks maintain stability, allowing deep learning without degradation.
problem Training error increases with network depth.
method Lyapunov stability analysis and convex decomposition of functions.
result Residual networks maintain stable equilibria, preventing overfitting.
Proves smooth critical points of Möbius energy are analytic.
problem Analyticity of critical points of Möbius energy.
method Cauchy's method of majorants and gradient decomposition.
result Smooth critical points of Möbius energy are analytic.
A new method for traffic data imputation considering spatiotemporal correlations.
problem Traffic data imputation, especially for high-level missing scenarios.
method Spatiotemporal regularized Tucker decomposition approach.
result The proposed method outperforms existing methods on real-world traffic datasets.
A new multi-agent learning method improves performance in complex games.
problem Performance gap between MAPG and value-based multi-agent approaches.
method Introduces value function decomposition into multi-agent actor-critic framework for off-policy learning.
result DOP significantly outperforms state-of-the-art multi-agent reinforcement learning algorithms.
Analyticity of critical points for O'Hara's knot energies proved.
problem Analyzing the regularity of critical points for O'Hara's knot energies.
method Cauchy's method of majorants and a Möbius energy-inspired gradient decomposition.
result Smooth critical points of O'Hara's knot energies are analytic.
New bounds found for optimizing non-convex functions with noisy data.
problem Limits of first-order stochastic optimization in non-convex settings.
method Divergence decomposition to construct challenging subclasses.
result Sharp lower bounds on noisy gradient queries for various non-convex classes.
Continuous optimization is an important problem in many areas of AI, including vision, robotics, probabilistic inference, and machine learning. Unfortunately, most real-world optimization problems are nonconvex, causing standard convex techniques to find only local optima, even with extensions like random restarts and …
Study of Ricci-Yamabe solitons on Walker 3-manifolds.
problem Characterizing Ricci-Yamabe solitons on Walker 3-manifolds.
method Using Hodge decomposition of De-Rham, the soliton field is found from the potential function.
result Classification of all Ricci-Yamabe and gradient Ricci-Yamabe solitons in a Walker 3-manifold.
We prove that a four-dimensional gradient shrinking Ricci soliton with δW±=0 is either Einstein, or a finite quotient of S3×R, S2×R2 or R4. We also prove that a four-dimensional cscK gradient Ricci soliton is either Kähler-Einstein, or a finite quotient of $M\times\…
Gradient EM converges globally for over-parameterized Gaussian mixtures.
problem Recovering ground truth Gaussian mixtures with over-parameterized models.
method Gradient EM with over-parameterization, using Hermite polynomials and tensor decomposition.
result Gradient EM globally converges to ground truth with n=Ω(mlogm) over-parameterization. Optimization problems with rank constraints arise in many applications, including matrix regression, structured PCA, matrix completion and matrix decomposition problems. An attractive heuristic for solving such problems is to factorize the low-rank matrix, and to run projected gradient descent on the nonconvex factoriz…
New conditions for GRW space-times to be perfect-fluid space-times.
problem Conditions for GRW space-times to be perfect-fluid.
method Gray's decomposition of the gradient of the Ricci tensor, determining Ricci tensor forms in invariant subspaces.
result For most GRW space-times, the Ricci tensor is Einstein or perfect fluid.
We study the stochastic Riemannian gradient algorithm for matrix eigen-decomposition. The state-of-the-art stochastic Riemannian algorithm requires the learning rate to decay to zero and thus suffers from slow convergence and sub-optimal solutions. In this paper, we address this issue by deploying the variance reductio…