Study uses spectral risk for learning with heavy-tailed data.
problem Learning with heavy-tailed loss distributions.
method Spectral risk with Lipschitz-continuous density, derivative-free learning.
result Excess risk guarantees and improved performance over traditional methods.
Bayesian parametric matrix models provide uncertainty quantification for spectral learning.
problem Uncertainty quantification in spectral learning for safety-critical applications.
method Bayesian parametric matrix models (B-PMMs) that extend PMMs to provide uncertainty estimates.
result B-PMMs achieve exceptional uncertainty calibration (ECE < 0.05) while maintaining favorable scaling.
We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization. Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics. As such, the…
Spectral deconfounding improves machine learning models by reducing hidden confounding effects.
problem Machine learning models can be misled by hidden confounders, leading to unreliable predictions.
method Develops a nonlinear spectral deconfounding framework for gradient boosting that modifies boosting dynamics to slow down in confounding-aligned directions.
result Spectrally deconfounded boosting improves estimation of the target function under hidden confounding and is more scalable.
Root Laplacian Eigenmaps help in spectral embedding of graphs.
problem Efficient spectral embedding of graphs.
method Square root of graph-Laplacian operator.
result Improved spectral embedding techniques.
A new convolutional spectral kernel network learns hierarchical and local features.
problem Lack of deep learning in non-stationary spectral kernels.
method Introduces convolutional filters and deep architectures into non-stationary spectral kernels, derives generalization error bounds, and introduces regularizers.
result Validated the effectiveness of the convolutional spectral kernel network on real-world datasets.
Learning meaningful graphs from data plays important roles in many data mining and machine learning tasks, such as data representation and analysis, dimension reduction, data clustering, and visualization, etc. In this work, for the first time, we present a highly-scalable spectral approach (GRASPEL) for learning large…
A new method learns outcome-aware spectral features for causal effect estimation.
problem Estimation of causal effects in the presence of hidden confounders.
method Augmented Spectral Feature Learning framework that minimizes a contrastive loss derived from an augmented operator incorporating outcome information.
result Our method remains effective even under spectral misalignment.
Survey of spectral, probabilistic, and deep metric learning methods.
problem Developing effective distance metrics for various machine learning tasks.
method Divided into spectral, probabilistic, and deep approaches, covering various techniques and their applications.
result Comprehensive overview of metric learning methods, including new developments and applications.
Quantum computers can enhance spectral methods in machine learning.
problem Spectral methods are fundamental but challenging for classical models.
method Utilizing quantum Fourier Transform for spectral manipulations.
result Quantum computing can offer more efficient spectral design.
This study bridges the gap between spatial and spectral GNNs.
problem Lack of direct comparison and cross-reference of existing GNNs.
method Systematically categorizes and examines GNNs into spatial and spectral domains.
result Establishes a strong relationship between spatial and spectral GNNs.
A new method for nonstationary Gaussian processes using Fourier features.
problem Efficient simulation of nonstationary Gaussian processes with high-dimensional distributions.
method Discretizes the spectral representation of nonstationary processes, avoiding probability measure assumptions.
result An efficient low-rank approximation of nonstationary spectral densities, consistent and positive semi-definite.
Spectral feature learning improves IV regression for causal effect estimation.
problem Estimating causal effects in the presence of hidden confounders.
method Two-stage least squares estimator based on spectral features.
result Performance of the method depends on strong spectral alignment and slow eigenvalue decay.
SPEDER extracts state-action abstraction from dynamics for reinforcement learning.
problem Curse of dimensionality and limited applicability of spectral methods.
method Spectral Decomposition Representation (SPEDER) that extracts state-action abstraction from dynamics without policy dependence.
result Theoretical analysis establishes sample efficiency in online and offline settings.
Proposes a Gaussian process for graph signals using adaptive spectral kernels.
problem Predicting signals on graph nodes with various structures.
method Spectral kernel learning approach that incorporates a polynomial function in the graph spectral domain.
result The model accurately recovers ground truth spectral filters and outperforms in real-world graph data.
In spectral clustering and spectral image segmentation, the data is partioned starting from a given matrix of pairwise similarities S. the matrix S is constructed by hand, or learned on a separate training set. In this paper we show how to achieve spectral clustering in unsupervised mode. Our algorithm starts with a se…
We introduce a new parameterization method for deep learning layers using spectral tensor train decomposition.
problem Efficiency and stability in deep learning models with weight matrix compression.
method Spectral Tensor Train Parameterization (STTP) of weight matrices.
result Improved compression and training stability in neural networks.
Laplacian Eigenvectors of the graph constructed from a data set are used in many spectral manifold learning algorithms such as diffusion maps and spectral clustering. Given a graph constructed from a random sample of a d-dimensional compact submanifold M in RD, we establish the spectral convergence rate…
Graph convolutional networks fail to use eigenvectors beyond the first, unlike spectral embedding.
problem Understanding when graph convolutional networks fail compared to spectral embedding.
method Presented a simple generative model to illustrate failure.
result Graph convolutional networks fail to use eigenvectors beyond the first in certain graphs.
In the past decades, spectral clustering (SC) has become one of the most effective clustering algorithms. However, most previous studies focus on spectral clustering tasks with a fixed task set, which cannot incorporate with a new spectral clustering task without accessing to previously learned tasks. In this paper, we…
SpGAT learns graph representations using spectral attention for efficiency.
problem Efficiently capturing global graph patterns with minimal parameters.
method Introduces Spectral Graph Attention Network (SpGAT) using spectral domain attention mechanisms and a fast Chebychev approximation.
result SpGAT achieves better global pattern recognition with fewer parameters compared to GAT.
New method extrapolates spectral densities from smaller models to larger ones.
problem Limited practical computations for large machine learning models.
method Algebraic spectral curve theory for free decompression.
result Framework enables extrapolation of spectral densities with multiple or multi-modal bulks.
Study spectral learning for odeco tensors, addressing initialization bottlenecks.
problem Recovering orthogonally decomposable tensors under noise.
method Investigates perturbation bounds, non-convex optimization, and initialization strategies.
result Initialization is the main bottleneck for efficient algorithms.
The generalization performance of kernel methods is largely determined by the kernel, but common kernels are stationary thus input-independent and output-independent, that limits their applications on complicated tasks. In this paper, we propose a powerful and efficient spectral kernel learning framework and learned ke…
Introduces Spectral Graph Network combining spatial and spectral message passing.
problem Relational reasoning in graph structured data.
method Applies message passing to both spatial and spectral domains of a graph.
result Promotes efficient training with fewer iterations and robustness to edge dropout.
Unified framework for spectral methods, kernel learning, and manifold unfolding.
problem Tackles the unification and optimization of spectral dimensionality reduction methods.
method Unified spectral methods as kernel PCA, kernel learning by SDP, and detailed explanation of MVU variants.
result Unified understanding and optimization of manifold learning techniques.
Unified framework explains why overfitting is benign in interpolating learning.
problem Understanding why overfitting is benign in highly overparameterized models.
method Spectral-transport stability framework.
result Sharp benign-overfitting criterion and explicit phase-transition rates.
Paper explores SNN for learning spectral geometric info from data.
problem Challenges in applying traditional eigensolvers to big data.
method Introduces Spectral Neural Networks (SNN) as an alternative.
result Investigates tradeoffs and optimization landscape of SNN.
Efficiently scales continuous kernels with sparse Fourier domain learning.
problem High computational and memory demands, spectral bias in continuous kernels.
method Sparse learning in the Fourier domain.
result Efficient scaling of continuous kernels, reduced computational and memory requirements, mitigated spectral bias.
This paper improves spectral clustering for large datasets using the Nystrom method.
problem Spectral clustering's scalability issues with large datasets.
method A principled spectral clustering algorithm exploiting Nystrom approximation's spectral properties.
result Improved spectral clustering efficiency and accuracy compared to existing methods.
Spectral regularization improves learning over combinatorial spaces with limited data.
problem Learning pseudo-Boolean functions with scarce labeled data.
method Regularizing the spectral representation of learned functions using the L_1 norm.
result Regularization allows for data-frugal learning and achieves statistically optimal generalization performance.
Spectral clustering identifies clusters of multivariate extremes.
problem Analyzing the dependence structure of multivariate extremes.
method Spectral clustering based on a random k-nearest neighbor graph. result Spectral clustering can consistently identify clusters of multivariate extremes under certain conditions.
FNOs learn solution operators of dissipative equations efficiently via spectral methods.
problem Learning and approximation of solution operators for dissipative equations.
method Introducing spectral methods and deriving FNO approximation bounds and sample complexity guarantees.
result Polynomial sample complexity guarantees for FNOs learning solution operators of dissipative equations.
Algorithm learns dynamics from past observations.
problem Learning a nonlinear dynamical system.
method Spectral filtering, online convex optimization.
result Vanishing prediction error for marginally stable systems.
We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spe…
A large number of algorithms in machine learning, from principal component analysis (PCA), and its non-linear (kernel) extensions, to more recent spectral embedding and support estimation methods, rely on estimating a linear subspace from samples. In this paper we introduce a general formulation of this problem and der…
Deep learning speeds spectral density estimation for large 2D/3D grids.
problem Computational challenges in estimating spectral densities for large grids.
method Deep learning neural network for spectral density estimation.
result Deep learning estimator is a universal approximator and faster than existing methods.
IGT learns graph representations without supervision.
problem Building deep unsupervised graph representations.
method Generic complex-valued spectral graph architecture from Fourier transform generalization, greedy concave objective for discriminative and invariant features.
result IGT learns both discriminative and invariant features from graph topology.
Supervised topic models simultaneously model the latent topic structure of large collections of documents and a response variable associated with each document. Existing inference methods are based on variational approximation or Monte Carlo sampling, which often suffers from the local minimum defect. Spectral methods …
Paper revisits graph-CNNs using Laplace-Beltrami spectral filters and polynomials.
problem Improving spectral graph convolutional neural networks (graph-CNNs).
method Developed Laplace-Beltrami CNN (LB-CNN) by replacing graph Laplacian with LB operator and approximating spectral filters using Chebyshev, Laguerre, and Hermite polynomials.
result Classification accuracy of LB-CNN is not dependent on the type of polynomials or operators.
New neural architectures invariant to sign flips and basis symmetries for graph representation learning.
problem Learning invariant graph representations from eigenvectors.
method SignNet and BasisNet neural architectures that are invariant to sign flips and basis symmetries.
result Proven to be universal, approximating any continuous function of eigenvectors with desired invariances.
Deep learning can learn compositional functions more efficiently by breaking them into stages.
problem Understanding why deep learning performs better than shallow models in learning compositional functions.
method Analyzed learnability of compositional target functions using a three-layer fitting model trained with layer-wise spectral estimators.
result Learning compositional functions can be simplified by breaking them into stages, reducing the complexity of the learning problem.
Spectral learning extends matrix methods to tensors for better latent variable modeling.
problem Limitations of matrix-based spectral methods in capturing non-Gaussian data.
method Extend spectral decomposition to tensor-based methods for higher-order moments.
result Tensor decomposition can identify latent effects missed by matrix methods.
Graph spectral techniques for measuring graph similarity, or for learning the cluster number, require kernel smoothing. The choice of kernel function and bandwidth are typically chosen in an ad-hoc manner and heavily affect the resulting output. We prove that kernel smoothing biases the moments of the spectral density.…
New method trains neural networks in spectral domain for improved performance.
problem Training deep neural networks in the space of nodes.
method Trains neural networks in the spectral domain, modifying eigenvalues and eigenvectors of transfer operators.
result Superior performance compared to standard methods, especially when adjusting eigenvalues.
We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…
Study confirms learning rates for vector-valued spectral algorithms, proving consistency.
problem Theoretical confirmation of learning rates for vector-valued spectral algorithms.
method Rigorous analysis of learning rates for various vector-valued spectral algorithms, including kernel ridge regression and gradient descent.
result Upper and lower bounds on learning rates for vector-valued spectral algorithms, proving minimax optimality in various scenarios.
New equivariant filters improve graph classification.
problem Designing deep learning models for graph symmetries.
method Nonlinear spectral filters (NLSFs) that are equivariant to graph functional shifts.
result NLSFs outperform existing spectral GNNs in graph classification.