Bayesian method for semi-structured models accounts for both types of uncertainty.
problem Lack of work on epistemic uncertainty in semi-structured regression models.
method Bayesian approximation with subspace inference for joint posterior sampling.
result Validated approach recovers structured effect posteriors and approaches full-space posterior.
This work simplifies Bayesian inference for neural networks by identifying influential parameter directions.
problem High computational complexity in Bayesian inference for neural networks due to high-dimensional parameter space.
method Constructing an active subspace of influential parameter directions to reduce dimensionality.
result Effective and scalable Bayesian inference achieved via reduced active subspace.
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we …
Sparse subspace clustering (SSC) is an elegant approach for unsupervised segmentation if the data points of each cluster are located in linear subspaces. This model applies, for instance, in motion segmentation if some restrictions on the camera model hold. SSC requires that problems based on the l1-norm are solved …
Develops a method for causal inference with noisy confounders.
problem Noisy measurements of confounders in treatment effects models.
method Local principal subspace approximation combining K-nearest neighbors matching and PCA.
result Estimators of treatment effects and counterfactual distributions are constructed.
SSVI efficiently trains sparse Bayesian neural networks with minimal compression and performance loss.
problem Efficiently training Bayesian neural networks with uncertainty quantification.
method SSVI optimizes a sparse subspace basis selection and its parameters alternately, guided by weight distribution statistics.
result SSVI achieves significant compression (10-20x model size reduction) with minimal performance drop (under 3%) and FLOPs reduction (up to 20x) compared to dense Variational Inference.
The paper extends hypothesis testing to non-diagonalizable matrices, improving network statistics inference.
problem Testing on non-diagonalizable matrices for network statistics.
method Generalizes Wald and t-tests to non-symmetric matrices, controlling convergence rates.
result Improved inference on network statistics from directed networks.
BO method identifies sparse subspaces for efficient high-dimensional optimization.
problem Efficient optimization of high-dimensional black-box functions.
method Sparse Gaussian process surrogate models on axis-aligned subspaces with Hamiltonian Monte Carlo inference.
result SAASBO achieves excellent performance on synthetic and real-world problems.
Proposes a new algorithm to estimate invariant subspaces across multilayer networks.
problem Estimating invariant subspaces across heterogeneous multiple networks.
method Bias-corrected joint spectral embedding algorithm that recursively calibrates diagonal bias and iteratively updates the subspace estimator.
result Established entrywise subspace perturbation bound and entrywise eigenvector central limit theorem for the algorithm.
Paper projects GP basis functions using tensor networks to reduce complexity.
problem Efficiently approximating Gaussian process regression with a large number of basis functions.
method Develops a method using tensor networks to approximate GP regression with an exponential number of basis functions without exponential computational complexity.
result Shows efficient GP regression on an 18-dimensional benchmark data set.
Novel approach embeds loss tunnels in neural networks, revealing insights into their structure.
problem Understanding the structure of neural network loss surfaces, especially low-loss tunnels.
method Directly embedding loss tunnels into the loss landscape of neural networks.
result Improved insights into the length and structure of loss tunnels, and better subspace inference in Bayesian neural networks.
Two types of nonidentifiability in latent position graphs identified and characterized.
problem Identifying and characterizing nonidentifiability in latent position random graph models.
method Defined and examined subspace nonidentifiability and model-based nonidentifiability, providing examples and characterizing limits.
result Characterized the limits of model-based nonidentifiability and obtained additional limiting results for specific graph models.
Paper develops methods for PCA inference with missing data and heteroskedastic noise.
problem Constructing confidence regions for PCA in high dimensions with missing data and heteroskedastic noise.
method Proposes HeteroPCA and develops non-asymptotic distributional guarantees for valid inference.
result Valid inference on principal subspace and spiked covariance matrix with missing data.
SIG model identifies invariant variables for MSDA with fewer domain constraints.
problem Challenges in enforcing minimal changes across domains for MSDA.
method Subspace identification theory and variational inference.
result SIG model outperforms existing techniques on various benchmark datasets.
We propose a framework for solving high-dimensional Bayesian inference problems using \emph{structure-exploiting} low-dimensional transport maps or flows. These maps are confined to a low-dimensional subspace (hence, lazy), and the subspace is identified by minimizing an upper bound on the Kullback--Leibler divergence …
Modern inference and learning often hinge on identifying low-dimensional structures that approximate large scale data. Subspace clustering achieves this through a union of linear subspaces. However, in contemporary applications data is increasingly often incomplete, rendering standard (full-data) methods inapplicable. …
Transformers infer tasks from context via two modes, geometrically shaped task vectors explain their behavior.
problem Understanding how transformers infer tasks from context and the geometric properties of task vectors.
method Synthetic setting to train small transformers, mathematical characterization of task-vector geometry and inference modes.
result Task-vector geometry shapes in-distribution and out-of-distribution behavior of transformers.
Quantum Fourier Transform aids machine learning inference.
problem Generalizing from finite data samples to ground truth.
method Inspired by quantum algorithms, uses Quantum Fourier Transform to expose invariant subspace for data comparison.
result Proposes a concrete implementation for machine learning applications leveraging symmetries.
The random subspace method, known as the pillar of random forests, is good at making precise and robust predictions. However, there is not a straightforward way yet to combine it with deep learning. In this paper, we therefore propose Neural Random Subspace (NRS), a novel deep learning based random subspace method. In …
A robust visual tracking system requires an object appearance model that is able to handle occlusion, pose, and illumination variations in the video stream. This can be difficult to accomplish when the model is trained using only a single image. In this paper, we first propose a tracking approach based on affine subspa…
Paper introduces Prob-SSI for robust OMA in noisy data.
problem Challenges in estimating modal parameters from noisy data.
method Probabilistic formulation of SSI, robust Prob-SSI algorithm.
result Robust Prob-SSI outperforms conventional SSI in corrupted data.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
Transfer knowledge from multiple sources to improve matrix completion.
problem Matrix completion with noisy data.
method Aggregating singular subspaces information from multiple sources to solve a two-way PCA problem and transform into a low-dimensional linear regression.
result Guaranteed statistical efficiency in transforming the high-dimensional target matrix completion problem.
Researchers use active subspaces to quantify uncertainty in deep generative models for molecular design.
problem Uncertainty quantification in deep generative models for molecular design due to high parameter space.
method Leveraging active subspaces to approximate posterior distribution over low-dimensional parameters.
result The proposed UQ scheme effectively estimates epistemic uncertainty in high-dimensional parameter space without altering model architecture.
In many real-world problems, we are dealing with collections of high-dimensional data, such as images, videos, text and web documents, DNA microarray data, and more. Often, high-dimensional data lie close to low-dimensional structures corresponding to several classes or categories the data belongs to. In this paper, we…
A new diffusion sampling method combines Krylov subspace and diffusion models for faster and more efficient inverse problems.
problem Efficiently solving large-scale inverse problems in high-performance computing.
method Proposes a novel diffusion sampling strategy that integrates Krylov subspace methods with diffusion models.
result Demonstrates significant speedup (80x faster inference time) and improved reconstruction quality on real-world medical imaging problems.
The assumption of positivity in causal inference (also known as common support and co-variate overlap) is necessary to obtain valid causal estimates. Therefore, confirming it holds in a given dataset is an important first step of any causal analysis. Most common methods to date are insufficient for discovering non-posi…
Test-time training adapts a pretrained model to each prompt via parameter updates, improving accuracy under pretraining-to-test distribution shifts.
problem Improving accuracy of pretrained models under distribution shifts.
method Explaining TTT behavior through a decision-theoretic lens.
result TTT reduces prediction error when updates are spectrally matched to the prompt's signal-to-noise ratio and aligned with query-relevant eigen-directions.
Networked sensing, where the goal is to perform complex inference using a large number of inexpensive and decentralized sensors, has become an increasingly attractive research topic due to its applications in wireless sensor networks and internet-of-things. To reduce the communication, sensing and storage complexity, t…
A new method detects outliers using ensembles of Dirichlet process mixtures.
problem Challenges in unsupervised outlier detection using Dirichlet process mixtures.
method Ensembles of Dirichlet process Gaussian mixtures with random subspace and subsampling.
result Empirically outperforms existing approaches in unsupervised outlier detection.
This work improves understanding of projection robust optimal transport distances.
problem Understanding the behavior of minimum Wasserstein estimators in high-dimensional and misspecified models.
method Adopting projection robust (PR) optimal transport, establishing statistical properties, proposing IPRW distance, and providing asymptotic guarantees.
result Established fundamental statistical properties and proposed new distances that outperform Wasserstein distances empirically.
We introduce the probabilistic sequential matrix factorization (PSMF) method for factorizing time-varying and non-stationary datasets consisting of high-dimensional time-series. In particular, we consider nonlinear Gaussian state-space models where sequential approximate inference results in the factorization of a data…
CPCR mitigates bias in PCR for overparameterized models.
problem Bias in Principal Component Regression (PCR) for overparameterized models.
method Calibrated Principal Component Regression (CPCR) learns a low-variance prior in the PC subspace and calibrates the model in the original feature space.
result CPCR outperforms standard PCR in overparameterized settings, improving prediction across multiple problems.
A new method for SVGD reduces variance in high dimensions.
problem High-dimensional variance in SVGD.
method Grassmann Stein Variational Gradient Descent (GSVGD) projects onto arbitrary subspaces and uses coupled Grassmann-valued diffusion.
result GSVGD explores high-dimensional problems with intrinsic low-dimensional structure efficiently.
Extracting an understanding of the underlying system from high dimensional data is a growing problem in science. Discovering informative and meaningful features is crucial for clustering, classification, and low dimensional data embedding. Here we propose to construct features based on their ability to discriminate bet…
We present the Bayesian Case Model (BCM), a general framework for Bayesian case-based reasoning (CBR) and prototype classification and clustering. BCM brings the intuitive power of CBR to a Bayesian generative framework. The BCM learns prototypes, the "quintessential" observations that best represent clusters in a data…
The curse of dimensionality is a longstanding challenge in Bayesian inference in high dimensions. In this work, we propose a projected Stein variational gradient descent (pSVGD) method to overcome this challenge by exploiting the fundamental property of intrinsic low dimensionality of the data informed subspace stemmin…
A new estimator, OddSHAP, simplifies Shapley value computation by focusing on odd components.
problem Efficient computation of Shapley values in machine learning.
method Proved Shapley value depends on odd components, proposed OddSHAP for polynomial regression on odd subspace.
result OddSHAP achieves state-of-the-art estimation accuracy with reduced sampling.
Multi-output Gaussian processes (MOGPs) leverage the flexibility and interpretability of GPs while capturing structure across outputs, which is desirable, for example, in spatio-temporal modelling. The key problem with MOGPs is their computational scaling O(n3p3), which is cubic in the number of both inputs n (e…
uMoE trains NNs with uncertain data by embedding uncertainty into training.
problem Managing aleatoric uncertainty in NN-based predictive models.
method Divide and Conquer strategy, Expert components, Gating Unit.
result uMoE outperforms baseline methods in uncertainty management.
For many modern applications in science and engineering, data are collected in a streaming fashion carrying time-varying information, and practitioners need to process them with a limited amount of memory and computational resources in a timely manner for decision making. This often is coupled with the missing data pro…
A new method for Bayesian inference tackles high-dimensional problems.
problem Bayesian inference in high-dimensional settings with kernel density estimation issues.
method Projected Wasserstein gradient descent (pWGD) method to overcome curse of dimensionality.
result pWGD method effectively addresses high-dimensional Bayesian inference problems.
Geometric methods solve sampling, optimisation, inference, and adaptive decision-making.
problem Efficient solutions for sampling, optimisation, inference, and adaptive decision-making.
method Derive algorithms exploiting geometric structures of Hamiltonian systems, Hilbertian subspaces, and information geometry.
result Wide range of geometric theories emerge in these fields, enabling efficient solutions.
Develops a new framework for analyzing MFVI algorithms.
problem Analyzes mean field variational inference (MFVI) formulations.
method Inspired by variational Bayesian formulations, represents MFVI problem in three ways: gradient flow, Fokker-Planck-like equations, and diffusion process.
result Establishes rigorous guarantees for convergence of time-discretized coordinate ascent variational inference algorithms.
Paper bounds subspace estimator error from noisy projections.
problem Estimating subspaces from noisy data.
method Derives perturbation bound on optimal subspace estimator.
result Fundamental result with implications in matrix completion and clustering.
We reframe linear dimensionality reduction as a problem of Bayesian inference on matrix manifolds. This natural paradigm extends the Bayesian framework to dimensionality reduction tasks in higher dimensions with simpler models at greater speeds. Here an orthogonal basis is treated as a single point on a manifold and is…
PCA is one of the most widely used dimension reduction techniques. A related easier problem is "subspace learning" or "subspace estimation". Given relatively clean data, both are easily solved via singular value decomposition (SVD). The problem of subspace learning or PCA in the presence of outliers is called robust su…
Union of Subspaces (UoS) is a popular model to describe the underlying low-dimensional structure of data. The fine details of UoS structure can be described in terms of canonical angles (also known as principal angles) between subspaces, which is a well-known characterization for relative subspace positions. In this pa…