VRSGT algorithm reduces orthogonality constraints in decentralized optimization.
problem Decentralized optimization with orthogonality constraints.
method VRSGT algorithm with variance reduction and orthogonal techniques.
result VRSGT achieves convergence rate of O(1 / k) for orthogonality constraints.
New algorithms reduce orthogonality constraint enforcement time in machine learning.
problem Efficiently solving orthogonality constraints in machine learning.
method Extending the landing algorithm to Stiefel manifold, incorporating stochastic and variance reduction techniques.
result All proposed methods achieve the same convergence rate as Riemannian counterparts enforcing constraints.
New metrics improve landing algorithms for orthogonality constraints.
problem Optimizing landing algorithms with orthogonality constraints.
method Proposed a family of metrics over full-rank matrices to enhance landing algorithms.
result Natural extension of β-metric improves landing performance.
New method relaxes PCA orthogonality constraints using explained variance of correlated components.
problem Difficulty in using PCA for sparse design due to orthogonality constraints and non-differentiable penalty.
method Introduce expvar(Y) to measure variance explained by correlated components, relax orthogonality constraints.
result Two expvar(Y) definitions suitable for block PCA formulations without orthogonality constraints.
Formula for Laplace-Beltrami on orthogonal group in Euclidean coords.
problem Computing Laplace-Beltrami on constrained submanifolds.
method Embedded gradient vector field method, explicit formula derivation.
result Explicit formula for Laplace-Beltrami on orthogonal group.
DONUT improves treatment effect estimation by enforcing orthogonality constraints.
problem Estimating treatment effects from observational data is challenging due to unobserved outcomes.
method DONUT uses a regularization framework that formalizes unconfoundedness as orthogonality, leading to deep orthogonal networks.
result DONUT outperforms state-of-the-art methods in estimating average treatment effects.
A new algorithm POGO optimizes thousands of orthogonal matrices efficiently.
problem Optimizing thousands of orthogonal constraints at scale is computationally expensive.
method Revisits Landing algorithm, uses modern adaptive optimizers, reduces hyperparameters.
result POGO optimizes thousands of orthogonal matrices in minutes, outperforming alternatives.
We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidea…
A new optimizer preserves orthogonality constraints on matrices efficiently.
problem Optimization on Stiefel manifold with orthogonality constraints.
method Interplay between continuous and discrete dynamics leading to a gradient-based optimizer with momentum.
result The method optimizes matrices on Stiefel manifold efficiently and accurately.
New algorithm tackles optimization with distributed constraints.
problem Optimization problems with generalized orthogonality constraints in a decentralized setting.
method Introduced a novel algorithm that tracks gradients and Jacobians simultaneously.
result Global convergence with an iteration complexity established.
The non-negative matrix factorization (NMF) model with an additional orthogonality constraint on one of the factor matrices, called the orthogonal NMF (ONMF), has been found a promising clustering model and can outperform the classical K-means. However, solving the ONMF model is a challenging optimization problem becau…
An algorithm simplifies optimization with nonnegative and orthogonal constraints.
problem Optimization problems with nonnegative and orthogonal constraints.
method Support-set algorithm exploiting structural sparsity.
result Global convergence to first-order stationary point with iteration complexity O(ε−2). A new method reduces complexity for optimizing large-scale problems with orthogonality constraints.
problem Optimizing large-scale problems with orthogonality constraints.
method Randomized Riemannian submanifold method that restricts updates to random submanifolds.
result Significantly reduces per-iteration complexity for large-scale problems.
New method solves sparse PCA for multiple components efficiently.
problem Sparse PCA for multiple orthogonal components.
method Reformulates orthogonality as rank constraints, uses semidefinite relaxations and bounds.
result Exact solutions with near-optimal variance explained and orthogonality.
SOFARI improves inference on multi-task learning latent factors.
problem Challenges in precise inference on multi-task learning latent factor matrices.
method High-dimensional manifold-based Neyman near-orthogonality inference on Stiefel manifold structure.
result Easy-to-use bias-corrected estimators for latent factor vectors and singular values with asymptotic normal distributions.
Wasserstein-GANs have been introduced to address the deficiencies of generative adversarial networks (GANs) regarding the problems of vanishing gradients and mode collapse during the training, leading to improved convergence behaviour and improved image quality. However, Wasserstein-GANs require the discriminator to be…
New method for sampling on constrained domains using orthogonal-space gradient flow.
problem Sampling on manifolds defined by constraints is challenging.
method Orthogonal-Space Variational Gradient Descent (O-Gradient)
result O-Gradient converges to the target constrained distribution efficiently.
Principal component analysis (PCA) is an unsupervised method for learning low-dimensional features with orthogonal projections. Multilinear PCA methods extend PCA to deal with multidimensional data (tensors) directly via tensor-to-tensor projection or tensor-to-vector projection (TVP). However, under the TVP setting, i…
Formula derived for Laplace-Beltrami on Stiefel manifold.
problem Finding Laplace-Beltrami operator on Stiefel manifold.
method Using the general framework of Laplace operators on constraint manifolds, derived the explicit formula in terms of ambient Euclidean coordinates.
result Extended previously known formulas for sphere and special orthogonal group.
In this paper, we propose a scalable algorithm for spectral embedding. The latter is a standard tool for graph clustering. However, its computational bottleneck is the eigendecomposition of the graph Laplacian matrix, which prevents its application to large-scale graphs. Our contribution consists of reformulating spect…
Novel prior for orthogonal functions improves functional component estimation.
problem Improving orthogonality in functional principal component analysis.
method Sequential adaptive priors for orthogonal functions using hierarchical conditionally normal distributions.
result Proposed prior leads to nearly orthogonal posterior estimates.
Introduces new curvature concept for Kähler manifolds.
problem Optimizing curvature constraints for projective Kähler manifolds.
method Introduces weighted orthogonal Ricci curvature and proves vanishing theorems.
result Proves optimal curvature constraints for projective Kähler manifolds.
Bayesian SPCA method tackles orthogonality constraint with spike and slab prior.
problem Bayesian SPCA method for high-dimensional data with orthogonality constraint.
method Parameter-expanded coordinate ascent variational inference (PX-CAVI) with spike and slab prior.
result PX-CAVI algorithm outperforms existing SPCA approaches in performance.
The paper studies surfaces in a bounded domain with orthogonal boundaries and proves curvature estimates.
problem Estimating the area of surfaces with orthogonal boundaries in a bounded domain.
method Weak formulation of orthogonality for curvature varifolds, classification of vanishing curvature varifolds.
result Existence of an orthogonal 2-varifold that minimizes L2 curvature in the integer rectifiable class. DFSOS improves sparse discriminant analysis for high-dimensional data.
problem Sparse discriminant analysis in high-dimensional settings with feature selection.
method Deflation-Free Sparse Optimal Scoring (DFSOS) using Bregman iteration and orthogonality-constrained optimization.
result DFSOS achieves comparable or better classification accuracy than deflation-based methods.
Proposes FOAGP for efficient orthogonal effect decomposition of black-box computer experiments.
problem Challenges in sensitivity analysis of black-box computer experiments with complex, nonlinear functional outputs.
method Functional-output orthogonal additive Gaussian process (FOAGP) with conditional orthogonality constraint.
result Demonstrates effectiveness in orthogonal effect decomposition and variance decomposition through simulations and real-world application.
New method speeds up solving orthogonality constrained problems.
problem Solving orthogonality constrained problems efficiently.
method Riemannian optimization and Riemannian preconditioning.
result Preconditioning improves computational costs and convergence.
On a constraint manifold we give an explicit formula for the Hessian matrix of a cost function that involves the Hessian matrix of a prolonged function and the Hessian matrices of the constraint functions. We give an explicit formula for the case of the orthogonal group O(n) by using only Euclidean coordinates …
Soft-Radial Projection solves gradient saturation in constrained deep learning.
problem Gradient saturation in deep learning models when integrating hard constraints.
method Introduces Soft-Radial Projection, a differentiable layer that maps predictions onto constraint boundaries without rank-deficient Jacobians.
result Improves convergence and solution quality over state-of-the-art methods.
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However …
New model handles complex output dependence in large datasets.
problem Complex output dependence in large datasets.
method Orthogonal Stochastic Linear Mixing Model (OSLMM) with Markov chain Monte Carlo inference.
result OSLMM reduces prediction error compared to state-of-the-art methods.
Orthogonium offers unified, efficient layers for robust deep learning.
problem Fragmented and computationally demanding implementations of orthogonal and 1-Lipschitz layers.
method Unified, efficient PyTorch library providing orthogonal and 1-Lipschitz layers.
result Reduced overhead and standardized tools for robust experimentation.
A new method for disentangled representations without supervision.
problem Learning disentangled representations in unsupervised learning.
method Constr-DRKM, a deep kernel method with orthogonality constraints.
result Constr-DRKM performs similarly to β-VAE on disentanglement metrics.
Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuristic approaches such as the gradient penalty have seen much practical success, it is challenging to achieve similar practical performance wh…
The study explores rigid constraints on almost Ricci-Bourguignon solitons on contact metric three-manifolds.
problem Investigating constraints on almost Ricci-Bourguignon solitons on contact metric three-manifolds.
method Using a local orthonormal \(\varphi\)-basis, derived the full component form of the almost Ricci-Bourguignon soliton equation.
result For contact metric three-manifolds satisfying \(Qξ=σξ\), a collinear potential field must vanish on the non-Sasakian region whenever \(ξ(σ)=0\).
Improved Gaussian process models for interpretable predictions.
problem Complex responses require high-dimensional interaction terms in additive Gaussian processes.
method Orthogonal additive kernel (OAK) with orthogonality constraint on additive functions.
result OAK models achieve similar or better predictive performance with fewer terms, retaining interpretability.
We propose an efficient algorithm for solving orthogonal canonical correlation analysis (OCCA) in the form of trace-fractional structure and orthogonal linear projections. Even though orthogonality has been widely used and proved to be a useful criterion for pattern recognition and feature extraction, existing methods …
Study finds equivalence between MMV and MV preferences with conic constraints.
problem Monotone mean-variance portfolio selection under conic constraints.
method Closed-form solutions for optimal strategies under MMV and MV preferences.
result Optimal strategies coincide with and without the conic constraint.
Unified analysis of multilabel Fisher discriminants with improved dimensionality and robustness.
problem Improving discriminant analysis for multilabel classification with enhanced dimensionality and robustness.
method Unified theoretical analysis of multilabel Fisher discriminants with algebraic and statistical guarantees.
result Unified characterization of multilabel Fisher objectives and their equivalence under orthogonality constraints.
A new model detects anomalies in time series data efficiently.
problem Detect anomalies in high-dimensional time series data.
method r-ssGPFA, an unsupervised online anomaly detection model using state space Gaussian processes.
result The model detects anomalies efficiently and is computationally cheaper.
Orthogonal deep models defend against black-box attacks by ensuring internal representations are nearly orthogonal.
problem Vulnerability of deep learning models to black-box adversarial attacks.
method Introduce a gradient regularization scheme to encourage deep models' internal representations to be orthogonal to another model's.
result Orthogonal deep models significantly boost robustness against transferable black-box adversarial attacks.
In this paper, we introduce McTorch, a manifold optimization library for deep learning that extends PyTorch. It aims to lower the barrier for users wishing to use manifold constraints in deep learning applications, i.e., when the parameters are constrained to lie on a manifold. Such constraints include the popular orth…
Prediction accuracy and model explainability are the two most important objectives when developing machine learning algorithms to solve real-world problems. The neural networks are known to possess good prediction performance, but lack of sufficient model interpretability. In this paper, we propose to enhance the expla…
Independent Component Analysis (ICA) is a technique for unsupervised exploration of multi-channel data widely used in observational sciences. In its classical form, ICA relies on modeling the data as a linear mixture of non-Gaussian independent sources. The problem can be seen as a likelihood maximization problem. We i…
The paper decomposes spacelike hypersurface properties for general relativistic vacuum equations.
problem Analyzing properties of spacelike hypersurfaces in general relativity.
method Used L2-orthogonal decomposition and Ahlfors Laplacian. result Decomposed the second fundamental form of spacelike hypersurfaces.
We address the problem of algorithmic fairness: ensuring that sensitive variables do not unfairly influence the outcome of a classifier. We present an approach based on empirical risk minimization, which incorporates a fairness constraint into the learning problem. It encourages the conditional risk of the learned clas…
Proposes methods to add constraints to neural networks to improve stability and generalization.
problem Improving stability and generalization of neural networks.
method Constraint-based regularization using stochastic gradient Langevin dynamics.
result Constraints help stabilize and improve the robustness of deep neural networks.
A new method ODR-BINDy improves model discovery from noisy data.
problem Discovering models from noisy datasets with error-in-variable problem.
method ODR-BINDy uses orthogonal distance regression with Bayesian model selection.
result ODR-BINDy consistently outperforms existing methods in recovering correct models.