In the multiple linear regression setting, we propose a general framework, termed weighted orthogonal components regression (WOCR), which encompasses many known methods as special cases, including ridge regression and principal components regression. WOCR makes use of the monotonicity inherent in orthogonal components …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “weighted orthogonal regression”
A new classifier uses weighted orthogonal regression for robust classification with limited data.
problem Challenges in classification with insufficient training data.
method Exploits intrinsic structure of data through Eigen components with specific weights determined by eigenvalues.
result Robust learning in classification problems with limited data.
Many scientific questions require estimating the effects of continuous treatments. Outcome modeling and weighted regression based on the generalized propensity score are the most commonly used methods to evaluate continuous effects. However, these techniques may be sensitive to model misspecification, extreme weights o…
Effective features can improve the performance of a model, which can thus help us understand the characteristics and underlying structure of complex data. Previous feature selection methods usually cannot keep more local structure information. To address the defects previously mentioned, we propose a novel supervised o…
New method optimizes model selection in high-dimensional regression models.
problem Model selection in high-dimensional misspecified regression models with covariate shift.
method Importance-weighted orthogonal greedy algorithm (IWOGA) and high-dimensional importance-weighted information criterion (HDIWIC).
result IWOGA + HDIWIC achieves optimal convergence rates in terms of prediction error.
Gradient-enhanced GSA uses Poincaré chaos expansions for accurate sensitivity analysis.
problem Accurately estimating Sobol' indices with limited data.
method Integrates sparse, gradient-enhanced regression with Poincaré chaos expansions for derivative-based sensitivity analysis.
result Accurately estimated Sobol' indices using limited data.
Unified framework for debiased machine learning using Riesz representer and Bregman divergence.
problem Estimating causal and structural parameters in machine learning.
method Generalized Riesz regression for fitting Riesz representer via Bregman divergence minimization.
result Automatic covariate balancing and Neyman orthogonality properties for debiased estimation.
One reflection suffices for orthogonal weights, reducing GPU usage.
problem Efficiently computing orthogonal weight matrices without high GPU utilization.
method Use an auxiliary neural network to compute one reflection instead of many.
result One reflection is sufficient for orthogonal weights, improving GPU utilization.
PROD method improves high-dimensional regression by handling strong correlations.
problem Violation of Irrepresentable Condition in LASSO for high-dimensional data.
method PROD procedure based on orthogonal decomposition of design matrix.
result PROD enhances performance of high-dimensional penalized regression.
Introduces new curvature concept for Kähler manifolds.
problem Optimizing curvature constraints for projective Kähler manifolds.
method Introduces weighted orthogonal Ricci curvature and proves vanishing theorems.
result Proves optimal curvature constraints for projective Kähler manifolds.
Pion optimizes LLMs by preserving weight matrix singular values.
problem Training large language models (LLMs) with standard optimizers leads to unstable weight matrices.
method Pion uses orthogonal transformations to update weight matrices, preserving their singular values.
result Pion offers a stable alternative to standard optimizers for LLM pretraining and finetuning.
Orthogonal initialization does not speed up training in ultra-wide neural networks.
problem Exploring the effect of orthogonal initialization on training speed in deep neural networks.
method Study of neural tangent kernel dynamics in FCNs and CNNs with orthogonal initialization.
result The NTK of orthogonally-initialized networks remains constant during training, suggesting no speedup in the NTK regime.
Study shows RFRR's effectiveness with nearly orthogonal data in overparameterized settings.
problem Understanding the effectiveness of random feature regression with nearly orthogonal data.
method Investigates RFRR with nearly orthogonal deterministic unit-length input data vectors in the overparameterized regime.
result Shows high-probability non-asymptotic concentration results for RFRR's training, cross-validation, and generalization errors.
OOMP selects features online for sparse linear regression.
problem Feature selection in high-dimensional sparse linear models.
method Online algorithm that alternates between feature selection and coefficient estimation.
result Theoretical guarantees and computational complexity analysis of OOMP.
The paper argues for using Neyman orthogonal score for balancing in debiased machine learning.
problem Debiased machine learning requires a proper approach to balance covariates.
method The paper advocates for using Riesz regression with basis functions of X for balancing.
result Covariate balancing is only valid when the score-relevant regression error is a function of covariates alone.
Paper develops robust econometric methods for staggered adoption studies.
problem Estimation challenges in event studies with staggered adoption.
method Design-first framework with exact probability limits, diagnostics, and orthogonal score constructions.
result Uniformly valid inference under restricted violations of parallel trends.
A l1-norm penalized orthogonal forward regression (l1-POFR) algorithm is proposed based on the concept of leaveone- out mean square error (LOOMSE). Firstly, a new l1-norm penalized cost function is defined in the constructed orthogonal space, and each orthogonal basis is associated with an individually tunable regulari…
Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.
problem Fluctuations in deep networks with Gaussian weights can impair training, especially in networks with depth comparable to width.
method Analytical and numerical studies of fully-connected networks with orthogonal weight initialization and tanh activations.
result Rectangular networks with orthogonal weights have stable fluctuations independent of network depth, leading to better generalization and training speed.
We propose two nonlinear regression methods, named Adversarial Orthogonal Regression (AdOR) for additive noise models and Adversarial Orthogonal Structural Equation Model (AdOSE) for the general case of structural equation models. Both methods try to make the residual of regression independent from regressors while put…
MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.
problem Training matrix-valued parameters with orthogonalized-update optimizers like Muon.
method MuonEq introduces three lightweight pre-orthogonalization equilibration schemes: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).
result Row/column normalization acts as a zeroth-order surrogate for whitening and improves the geometry seen by orthogonalization.
Semi-parametric framework for nonlinear system identification
problem Nonlinear system identification
method Orthogonal Gaussian process regression
result Interpretable models from incomplete physics
Fibonacci Ensembles use Fibonacci weights to improve ensemble learning, inspired by natural growth patterns.
problem Improving ensemble learning methods to enhance model performance and interpretability.
method Introduces Fibonacci weights and a recursive ensemble dynamic to reduce variance and enrich representational depth.
result Fibonacci weighting can match or improve upon uniform averaging in ensemble learning experiments.
We propose a penalized orthogonal-components regression (POCRE) for large p small n data. Orthogonal components are sequentially constructed to maximize, upon standardization, their correlation to the response residuals. A new penalization framework, implemented via empirical Bayes thresholding, is presented to effecti…
Wasserstein-GANs have been introduced to address the deficiencies of generative adversarial networks (GANs) regarding the problems of vanishing gradients and mode collapse during the training, leading to improved convergence behaviour and improved image quality. However, Wasserstein-GANs require the discriminator to be…
New algorithm identifies best arm in semiparametric bandits with near optimal efficiency.
problem Fixed-confidence Best Arm Identification in semiparametric bandits with unknown baseline shift.
method Phase-elimination algorithm based on orthogonalized regression design.
result Nearly optimal high-probability sample-complexity upper bound established.
A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neura…
Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an -constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…
This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
problem Proving the asymptotic freeness of layerwise Jacobians in multilayer perceptrons (MLPs).
method Replacing each layer's parameter matrix with itself multiplied by a Haar orthogonal matrix, and using the invariance of the MLP.
result Proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to a prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near $…
A new method improves continual learning by replaying pseudo data and using orthogonal weight modification.
problem Catastrophic forgetting in class incremental learning.
method Generative replay with orthogonal weight modification.
result Our method achieves substantial improvement over conventional generative replay and OWM.
We propose a novel application of the Simultaneous Orthogonal Matching Pursuit (S-OMP) procedure for sparsistant variable selection in ultra-high dimensional multi-task regression problems. Screening of variables, as introduced in \cite{fan08sis}, is an efficient and highly scalable way to remove many irrelevant variab…
Deep networks without non-linearities are equivalent to shallow ones.
problem Training deep orthogonal linear networks with no non-linearity.
method Riemannian gradient descent and gradient descent on factorization.
result Training deep overparametrized networks is equivalent to shallow ones.
Self-Consistent Asset Pricing Modelsphysics.soc-ph
We discuss the foundations of factor or regression models in the light of the self-consistency condition that the market portfolio (and more generally the risk factors) is (are) constituted of the assets whose returns it is (they are) supposed to explain. As already reported in several articles, self-consistency implie…
Study connects curvature to graph theory and reveals differences.
problem Exploring differences between Quadratic Orthogonal Bisectional Curvature and Real Bisectional Curvature.
method Real (1,1)--forms and Weitzenböck curvature operator used to represent graph Dirichlet energy.
result Curvature differences illuminated between Quadratic Orthogonal Bisectional Curvature and Real Bisectional Curvature.
Orthogonal initialization speeds up convergence in deep linear networks.
problem The impact of initialization on convergence speed and model performance in deep neural networks.
method Analysis of orthogonal initialization in deep linear networks, proving its superiority over Gaussian initialization.
result Orthogonal initialization speeds up convergence relative to Gaussian initialization in deep networks.
SOFARI improves inference on multi-task learning latent factors.
problem Challenges in precise inference on multi-task learning latent factor matrices.
method High-dimensional manifold-based Neyman near-orthogonality inference on Stiefel manifold structure.
result Easy-to-use bias-corrected estimators for latent factor vectors and singular values with asymptotic normal distributions.
OPT framework improves neural network generalization by learning an orthogonal transformation.
problem Improving neural network generalization.
method Orthogonal over-parameterized training (OPT) framework that minimizes hyperspherical energy.
result OPT framework provably minimizes hyperspherical energy and improves empirical generalization.
In this short note we show the following result: Let () be a compact Sasaki manifold with positive transverse orthogonal bisectional curvature. Then is finite, and the universal cover of is isomorphic to a weighted Sasaki sphere. We also get some results in the case of n…
The paper improves GP regression for sparse sensor data in structural mode shape reconstruction.
problem Reconstructing full-field structural mode shapes from sparse sensor data.
method Physics-Constrained Single-Output Gaussian Process (CONS-SOGP) framework.
result The proposed method provides more accurate and reliable mode shapes.
Distributed-OMP recovers sparse vectors with low communication costs.
problem High-dimensional sparse linear regression with limited computation and communication.
method Distributed orthogonal matching pursuit (OMP) scheme.
result Support of the regression vector can be recovered with linear communication per machine and logarithmic in dimension.
New LT-O-learners improve HLTE estimation with low overlap.
problem Challenges in estimating heterogeneous long-term treatment effects due to limited overlap.
method Introduces LT-O-learners that use custom overlap weights to downweight low-overlap samples.
result LT-O-learners provide robust HLTE estimates with lower variance in low-overlap regimes.
VAR-GPs solve continual learning by updating posteriors sequentially.
problem Catastrophic forgetting in sequential learning tasks.
method Sparse inducing point approximations and auto-regressive variational distribution.
result VAR-GPs prevent catastrophic forgetting and outperform baselines.
Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simp…
Double machine learning provides -consistent estimates of parameters of interest even when high-dimensional or nonparametric nuisance parameters are estimated at an rate. The key is to employ Neyman-orthogonal moment equations which are first-order insensitive to perturbations in the nuisance param…
LOFT separates subspace rotation and transformation for orthogonal fine-tuning.
problem Conflating subspace rotation and transformation in orthogonal fine-tuning.
method LOFT explicitly separates subspace rotation and transformation, using task-aware support selection.
result LOFT recovers principal-subspace orthogonal adaptation and improves efficiency-performance trade-off.
New method enforces orthogonality in convolutional layers for improved robustness.
problem Improving adversarial robustness in deep learning models.
method Applying the Cayley transform to skew-symmetric convolutions in the Fourier domain.
result The proposed method preserves orthogonality and enhances adversarial robustness compared to existing techniques.
We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidea…
Extends deep learning with interpretable additive models.
problem Identifiability issues between neural networks and additive models.
method Orthogonalization cell to separate deep neural network and structured model parts.
result Stable estimation and interpretability of structured model parts.