Regression learns Mori-Zwanzig operators for dynamical systems.
problem Learning Mori-Zwanzig operators for complex dynamical systems.
method Statistical regression to extract Markov and memory operators.
result Regression models improve learning of memory-dependent corrections.
Study shows double descent curve in high-dimensional linear regression with random projections.
problem Understanding the generalization performance in high-dimensional settings with random projections.
method Fixed prediction problem, ridge regression estimator, minimum norm least-squares fit, random matrix theory, asymptotic equivalents.
result Exhibit a double descent curve for high-dimensional linear regression with random projections.
A new framework for dimension reduction using ensemble of random projections.
problem High-dimensional regression problems with limited data.
method Aggregating an ensemble of carefully chosen random projections, retaining based on empirical performance, and selecting singular vectors.
result The proposed method stabilizes error as the number of projection groups increases.
A new inference method using regression and batched discrepancies.
problem Simulating parameters from simulator outputs.
method Regression-based projection and batched discrepancy weighting.
result Method produces a self-normalized pseudo-posterior.
Aggregates predictions from multiple regression models using random projections and kernel methods.
problem Combining predictions from multiple regression models to improve accuracy.
method Random projection of high-dimensional feature space, followed by kernel-based consensual aggregation.
result The aggregation scheme performs similarly to using the original high-dimensional features, with high probability.
We show how to efficiently project a vector onto the top principal components of a matrix, without explicitly computing these components. Specifically, we introduce an iterative algorithm that provably computes the projection using few calls to any black-box routine for ridge regression. By avoiding explicit principal …
A neural network solves logistic regression with ℓ1 regularization efficiently.
problem Efficiently solving logistic regression with ℓ1 regularization due to non-differentiability of ℓ1 norm. method A simple projection neural network that avoids auxiliary variables and smooth approximations.
result The neural network converges to a solution of the problem with any initial value and outperforms existing methods.
Principal Components Regression (PCR) is a traditional tool for dimension reduction in linear regression that has been both criticized and defended. One concern about PCR is that obtaining the leading principal components tends to be computationally demanding for large data sets. While random projections do not possess…
Project fair estimators while maintaining accuracy.
problem Making estimators fair without sacrificing accuracy.
method Optimal transport tools to find closest fair estimator.
result Efficiently constructs fair estimators with quantified cost.
Efficient boosting method for regression with limited feedback.
problem Online boosting for regression tasks with noisy multi-point bandit feedback.
method Efficient regret minimization method with online boosting algorithm and projection-free online convex optimization.
result Improved state-of-the-art guarantees in efficiency.
New method selects variables for GP regression using sparse projection.
problem Identifying environmental factors affecting metal corrosion.
method Sparse projection of input variables, gradient descent optimization, non-convex marginal likelihood.
result Proposed method outperforms benchmarks in variable selection accuracy.
We propose LOCO, an algorithm for large-scale ridge regression which distributes the features across workers on a cluster. Important dependencies between variables are preserved using structured random projections which are cheap to compute and must only be communicated once. We show that LOCO obtains a solution which …
Paper introduces a new project control method using Monte Carlo and statistical learning.
problem Project control under uncertainty.
method Integrates Earned Value Methodology with Monte Carlo simulation and statistical learning.
result Estimates probabilities of project success and duration.
In this paper, we consider the problem of learning high-dimensional tensor regression problems with low-rank structure. One of the core challenges associated with learning high-dimensional models is computation since the underlying optimization problems are often non-convex. While convex relaxations could lead to polyn…
Projection pursuit model improves Gaussian process regression for high-dimensional data.
problem Scalability issues with traditional Gaussian process models in high dimensions.
method Additive Gaussian process regression with dimension expansion and gradient descent.
result The proposed method approximates more complex functions and outperforms traditional models.
We consider a least squares regression problem where the data has been generated from a linear model, and we are interested to learn the unknown regression parameters. We consider "sketch-and-solve" methods that randomly project the data first, and do regression after. Previous works have analyzed the statistical and c…
ParK efficiently solves kernel ridge regression for large datasets.
problem Large-scale kernel ridge regression efficiency and accuracy.
method Partitioning feature space with random projections and iterative optimization.
result Provably maintains statistical accuracy with reduced space and time complexity.
Paper introduces P-FGD for online quantile regression models.
problem Training nonparametric additive quantile regression models in online settings.
method Projected functional gradient descent algorithm (P-FGD) for pinball loss.
result P-FGD achieves minimax optimal consistency rate O(t−2s+12s). A new method for sparse regression models using graph structure.
problem Sparse regression models for high-dimensional data.
method Decomposes coefficient vector into latent variables, performs regularization on latent variables, uses proximal projection.
result Stable performance compared to other models, especially for high-dimensional data.
This research uses DPPs to improve semi-parametric regression models.
problem Improving comprehensibility in semi-parametric regression models without sacrificing accuracy.
method Introduced a novel representation of finite DPPs and used it to derive a key identity illustrating implicit regularization.
result Demonstrated the implicit regularization effect of determinantal sampling for semi-parametric regression.
VPNet uses variable projection for efficient neural network training.
problem Efficient and interpretable neural network training for signal processing.
method Variable projection (VP) applied to neural networks.
result VPNet achieves fast learning and good accuracy with low computational cost.
This paper reviews SDR methods for multivariate response regression.
problem Handling sufficient dimension reduction for multivariate response regression.
method Characterizes SDR estimators as inverse or forward regression methods.
result Pooled marginal, projective resampling, distance-based, ordinary least squares, partial least squares, and semiparametric SDR estimators are discussed.
New statistical methods for analyzing distributions using Wasserstein metric.
problem Statistical analysis of probability distributions on the real line.
method Projected methods exploiting Wasserstein metric and Riemannian structure.
result Projected PCA and regression methods are faster and more flexible.
This paper proposes robust matrix variate regression models with rank constraints and vector regularization.
problem High dimensional and noisy matrix-valued predictors in regression models.
method Rank constraint, vector regularization, alternating projected gradient descent algorithm.
result The proposed method achieves the minimax rate of estimation errors.
This paper deals with sparse feature selection and grouping for classification and regression. The classification or regression problems under consideration consists in minimizing a convex empirical risk function subject to an ℓ1 constraint, a pairwise ℓ∞ constraint, or a pairwise ℓ1 constraint. …
Projected random forests improve circular data prediction with adaptive arc length and finite-sample coverage.
problem Regression with circular responses.
method Adapting linear-response models to circular data using projection and random forest out-of-bag mechanism.
result Projected random forest out-of-bag conformal prediction sets are more efficient and shorter than alternative methods.
Gaussian processes (GPs) provide flexible distributions over functions, with inductive biases controlled by a kernel. However, in many applications Gaussian processes can struggle with even moderate input dimensionality. Learning a low dimensional projection can help alleviate this curse of dimensionality, but introduc…
The paper shows how sketching data can simplify regression inference even when errors are heteroskedastic.
problem Performing robust inference with heteroskedastic errors using sketched data.
method Using random projections to sketch data, the paper shows that sketched estimates behave as if errors are homoskedastic.
result Estimation by random sampling does not have the same property, and sketched estimates are asymptotically normal with homoskedastic variance.
Efficient algorithms speed up adversarial training for linear models.
problem Adversarial training for linear models is computationally expensive.
method Tailored optimization algorithms for regression and classification.
result Significantly faster convergence rates for large-scale problems.
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
Pareto Front (PF) modeling is essential in decision making problems across all domains such as economics, medicine or engineering. In Operation Research literature, this task has been addressed based on multi-objective optimization algorithms. However, without learning models for PF, these methods cannot examine whethe…
The package High-dimensional Metrics (\Rpackage{hdm}) is an evolving collection of statistical methods for estimation and quantification of uncertainty in high-dimensional approximately sparse models. It focuses on providing confidence intervals and significance testing for (possibly many) low-dimensional subcomponents…
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
New methods combine low and high-fidelity data for accurate surrogate modeling.
problem Challenges in surrogate modeling for high-dimensional outputs with limited training data.
method Projection-based multifidelity linear regression methods integrating low-fidelity and high-fidelity data.
result Multifidelity methods achieve up to 12% improvement in median accuracy compared to single-fidelity methods.
This paper improves computational efficiency in kernel ridge regression under covariate shift.
problem Covariate shift in nonparametric regression.
method Random projections in RKHS to reduce computational demands.
result Significant computational savings can be achieved without compromising learning performance under covariate shift.
Random projections (RP) are a popular tool for reducing dimensionality while preserving local geometry. In many applications the data set to be projected is given to us in advance, yet the current RP techniques do not make use of information about the data. In this paper, we provide a computationally light way to extra…
PCA-based dimensionality reduction improves robustness in overparameterized linear models.
problem Improving robustness in overparameterized linear models.
method PCA-based dimensionality reduction (PCA-OLS)
result PCA-OLS can achieve better generalization than ordinary least squares (OLS) in the overparameterized regime.
Nonlinear kernel regression models are often used in statistics and machine learning because they are more accurate than linear models. Variable selection for kernel regression models is a challenge partly because, unlike the linear regression setting, there is no clear concept of an effect size for regression coeffici…
Improved statistical computation through efficient matrix sampling.
problem Reducing computational cost in large-scale statistical methods.
method Accumulative sub-sampling method to improve statistical efficiency.
result Effective matrix size control improves computational efficiency.
Field canals improvement projects (FCIPs) are one of the ambitious projects constructed to save fresh water. To finance this project, Conceptual cost models are important to accurately predict preliminary costs at the early stages of the project. The first step is to develop a conceptual cost model to identify key cost…
In many applications of supervised learning, multiple classification or regression outputs have to be predicted jointly. We consider several extensions of gradient boosting to address such problems. We first propose a straightforward adaptation of gradient boosting exploiting multiple output regression trees as base le…
This work connects Cramér distance to QR-DQN for DRL.
problem Improving performance in DRL by capturing full distribution of returns.
method Proves Cramér distance's equivalence to 1-Wasserstein distance and proposes a low-complexity algorithm to compute Cramér distance.
result Cramér distance and quantile regression losses yield collinear gradients under non-crossing constraints.
Ability for accurate hospital case cost modelling and prediction is critical for efficient health care financial management and budgetary planning. A variety of regression machine learning algorithms are known to be effective for health care cost predictions. The purpose of this experiment was to build an Azure Machine…
In this paper, we propose a random projection approach to estimate variance in kernel ridge regression. Our approach leads to a consistent estimator of the true variance, while being computationally more efficient. Our variance estimator is optimal for a large family of kernels, including cubic splines and Gaussian ker…
We propose and study kernel conjugate gradient methods (KCGM) with random projections for least-squares regression over a separable Hilbert space. Considering two types of random projections generated by randomized sketches and Nyström subsampling, we prove optimal statistical results with respect to variants of norms …
Optimal convex loss function improves regression coefficient estimation.
problem Asymptotic variance improvement in linear regression estimation.
method Score matching extension for log-concave projection.
result Semiparametric estimator attains minimal asymptotic covariance.
Develops a new fairness learning approach for multi-task regression models.
problem Fairness in multi-task regression models with biased datasets.
method Uses rank-based non-parametric independence test (Mann Whitney U statistic) and reformulates as non-convex optimization problem.
result Outperforms state-of-the-art methods on fairness metrics.
FlowSDR learns a low-dimensional projection preserving the response's conditional distribution.
problem Learning a low-dimensional projection that captures the response's conditional distribution.
method FlowSDR uses conditional log-likelihood maximization with monotone rational-quadratic spline flows to learn the projection and conditional density.
result FlowSDR outperforms existing SDR methods in various simulation settings and a face-age prediction task.