Study on online nonparametric regression using Sobolev kernel methods.
problem Adversarial nonparametric regression in high dimensions.
method Online kernelized ridge regression with Sobolev kernel analysis.
result Upper bounds on regret for Sobolev space classes, revealing optimality in certain cases.
Improves regression efficiency by separating material and immaterial parts of responses.
problem Improving estimation efficiency in nonlinear multivariate regressions.
method Kernel envelope (KENV) estimator for nonparametric response envelopes in reproducing kernel Hilbert space.
result KENV achieves lower in-sample prediction risk than kernel ridge regression in non-trivial immaterial components.
Paper proposes a method for early stopping in regression using reproducing kernels.
problem Early stopping for iterative learning algorithms in nonparametric regression.
method Data-driven rule based on minimum discrepancy principle, validated by fixed-point analysis of localized Rademacher complexities.
result The proposed rule is minimax-optimal and performs comparably to cross-validation.
Inference in popular nonparametric Bayesian models typically relies on sampling or other approximations. This paper presents a general methodology for constructing novel tractable nonparametric Bayesian methods by applying the kernel trick to inference in a parametric Bayesian model. For example, Gaussian process regre…
Paper proves convergence rates for Gaussian kernel ridge regression.
problem Understanding convergence rates for Gaussian kernel ridge regression.
method Establishes polynomial convergence rates for KRR with fixed hyperparameters.
result First polynomial convergence rates for Gaussian kernel ridge regression.
Kernel ridge regression imputation with consistent variance estimation for handling missing data.
problem Handling missing data in statistical analysis.
method Kernel ridge regression imputation combined with entropy method for variance estimation.
result Root-n consistency of the imputation estimator in a Sobolev space setting.
The paper introduces a method for learning nonparametric Volterra kernels using Gaussian processes.
problem Learning nonparametric nonlinear operators from data.
method NVKM model using Volterra series and Gaussian processes for unobserved and observed input functions.
result The NVKM model can perform both single and multiple output regression and system identification.
Kernel method improves instrumental variable regression rates.
problem Nonparametric instrumental variable regression with weak instruments.
method Kernel-based two-stage least-squares method, strong L2 convergence analysis. result Minimax optimal rates for instrumental regression under standard assumptions.
Modeling dynamical systems with ordinary differential equations implies a mechanistic view of the process underlying the dynamics. However in many cases, this knowledge is not available. To overcome this issue, we introduce a general framework for nonparametric ODE models using penalized regression in Reproducing Kerne…
Transformers can efficiently approximate nonparametric regression with minimal parameters and sequences.
problem Efficiently approximating nonparametric regression functions with transformers.
method Kernel-weighted polynomial basis and gradient descent.
result Achieves minimax optimal rate of convergence with fewer parameters and sequences.
Kernel ridge regression inference for nonstandard data.
problem Inferential theory for kernel ridge regression with nonstandard data.
method Constructs valid and sharp confidence sets using anti-symmetric multipliers.
result Develops a test for match effects in school matching mechanisms.
Gradient descent trains neural networks to match kernel regression's sharp generalization rate.
problem Training over-parameterized neural networks for nonparametric regression.
method Gradient descent with early stopping on over-parameterized two-layer neural networks.
result Trained neural networks achieve sharp generalization rate of O(εn2). Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art. We define a space of kernel structures which are built compositionally by adding and multiplying a small number of base kernels. We present a method for searching over this space of structures which mirro…
Meta-learning framework uses task similarity through nonparametric kernel regression.
problem Limited tasks and outliers/dissimilar tasks hinder meta-learning performance.
method Nonparametric kernel regression to quantify and use task similarity.
result Meta-learning algorithm outperforms existing methods in task-limited settings.
New estimator for estimating derivatives in nonparametric regression.
problem Estimating derivatives of regression functions.
method Plug-in kernel ridge regression (KRR) estimator.
result Plug-in property for derivatives estimation, optimal rate of convergence.
The paper analyzes covariate shift in nonparametric regression with Markovian data.
problem Covariate shift in regression problems with Markovian data.
method Extension of nonparametric convergence rates to Markovian dependence structures, using Hölder smoothness assumptions and similarity measures.
result Precise convergence rates for Nadaraya-Watson kernel estimators under specific Markovian conditions.
New method tackles concept shifts in nonparametric regression using robust and adaptive transfer learning.
problem Concept shifts and sample scarcity in target domains hinder nonparametric regression.
method Robust and adaptive transfer learning procedure leveraging fixed bandwidth Gaussian kernels.
result Spectral algorithms with fixed bandwidth Gaussian kernels attain minimax convergence rates for nonparametric regression.
Optimally tackles covariate shift in RKHS-based nonparametric regression.
problem Covariate shift in nonparametric regression over RKHS.
method Two families of covariate shift problems defined using likelihood ratios. Minimax rate-optimal estimators for KRR and reweighted KRR.
result KRR is minimax rate-optimal and strictly sub-optimal compared to naive estimator under covariate shift.
New kernel methods estimate complex causal relationships.
problem Estimating nonparametric causal functions like dose-response curves.
method Kernel ridge regression with decomposition property.
result Uniform consistency with finite sample rates proved.
This paper improves kernel-based regression using transfer learning.
problem Improving generalization performance in kernel-based regression.
method Two-step kernel-based estimator for known transferable sources and novel aggregation algorithm for unknown sources.
result Established statistical properties and validated effectiveness of proposed methods.
Two EM algorithms estimate prior distributions in mixture of linear regressions.
problem Estimating prior distributions in mixture of linear regressions.
method Two EM algorithms: one for continuous priors, one for discrete priors.
result Both algorithms accurately estimate prior distributions and the number of clusters.
Coercivity condition ensures learning of interacting particle systems.
problem Ensuring identifiability of interaction functions in learning systems of interacting particles.
method Equivalence of coercivity condition to strictly positive definiteness of an integral kernel.
result For ergodic systems, the integral kernel is strictly positive definite, satisfying the coercivity condition.
New method for causal inference with observed covariates improves learning rates.
problem Causal inference with observed covariates in nonparametric instrumental variable regression.
method Introduces novel Fourier measure for partial smoothing and adapts kernel lengthscales for anisotropic smoothness.
result Upper and lower learning rates for KIV-O show interpolation between NPIV and NPR rates.
Paper optimizes prediction in semi-functional linear models using kernel methods.
problem Optimizing prediction in semi-functional linear models with functional and nonparametric components.
method Double-penalized least squares method in reproducing kernel Hilbert spaces, with regularization parameter selection via generalized cross validation.
result Achieves minimax optimal rates of convergence for both functional and nonparametric components.
HAR regression improves performance on small datasets.
problem Small datasets with complex functions.
method Data-adaptive kernel ridge regression using tensor-product spline basis.
result Achieves n−1/3 convergence rate for right-continuous functions. In this paper, we propose a variable selection method for general nonparametric kernel-based estimation. The proposed method consists of two-stage estimation: (1) construct a consistent estimator of the target function, (2) approximate the estimator using a few variables by l1-type penalized estimation. We see that the…
Study on discrepancy principle for learning algorithms in nonparametric regression.
problem Determining optimal iteration number in nonparametric regression with unknown optimal iteration.
method Investigates discrepancy principle and modified principles for kernelized spectral filters, using deviation inequalities and change-of-norm arguments.
result Classical discrepancy principle is adaptive for slow rates, while modified principles are adaptive for faster rates.
Needlets have been recognized as state-of-the-art tools to tackle spherical data, due to their excellent localization properties in both spacial and frequency domains. This paper considers developing kernel methods associated with the needlet kernel for nonparametric regression problems whose predictor variables are de…
Bayesian nonparametric LABS model adapts to function smoothness in Besov spaces.
problem Estimating functions with unknown smoothness in Besov spaces.
method Lévy Adaptive B-spline (LABS) regression model with automatic smoothness adaptation.
result LABS posterior contracts around true function in Besov classes at nearly minimax-optimal rates.
Unified view of score estimators for flexible densities.
problem Estimating the score from unknown distributions.
method Regularized nonparametric regression framework.
result Unified convergence analysis and new estimators with desirable properties.
Instrumental variable (IV) regression is a strategy for learning causal relationships in observational data. If measurements of input X and output Y are confounded, the causal relationship can nonetheless be identified if an instrumental variable Z is available that influences X directly, but is conditionally independe…
The paper examines when importance weighting is needed for nonparametric and misspecified models.
problem When is importance weighting correction needed for covariate shift adaptation?
method Analysis of IW-corrected kernel ridge regression in various settings.
result The importance weighting correction is needed for nonparametric and misspecified models to obtain the best approximation of the true unknown function.
The paper develops methods to handle missing data using regularized M-estimation in reproducing kernel Hilbert space.
problem Handling missing data in statistical analysis.
method Kernel ridge regression for imputation and maximum entropy method for propensity score estimation.
result The proposed methods achieve statistical consistency and asymptotic equivalence.
This paper aims to solve a basic problem in distributed statistical inference: how many machines can we use in parallel computing? In kernel ridge regression, we address this question in two important settings: nonparametric estimation and hypothesis testing. Specifically, we find a range for the number of machines und…
Bayesian method with Gaussian process priors achieves optimal convergence rates for regression function and its derivatives.
problem Estimating the regression function and its derivatives in nonparametric regression.
method Bayesian approach with Gaussian process priors, focusing on convergence rates and plug-in property.
result Equivalence of convergence rates of posterior distributions and Bayes estimators for regression function and its derivatives.
We propose Bayesian extensions of two nonparametric regression methods which are kernel and mutual k-nearest neighbor regression methods. Derived based on Gaussian process models for regression, the extensions provide distributions for target value estimates and the framework to select the hyperparameters. It is show…
Signal processing tasks as fundamental as sampling, reconstruction, minimum mean-square error interpolation and prediction can be viewed under the prism of reproducing kernel Hilbert spaces. Endowing this vantage point with contemporary advances in sparsity-aware modeling and processing, promotes the nonparametric basi…
Improved nonparametric regression with debiasing for root-n consistency.
problem Challenges in achieving root-n consistency and normal distribution for nonparametric estimators.
method Debiasing technique by adding a correction term to nonparametric estimators.
result Achieves root-n consistency and asymptotic normality.
New algorithm for nonparametric IV regression using stochastic gradients.
problem Identifying causal effects in the presence of unobservable confounders.
method Functional stochastic gradient descent for NPIV regression.
result Superior stability and competitive performance compared to existing methods.
DFIV uses deep features for IV regression, achieving optimal rates.
problem Optimal IV regression with deep features for complex target functions.
method Two-stage approach: deep feature learning followed by IV regression.
result DFIV achieves minimax optimal learning rate under certain conditions.
Develops an MS-inspired algorithm for regression mode finding and space partitioning.
problem Finding local modes of regression functions and partitioning input space.
method Mean-shift-inspired algorithm for iterative gradient ascent.
result Proves convergence and rates of convergence for estimated local modes.
Most machine learning algorithms, such as classification or regression, treat the individual data point as the object of interest. Here we consider extending machine learning algorithms to operate on groups of data points. We suggest treating a group of data points as an i.i.d. sample set from an underlying feature dis…
Modal regression estimates the local modes of the distribution of Y given X=x, instead of the mean, as in the usual regression sense, and can hence reveal important structure missed by usual regression methods. We study a simple nonparametric method for modal regression, based on a kernel density estimate (KDE) of …
Proposes estimators for complex dose-response curves using kernel methods.
problem Estimating complex dose-response curves with continuous treatments, mediators, and covariates.
method Kernel ridge regression with sequential kernel embedding technique.
result Simple estimators for mediated and time-varying dose response curves with nonasymptotic uniform rates.
GD with early stopping trains shallow neural nets for nonparametric regression robustly.
problem Learning Lipschitz regression functions with noisy labels.
method Overparameterized shallow neural networks trained by GD with early stopping.
result Optimal rates of convergence for nonparametric regression.
The paper provides guarantees for feedback control with sensor errors.
problem Certifying performance and safety in feedback control systems with sensor errors.
method Solving a supervised learning problem to characterize sensor errors and providing uniform error bounds.
result Finite-time convergence rate on sub-optimality of using a regressor in closed-loop for waypoint tracking.
New method calibrates probabilistic regression models without restrictive assumptions.
problem Ensuring predictive distributions accurately reflect true uncertainty.
method Nonparametric re-calibration algorithm based on conditional kernel mean embeddings.
result Consistently outperforms prior re-calibration approaches across various benchmarks.
A common challenge in nonparametric inference is its high computational complexity when data volume is large. In this paper, we develop computationally efficient nonparametric testing by employing a random projection strategy. In the specific kernel ridge regression setup, a simple distance-based test statistic is prop…