A new optimizer, MVO, improves nonlinear regression performance.
problem Finding optimal coefficients in nonlinear regression models.
method Multi-Verse Optimizer (MVO) compared to Particle Swarm Optimizer (PSO).
result MVO statistically outperforms PSO in 10 nonlinear regression problems.
We analyzed optimism in linear and kernel regression models.
problem Understanding predictive complexity in regression models.
method Derived closed-form asymptotic optimism for linear and kernel regression models.
result Scaled optimism is a useful measure for model complexity.
The paper explores fair regression and classification under demographic parity constraints.
problem Ensuring fairness in regression and classification models under demographic parity constraints.
method Characterizes the optimal fair regression function using a barycenter problem with optimal transport costs and studies the connection between fair classification and regression.
result The optimal fair regression function is derived from the solution to a barycenter problem with optimal transport costs, and the optimal fair cost-sensitive classifiers can be derived by applying thresholds to this function.
Algorithm optimizes quantized isotonic regression with log-linear time updates.
problem Optimizing quantized isotonic regression estimations.
method Modified PAVA algorithm for sequential optimization.
result Log-linear time updates for optimal quantized mapping.
In this paper we combine two important extensions of ordinary least squares regression: regularization and optimal scaling. Optimal scaling (sometimes also called optimal scoring) has originally been developed for categorical data, and the process finds quantifications for the categories that are optimal for the regres…
Meta-theorems validate fair regression algorithms under demographic parity constraints.
problem Regression under demographic parity constraints.
method Meta-theorems and post-processing methods.
result Fair minimax optimal regression can be achieved through post-processing.
Optimizes binary regression models with gradient ascent-descent methods.
problem Regression problems with binary weights in quantized learning and digital communication.
method Maximin optimization using gradient ascent-descent methods.
result The approach is optimal in linear regression with low noise and robust regression with few outliers.
Optimal nonparametric regression estimator adapts to unknown smoothness.
problem Nonparametric regression with unknown smoothness.
method Constructs an interpolating estimator that adapts to unknown smoothness.
result Minimax optimal rates achieved on Hölder classes.
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
Paper proposes a new method for regression using optimal transport cost optimization.
problem Regression problem, especially with complex noise forms.
method Directly optimizes the optimal transport cost between true and estimated distributions.
result Achieves state-of-the-art results on various datasets.
Brenier isotonic regression extends multi-output isotonic regression using optimal transport.
problem Enforcing cyclic monotonicity in multi-output regression.
method Leverage Kantorovich's optimal transport to find cyclically monotone couplings.
result Brenier isotonic regression outperforms baselines in probability calibration.
Regression-via-Classification (RvC) is the process of converting a regression problem to a classification one. Current approaches for RvC use ad-hoc discretization strategies and are suboptimal. We propose a neural regression tree model for RvC. In this model, we employ a joint optimization framework where we learn opt…
Transformers can learn optimal regression mixtures efficiently.
problem Limited adoption of tailored regression methods due to their model-specific nature.
method Constructed a generative process for a mixture of linear regressions and used transformers to learn optimal predictors.
result Transformers achieve low mean-squared error and make predictions close to the optimal procedure.
New algorithm reduces contextual bandits to efficient regression.
problem Developing efficient algorithms for contextual bandits with general function classes.
method Reduction from contextual bandits to online regression with oracle.
result First universal and optimal reduction with no overhead.
We find a convex model for traditional nonlinear regression under L2 loss.
problem Nonlinear regression under L2 loss with non-convex optimization.
method Showed a convex nonlinear regression model for least squares problem.
result Existence of a convex model simplifies training complex systems.
Optimal rates for vector-valued regression on various norms.
problem Optimal rates for vector-valued ridge regression on continuous norms.
method Combining standard capacity assumptions with tensor product constructions of vector-valued interpolation spaces.
result Optimal rates for vector-valued ridge regression, independent of output space dimension.
We find the optimal error for a constrained regression model under a linear model.
problem Minimizing error while adhering to demographic parity constraints.
method Proposed a minimax optimal error analysis for a demographic parity-constrained regression problem within a linear model.
result The minimax optimal error is characterized by $Θ( rac{dM}{n})$.
Paper studies distributed kernel regression with imperfect kernels, achieving optimal rates.
problem Optimal rates of distributed regression with imperfect kernels.
method Divide and conquer approach, response weighted base algorithms, leave one out analysis, bias correction.
result Achieves capacity independent optimal rates for distributed kernel regression with imperfect kernels.
In this paper, we study regression problems over a separable Hilbert space with the square loss, covering non-parametric regression over a reproducing kernel Hilbert space. We investigate a class of spectral/regularized algorithms, including ridge regression, principal component regression, and gradient methods. We pro…
Optimizes hyperparameter tuning for models using approximate leave-one-out cross-validation.
problem Finding optimal hyperparameters for regularized models using approximate leave-one-out cross-validation.
method Derive efficient formulas for gradient and hessian of approximate leave-one-out cross-validation, apply second-order optimization.
result Demonstrates the effectiveness of the approach on real-world data sets.
Adversarial online nonparametric regression achieves optimal rates with locally adaptive learning.
problem Adversarial online nonparametric regression with general convex losses.
method Parameter-free learning algorithm leveraging chaining trees to compete against H{ö}lder functions, dynamically tracking and adapting to local smoothness variations.
result First computationally efficient algorithm with locally adaptive optimal rates for online regression in an adversarial setting.
SBT model uses randomized sharding and sub-models to improve Bayesian Additive Regression Trees.
problem Improving efficiency and accuracy of Bayesian Additive Regression Trees.
method Randomized sharding, sub-models, intersection tree structure, optimal design.
result Theoretical optimal weights and worst-case complexity of SBT model.
The paper improves methods for generating prediction intervals in regression.
problem Uncertainty quantification in regression models.
method Formalizes prediction interval generation as an optimization problem, studying generalization and calibration.
result Empirical demonstration of improved testing performances compared to existing methods.
Adaptive algorithm for multi-objective optimization with binary constraints.
problem Optimization of black-box problems with binary constraints.
method Bayesian optimization using regression and classification models.
result Significantly faster expected hypervolume calculation.
New approach quantifies overfitting in high-dimensional regression.
problem Quantifying and avoiding overfitting in large neural networks.
method Information bottleneck theory to minimize residual information while maximizing relevant bits.
result Characterized the relative information efficiency of randomized regression compared to optimal algorithms.
Develops a numerical algorithm for stochastic impulse control using regression surrogates.
problem Optimal impulse control in stochastic processes.
method Generates statistical surrogates for continuation and intervention functions, recursively trained over simulated state trajectories.
result Demonstrates flexibility and extensibility of the numerical scheme through case studies.
OKRidge solves sparse ridge regression problems for nonlinear systems.
problem Identifying sparse governing equations for nonlinear dynamical systems.
method OKRidge algorithm using saddle point formulation and ADMM-based approach with efficient proximal operators.
result OKRidge achieves provable optimality with significantly faster run times than Gurobi.
The least squares Monte Carlo algorithm has become popular for solving portfolio optimization problems. A simple approach is to approximate the value functions on a discrete grid of portfolio weights, then use control regression to generalize the discrete estimates. However, the classical global control regression can …
FineMorphs models smooth transformations for multivariate regression.
problem Efficiently modeling complex transformations for multivariate regression.
method Optimal control of affine and diffeomorphic transformations using smooth vector fields.
result FineMorphs can reduce dimensionality and adapt to large datasets.
The paper improves Gaussian process regression by optimizing hyperparameters.
problem Hyperparameter tuning for Gaussian process regression models.
method Adaptive sparse variational approximations using variational Bayes.
result Minimax optimal rates of convergence for variational posterior.
A scalable algorithm improves AUC optimization for semi-supervised ordinal regression.
problem Optimizing AUC for semi-supervised ordinal regression with limited labeled data.
method Proposes QS3ORAO using quadruply stochastic gradients for scalable kernelized learning. result Converges to optimal solution at O(1/t) rate, demonstrating efficiency and effectiveness. Optimizes tensor rank selection for neural network compression.
problem Finding optimal tensor rank for regression models.
method Analyzes population expressions for training-testing discrepancy under Gaussian design.
result Optimal rank minimizes prediction error and aligns with cross-validation.
Transformers can efficiently approximate nonparametric regression with minimal parameters and sequences.
problem Efficiently approximating nonparametric regression functions with transformers.
method Kernel-weighted polynomial basis and gradient descent.
result Achieves minimax optimal rate of convergence with fewer parameters and sequences.
A new method for nonparametric regression using mesh-based solutions.
problem Estimating regression functions non-parametrically with computational tractability.
method Mesh-based approximate solution (MBS) for penalized regression problems.
result MBS transforms NPR to a discrete convex minimization problem, making it computationally feasible.
Optimal data split ratio is sqrt(p):1 for linear regression.
problem Lack of clear guidance on optimal training/testing data split ratio.
method Showed that optimal ratio is sqrt(p):1 for linear regression.
result Optimal ratio for training/testing split is sqrt(p):1.
Improved symbolic regression finds optimal formulas robust to noise.
problem Finding accurate formulas for noisy data.
method Exploits graph modularity, uses normalizing flows, and statistical hypothesis testing.
result Discoveres many formulas previously unattainable.
Paper investigates optimal interpolation methods in linear regression.
problem Understanding when interpolating methods generalize well in linear regression.
method Investigates optimal response-linear interpolators using functions linear in the response variable.
result Provides a closed-form expression for the optimal interpolator and shows it can be derived as the limit of gradient descent.
We establish optimal rates for online regression for arbitrary classes of regression functions in terms of the sequential entropy introduced in (Rakhlin, Sridharan, Tewari, 2010). The optimal rates are shown to exhibit a phase transition analogous to the i.i.d./statistical learning case, studied in (Rakhlin, Sridharan,…
Proposes a new algorithm for Sparse Bayesian Learning connected to Stepwise Regression.
problem Sparse Bayesian Learning for probabilistic models.
method Coordinate ascent algorithm (RMP) for SBL, showing connection to Stepwise Regression.
result RMP's noise variance parameter limit connects to Stepwise Regression, with derived guarantees.
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
The generalized linear model (GLM) plays a key role in regression analyses. In high-dimensional data, the sparse GLM has been used but it is not robust against outliers. Recently, the robust methods have been proposed for the specific example of the sparse GLM. Among them, we focus on the robust and sparse linear regre…
DFIV uses deep features for IV regression, achieving optimal rates.
problem Optimal IV regression with deep features for complex target functions.
method Two-stage approach: deep feature learning followed by IV regression.
result DFIV achieves minimax optimal learning rate under certain conditions.
NMDR estimates complex mixtures of distributions efficiently.
problem Estimating complex finite mixtures of distributions in high-dimensional settings.
method Flexible additive predictors, neural networks, and deep learning optimizers.
result Competitive performance in complex scenarios compared to existing approaches.
Neural networks perform differently when regression is treated as classification.
problem Understanding why neural networks perform better when regression is treated as classification.
method Analyzing two-layer ReLU networks and their feature spaces, focusing on the cross entropy loss vs. square loss.
result The support of the measure induced by the square loss differs from that of the cross entropy loss, indicating optimization difficulties.
This paper studies the addition of linear constraints to the Support Vector Regression (SVR) when the kernel is linear. Adding those constraints into the problem allows to add prior knowledge on the estimator obtained, such as finding probability vector or monotone data. We propose a generalization of the Sequential Mi…
GD outperforms ridge regression and SGD in linear regression problems.
problem Comparing the risks of GD, ridge regression, and SGD in linear regression problems.
method Instance-wise finite-sample risk analysis of GD, ridge regression, and SGD.
result GD outperforms ridge regression and is incomparable with SGD in some cases.
Safe screening rules reduce ℓ0-regression computation by fixing 76% of variables.
problem Efficiently solving ℓ0-regression problems with large datasets. method Convex relaxation and safe screening rules to eliminate variables.
result 76% of variables can be fixed to their optimal values, reducing computational burden.
New adaptive models improve prediction accuracy with missing data.
problem Improving prediction accuracy with missing data entries.
method Adaptive optimization approach, learning imputation and regression simultaneously.
result 2-10% improvement in out-of-sample accuracy in strongly non-random missing data settings.