We find the optimal error for a constrained regression model under a linear model.
problem Minimizing error while adhering to demographic parity constraints.
method Proposed a minimax optimal error analysis for a demographic parity-constrained regression problem within a linear model.
result The minimax optimal error is characterized by $Θ( rac{dM}{n})$.
Faster methods for large-scale constrained linear regression using sketching and optimization.
problem Efficiently solving large-scale constrained linear regression problems.
method Combining (accelerated) mini-batch SGD with two-step preconditioning.
result Achieves faster approximate solutions with lower time complexity.
Two novel approaches improve prediction with Clusterwise Linear Regression.
problem Predicting cluster labels for unseen test points in CLR.
method Two novel approaches: Predictive CLR and Constrained CLR.
result Both approaches significantly improve over known CLR-based regression methods.
New algorithm solves constrained ℓ_p regression problems efficiently.
problem Minimizing ℓ_p regression over unit vectors with constraints.
method Uses core-sets and provable constant factor approximation.
result First provable constant factor approximation algorithm.
Regularization aims to improve prediction performance of a given statistical modeling approach by moving to a second approach which achieves worse training error but is expected to have fewer degrees of freedom, i.e., better agreement between training and prediction error. We show here, however, that this expected beha…
c-lasso is a Python tool for robust and sparse regression with linear constraints.
problem Sparse and robust linear regression with linear constraints.
method Estimates coefficients and scale under linear constraints using perspective M-estimators.
result Provides estimators for various loss functions with linear constraints.
Paper tackles multivariate shape-constrained convex regression problems.
problem Fitting a convex function to data with component-wise monotonicity and uniform Lipschitz continuity.
method Least squares estimator via solving a constrained convex quadratic programming problem. Efficient algorithms designed: sGS-ADMM and pALM.
result Both proposed algorithms outperform state-of-the-art methods in numerical experiments.
Sparsity-constrained optimization has wide applicability in machine learning, statistics, and signal processing problems such as feature selection and compressive Sensing. A vast body of work has studied the sparsity-constrained optimization from theoretical, algorithmic, and application aspects in the context of spars…
The paper analyzes the generalizability of linear autoencoders and multivariate linear regression.
problem Limited theoretical understanding of linear autoencoders' performance.
method Proposes a PAC-Bayes bound for multivariate linear regression and shows LAEs as constrained models.
result The proposed PAC-Bayes bound is tight and correlates with practical metrics.
Optimizes binary regression models with gradient ascent-descent methods.
problem Regression problems with binary weights in quantized learning and digital communication.
method Maximin optimization using gradient ascent-descent methods.
result The approach is optimal in linear regression with low noise and robust regression with few outliers.
We derive computationally tractable methods to select a small subset of experiment settings from a large pool of given design points. The primary focus is on linear regression models, while the technique extends to generalized linear models and Delta's method (estimating functions of linear regression models) as well. …
New algorithms improve sampling from constrained distributions.
problem Generating samples from distributions under constraints.
method Kinetic Langevin dynamics and splitting schemes.
result Improved complexity bounds over existing methods.
Paper tackles distributed linear regression with compositional covariates.
problem Solving distributed statistical methodology and computing for massive compositional data.
method Proposes two distributed optimization techniques based on ADMM and CDMM for solving constrained convex optimization problems.
result Established convergence theories for the proposed algorithms under regularity conditions.
New method solves constrained self-concordant minimization problems efficiently.
problem Constrained self-concordant minimization problems.
method Newton Frank-Wolfe method using linear minimization oracles.
result The method uses nearly the same number of linear minimization calls as the Frank-Wolfe method.
Unified approach to linear regression using covariance fitting for optimal weights.
problem Finding optimal weights for linear regression models when weights are unknown.
method Covariance fitting SPICE-methodology to obtain data-adaptive weights.
result Tuned versions of known regularized estimators are unified under a common approach.
We describe dimensionally constrained symbolic regression which has been developed for mass measurement in certain classes of events in high-energy physics (HEP). With symbolic regression, we can derive equations that are well known in HEP. However, in problems with large number of variables, we find that by constraini…
New methods reduce computational cost for Gaussian Markov Random Fields with sparse constraints.
problem Inference and simulation of GMRFs are computationally prohibitive with many constraints.
method Proposes a basis transformation into blocks of constrained and non-constrained subspaces.
result Significantly outperforms existing alternatives in computational cost.
Sharp risk bounds for early-stopping in Gaussian linear regression are derived.
problem Minimizing in-sample mean squared error in high-dimensional Gaussian linear regression.
method Early-stopped mirror descent (ESMD) with local Gaussian width bounds.
result Sharp risk bounds extend to early-stopped mirror descent for least squares estimator (LSE).
Brenier isotonic regression extends multi-output isotonic regression using optimal transport.
problem Enforcing cyclic monotonicity in multi-output regression.
method Leverage Kantorovich's optimal transport to find cyclically monotone couplings.
result Brenier isotonic regression outperforms baselines in probability calibration.
The study examines robustness auditing for linear regression, improving existing methods and identifying computational challenges.
problem Detecting small subsets of data that can reverse regression coefficients.
method Empirical study of mixed integer quadratically constrained optimization and exact greedy methods, combined with a spectral algorithm.
result Existing methods largely outperform state of the art, but computational bottlenecks remain, especially for higher dimensions.
New method solves constrained optimization problems efficiently.
problem Equality-constrained nonlinear, nonconvex optimization problems.
method Adaptive inexact Newton method with randomized iterative sketching.
result Global almost sure convergence and local linear/superlinear convergence.
Convex-constrained sparse additive models improve regression performance.
problem High-dimensional nonparametric regression with shape constraints.
method Sparse difference of convex additive models (SDCAM) with regularization and efficient backfitting algorithm.
result SDCAM estimates functions without smoothness assumptions and outperforms existing methods.
Efficient method solves constrained Lasso problems.
problem Variable selection with prior information.
method Inexact augmented Lagrangian method exploiting second-order sparsity.
result Superior performance compared to first-order methods.
New model uses physics equations to predict complex systems without needing data.
problem Lack of training data for deep learning models of complex systems.
method Physics-constrained deep auto-regressive network.
result Model predicts non-linear dynamical systems with uncertainty quantification.
Proposes adaptive ridge regression for functional linear models with piecewise shapes.
problem Functional linear regression with unknown coefficient function.
method Adaptive piecewise function template with L2 penalization. result Improves predictive power and interpretability compared to standard methods.
New sparse Gaussian process method tackles unconstrained regression problems.
problem Dealing with physical systems that satisfy inequality constraints.
method Extends constrained Gaussian process by redefining hat basis functions.
result Reduces computational complexity from O(n3) to O(nm2). Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an l0-constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…
A new method resolves permutation issues in shuffled linear regression for large-scale applications.
problem Estimating latent features through linear transformation with unknown permutations.
method Spectral matching method to align spectral components of measurement and feature covariances.
result Achieves accurate estimates in shuffled LS and LASSO settings with sufficient samples.
This paper proposes MMD-SVR to improve SVR's margin distribution for better generalization.
problem Improving SVR's generalization performance by maximizing the margin distribution of the whole dataset.
method Introducing MMD-SVR with coupled constraints to convert a non-convex optimization problem into a convex one.
result MMD-SVR significantly improves prediction accuracy and generalization compared to classic SVR.
Shape-constrained symbolic regression improves model extrapolation with prior knowledge.
problem Improving model extrapolation with prior knowledge in symbolic regression.
method Shape-constrained symbolic regression using evolutionary algorithms with interval arithmetic.
result Models with shape constraints have improved extrapolation but lower accuracy on test sets.
Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
Proposes CLasso for high-dimensional regression with low-dimensional components.
problem Estimation and confidence intervals for low-dimensional parameters in high-dimensional models.
method Solves two estimating equations: zero-bias constraint and ℓ1-penalized procedure. result CLasso estimator is asymptotically normal and attains Cramér-Rao lower bound.
Three physics-constrained regression exercises for image velocimetry and turbulence modeling.
problem Image velocimetry and turbulence modeling challenges.
method Physics-constrained regression exercises implemented as toy problems.
result Python codes provided for all exercises.
Machine learning models predict brain age with systematic bias, corrected in this study.
problem Systematic bias in machine learning regression models for brain age prediction.
method General constrained optimization approach to correct bias.
result Our method effectively eliminates the bias from brain age predictions.
New method improves robustness of neural network-based debiasing.
problem Improving robustness of neural network-based debiasing.
method Moment-constrained learning for neural networks.
result Improved performance compared to state-of-the-art benchmarks.
Study shows how varying levels of supervision and orthonormality constraints affect generalization errors in subspace fitting.
problem Effects of varying levels of supervision and orthonormality constraints on generalization errors in subspace fitting.
method Flexible family of problems connecting unsupervised and supervised subspace fitting tasks, explored over a supervision-orthonormality plane.
result Generalization errors of subspace fitting problems follow double descent trends as they become more supervised and less orthonormally constrained.
No floating point, no multiplications, no problem! Training efficient networks for resource-constrained devices.
problem Designing efficient neural networks for resource-constrained devices without floating-point operations.
method Discretizing both in-network non-linearities and network weights to avoid floating-point and multiplication operations.
result Training networks without floating-point operations can achieve comparable performance to those using floating-point operations, with less memory usage.
KL-constrained API shows optimization issues and improved with regularization.
problem Optimization issues in KL-constrained API algorithms.
method Comparison of KL divergence as a constraint vs. regularizer, empirical evaluation.
result KL-constrained API is not guaranteed to converge and incurs linear regret.
Paper connects tensor regression and Gaussian processes for multi-way data analysis.
problem Learning high-order correlations from multi-way data.
method Demonstrates connections between low-rank tensor regression and Gaussian processes, proving oracle inequality and learning curve.
result Low-rank tensor regression is equivalent to constrained Bayesian inference in Gaussian processes, with learning dependent on eigenvalues and variable correlations.
This paper studies a class of exponential family models whose canonical parameters are specified as linear functionals of an unknown infinite-dimensional slope function. The optimal minimax rates of convergence for slope function estimation are established. The estimators that achieve the optimal rates are constructed …
We consider the problem of fitting a linear model to data held by individuals who are concerned about their privacy. Incentivizing most players to truthfully report their data to the analyst constrains our design to mechanisms that provide a privacy guarantee to the participants; we use differential privacy to model in…
New algorithm reduces ERM problem size while maintaining accuracy.
problem Empirical risk minimization problem size reduction.
method Adaptive Deterministic Uniform-Weight Trimming (ADUWT) algorithm.
result Uniform (1±ε) relative-error approximation for ERM objective. Improved scaling laws in linear regression using data reuse.
problem Sustainability of neural scaling laws when running out of new data.
method Data reuse in multi-pass stochastic gradient descent (multi-pass SGD) for M-dimensional linear models trained on N data with sketched features. result Multi-pass SGD achieves a test error of Θ(M1−b+L(1−b)/a) with L>N, improving scaling laws in data-constrained regimes. The purpose of these notes is to provide a systematic quantitative framework - in what is intended to be a "pedagogical" fashion - for discussing mean-reversion and optimization. We start with pair trading and add complexity by following the sequence "mean-reversion via demeaning -> regression -> weighted regression ->…
skscope simplifies sparsity-constrained optimization in Python.
problem Tedious mathematical deduction and programming for sparsity-constrained optimization.
method Introduces skscope, a Python library that allows users to solve sparsity-constrained optimization problems by just programming the objective function.
result skscope enables state-of-the-art solvers to quickly attain sparse solutions in high-dimensional spaces, achieving up to 80x speedup.
Improves SDCA convergence for convex objectives with linear constraints.
problem Minimizing convex objectives with linear constraints under gradient-Lipschitz assumption failure.
method Shifted Stochastic Dual Coordinate Ascent (SDCA) under smoothness assumption.
result Obtains linear convergence rate for Poisson regression and Hawkes process objectives.
A scalable PyTorch framework for non-crossing quantile regression.
problem Non-crossing quantile regression to avoid impossible negative probability densities.
method CJQR-ALM combining Augmented Lagrangian Method, differentiable pinball loss, and L-BFGS optimization.
result Achieves near-zero crossing rates on large datasets within minutes.
Unified R packages for forecast reconciliation of constrained series.
problem Improving accuracy and coherence of forecasts for linearly constrained multiple time series.
method Classical and machine learning-based linear reconciliation approaches for cross-sectional, temporal, and cross-temporal frameworks.
result Unified toolbox for forecast reconciliation in R.