New sparse Gaussian process method tackles unconstrained regression problems.
problem Dealing with physical systems that satisfy inequality constraints.
method Extends constrained Gaussian process by redefining hat basis functions.
result Reduces computational complexity from O ( n 3 ) O(n^{3}) O ( n 3 ) to O ( n m 2 ) O(nm^{2}) O ( n m 2 ) . The study connects fairness constraints with optimal transport to derive new insights in classification.
problem Ensuring fairness in classification models without sacrificing performance.
method Using Wasserstein barycenters and optimal transport, the study characterizes optimal classification functions under fairness constraints.
result Maximizing fairness under demographic parity is equivalent to solving a regression problem.
Study shows 'Ordinal Neural Collapse' in deep OR tasks, revealing simple geometric relationships.
problem Understanding neural collapse in deep Ordinal Regression tasks.
method Combining cumulative link models and Unconstrained Feature Model to investigate neural collapse.
result Demonstrates 'Ordinal Neural Collapse' (ONC) with three key properties.
Method uses DNNs to approximate functions with specific asymptotic behavior.
problem Approximating functions with given asymptotic behavior.
method Specifically constructed terms combined with unconstrained DNN.
result Enforcing asymptotic behavior leads to better approximation and faster convergence.
Quantum computing speeds up linear regression training.
problem Reducing training time for machine learning models.
method Formulated regression problem as QUBO, used D-Wave 2000Q for adiabatic optimization.
result Quantum approach achieves up to 2.8x speedup on larger datasets.
Study on consistency of ML methods for moving objects in non-stationary environments.
problem Consistency of machine learning methods for moving objects in non-stationary environments.
method Least squares, ridge regression, and ℓ s \ell_s ℓ s -penalized least squares methods under non-stationary spatial-temporal sampling. result Consistency and asymptotic normality of the estimates under weak conditions.
Optimal SD improves ridge regression performance strictly and precisely.
problem Improving ridge regression performance through self-distillation.
method Analyzes unconstrained SD for ridge regression, deriving optimal mixing weight and asymptotic risk.
result Optimal SD strictly improves ridge regression performance, with exact risk equivalents derived.
A scalable PyTorch framework for non-crossing quantile regression.
problem Non-crossing quantile regression to avoid impossible negative probability densities.
method CJQR-ALM combining Augmented Lagrangian Method, differentiable pinball loss, and L-BFGS optimization.
result Achieves near-zero crossing rates on large datasets within minutes.
New algorithm reduces online regression error in RKHS.
problem Online regression with time-varying functions in RKHS.
method Hierarchical Vovk-Azoury-Warmuth with discounting.
result Achieves optimal dynamic regret with O ( T 2 / 3 P T 1 / 3 + T ln T ) O(T^{2/3}P_T^{1/3} + \sqrt{T}\ln T) O ( T 2/3 P T 1/3 + T ln T ) regret bound. Unconstrained MLIPs outperform constrained ones in accuracy and speed.
problem Improving the efficiency and accuracy of machine-learned interatomic potentials.
method Investigated unconstrained models trained on large datasets compared to physically constrained models.
result Unconstrained MLIPs can be superior in accuracy and speed compared to physically constrained models.
Our paper explains deep neural collapse in multiple layers.
problem Understanding deep neural collapse in multi-layered neural networks.
method Generalized unconstrained features model for deep networks.
result Deep unconstrained features model exhibits deep neural collapse.
Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newto…
New approach reduces unconstrained linear bandits to simpler optimization problems.
problem Unconstrained linear bandits problem.
method Perturbation-based approach combined with comparator-adaptive OLO algorithms.
result First high-probability guarantees for both static and dynamic regret in unconstrained linear bandits.
We clarify what fairness guarantees we can and cannot expect to follow from unconstrained machine learning. Specifically, we characterize when unconstrained learning on its own implies group calibration, that is, the outcome variable is conditionally independent of group membership given the score. We show that under r…
Develops parameter-free online mirror descent for optimal dynamic regret.
problem Optimal online linear optimization in unbounded domains.
method Modified online mirror descent framework for parameter-free algorithms.
result First unconstrained online linear optimization achieving optimal dynamic regret.
Building upon recent advances in entropy-regularized optimal transport, and upon Fenchel duality between measures and continuous functions , we propose a generalization of the logistic loss that incorporates a metric or cost between classes. Unlike previous attempts to use optimal transport distances for learning, our …
A new optimization algorithm improves convergence in unconstrained problems.
problem Unconstrained optimization problems.
method Element-wise relaxed scalar auxiliary variable (E-RSAV) algorithm.
result Improved convergence and alignment of modified and original energy.
Unconstrained models learn physical symmetries effectively with simple data augmentation.
problem Ensuring physical symmetries in machine learning models.
method Rigorous metrics to measure symmetry content, data augmentation strategy, architectural analysis.
result Unconstrained models can learn approximate equivariant behavior with simple data augmentation.
Paper extends LME models to allow sign constraints on coefficients with SDTN random effects.
problem Inference with sign constraints on random effects in LME models.
method Proposes SDTN distribution for random effects and develops likelihood-based approaches for estimation.
result Proposed constrained model improves real-world interpretations and achieves satisfactory performance.
The paper explores solving inverse problems for ODEs with and without constraints.
problem Understanding when second order ODEs can represent Lagrangian models with or without constraints.
method Geometric techniques to address the inverse problem for both constrained and unconstrained systems of second order ODEs.
result The constrained case presents more ambiguities and complexities than the unconstrained one.
Proposes ConstraintMatch for semi-supervised clustering with unconstrained data.
problem Leveraging unconstrained data alongside constraints for clustering models.
method Semi-supervised context with pseudo-constraining and pseudo-labeling mechanisms.
result Demonstrates effectiveness of ConstraintMatch over baselines.
This paper uses quantum computing to solve sparse linear regression problems efficiently.
problem Sparse linear regression to identify important features from a large set of variables.
method Formulates the ℓ 0 \ell_0 ℓ 0 optimization problem as a QUBO problem and solves it using the D-Wave adiabatic quantum computer. result The QUBO solution matches the optimal solution for a wide range of sparsity penalty values across datasets.
Logistic regression is commonly used for modeling dichotomous outcomes. In the classical setting, where the number of observations is much larger than the number of parameters, properties of the maximum likelihood estimator in logistic regression are well understood. Recently, Sur and Candes have studied logistic regre…
In recent years, stochastic gradient descent (SGD) methods and randomized linear algebra (RLA) algorithms have been applied to many large-scale problems in machine learning and data analysis. We aim to bridge the gap between these two methods in solving constrained overdetermined linear regression problems---e.g., $\el…
This paper extends neural collapse to class-imbalanced datasets using an unconstrained ReLU feature model.
problem Understanding neural collapse in class-imbalanced datasets with cross-entropy loss.
method Generalized neural collapse to class-imbalanced settings using an unconstrained ReLU feature model.
result Class-means converge to orthogonal vectors with different lengths, and classifier weights align to these vectors.
New algorithms reduce online learning regret by tracking gradient variation.
problem Online learning with unconstrained losses and gradient variation.
method Parameter-free algorithms with adaptive updates for L L L -smooth convex losses. result Regret bounds of order O ~ ( ∥ u ∥ V T ( u ) + L ∥ u ∥ 2 + G 4 ) \widetilde{O}(\|u\|\sqrt{V_T(u)} + L\|u\|^2+G^4) O ( ∥ u ∥ V T ( u ) + L ∥ u ∥ 2 + G 4 ) achieved without prior knowledge of comparator norm or Lipschitz constant. Second-order methods improve differential privacy in convex optimization.
problem Improving differential privacy in convex optimization.
method Developed a private variant of the regularized cubic Newton method for strongly convex loss functions.
result Achieves quadratic convergence and optimal excess loss for strongly convex loss functions.
A new L-BFGS method tackles large-scale optimization with fewer evaluations.
problem Efficiently solving large-scale unconstrained optimization problems.
method Proposes a regularized L-BFGS method with line search techniques.
result Shows global convergence and robust performance in numerical tests.
Integrates prediction models into portfolio optimization for better asset allocation.
problem Traditional portfolio optimization ignores prediction models, leading to suboptimal decisions.
method Developed a framework that combines regression prediction with mean-variance optimization, providing analytical solutions and neural-network-based optimization for inequality constraints.
result Demonstrated through simulations that integrating prediction models improves portfolio performance.
New algorithm minimizes cumulative loss in dynamic linear bandits without prior knowledge of comparator switches.
problem Minimizing cumulative loss in dynamic linear bandits with unknown number of switches.
method Combining several bandit algorithms to adapt to unknown number of switches without prior knowledge.
result First algorithm achieving optimal regret guarantee of O ( d ( 1 + S T ) T ) \mathcal{O}\big(\sqrt{d(1+S_T) T}\big) O ( d ( 1 + S T ) T ) up to poly-logarithmic terms. We propose an online convex optimization algorithm (RescaledExp) that achieves optimal regret in the unconstrained setting without prior knowledge of any bounds on the loss functions. We prove a lower bound showing an exponential separation between the regret of existing algorithms that require a known bound on the los…
Proposes a Gaussian process model for constrained dynamics learning.
problem Challenges in identifying constrained dynamics of mechanical systems.
method Combines analytical mechanics with Gaussian process regression.
result Improves data efficiency and constraint integrity in predictions.
VAV method optimizes learning rate for faster, stable SGD convergence.
problem Optimizing learning rate for efficient and stable machine learning models.
method Energy-based self-adaptive learning rate with auxiliary variable r r r . result VAV method achieves faster convergence and superior stability with larger learning rates.
We consider a variant of online convex optimization in which both the instances (input vectors) and the comparator (weight vector) are unconstrained. We exploit a natural scale invariance symmetry in our unconstrained setting: the predictions of the optimal comparator are invariant under any linear transformation of th…
Solves VaR-constrained portfolio optimization in markets with stochastic volatility.
problem Optimizing portfolio in markets with stochastic volatility under VaR constraints.
method Dynamic programming approach to Heston's stochastic volatility model.
result Optimal investment strategy linked to unconstrained problem via a vega-neutral derivative.
Cookbook transforms constrained statistical inference into unconstrained problems.
problem Transforming constrained statistical inference into unconstrained problems.
method Bijective and diffeomorphisms parametrizations.
result Maintains statistical inference properties like identifiability.
A method for fast estimation of Wasserstein distances using sliced Wasserstein distances.
problem Efficiently computing Wasserstein distances for multiple pairs of distributions.
method Regression on sliced Wasserstein distances to predict true Wasserstein distances.
result The proposed method provides a better approximation of Wasserstein distance than state-of-the-art models, especially in low-data regimes.
Novel method uses Gaussian process to estimate particle sizes from scattering data.
problem Estimating particle size distributions from noisy optical scattering measurements.
method Constrained Gaussian process regression with normalization constraints.
result Accurately reconstructs particle size distributions from noisy data.
DP-GD achieves dimension-independent convergence for unconstrained private GLMs.
problem Differentially private empirical risk minimization for unconstrained GLMs.
method Differentially private gradient descent (DP-GD).
result DP-GD achieves an excess empirical risk of $ ilde O\left(\sqrt{ exttt{rank}}/εn
ight)$ for unconstrained GLMs.
In spite of the recent surge of interest in quantile regression, joint estimation of linear quantile planes remains a great challenge in statistics and econometrics. We propose a novel parametrization that characterizes any collection of non-crossing quantile planes over arbitrarily shaped convex predictor domains in a…
Single tree outperforms random forest in testing accuracy.
problem The challenge of improving single decision tree performance.
method Gradient-based entire tree optimization framework, scaled sigmoid approximation, numerical stability algorithm, subtree polish strategy.
result Optimized single tree outperforms classic random forest by 2.03% on average.
FIGS improves prediction performance while maintaining interpretability, especially in medical domains.
problem Lack of interpretability in machine learning models, particularly in high-stakes domains like medicine.
method Generalizes CART algorithm to grow multiple trees in summation, combining logical rules with addition.
result FIGS achieves state-of-the-art prediction performance and derives interpretable clinical decision instruments (CDIs).
Yau's Affine Normal Descent optimizes smooth unconstrained problems with geometrically adapted directions.
problem Optimizing smooth unconstrained problems with geometrically adapted directions.
method Yau's Affine Normal Descent (YAND) uses the equi-affine normal of level-set hypersurfaces as search directions.
result YAND converges globally under standard smoothness assumptions and locally quadratically near nondegenerate minimizers.
Study optimal portfolio management with periodic evaluations in stochastic models, considering convex constraints.
problem Optimal portfolio management under ratio-type periodic evaluations in stochastic factor models with convex trading constraints.
method Transformed infinite horizon optimal control problem into an auxiliary terminal wealth optimization problem. Introduced an auxiliary unconstrained optimization problem in a modified market model. Used martingale duality approach to establish dual minimizer and optimal unconstrained wealth process.
result Derived and verified the optimal constrained portfolio process for the original problem over an infinite horizon.
Adam converges with high probability under unconstrained non-convex smooth stochastic optimizations.
problem Theoretical limitations of Adam's convergence under unconstrained non-convex smooth stochastic optimizations.
method Deep analysis of Adam's convergence rate under affine variance noise, without bounded gradient assumptions.
result Adam converges to the stationary point with a high probability rate of $\mathcal{O}\left({
m poly}(\log T)/\sqrt{T}
ight)$ .
Paper analyzes regret bounds for unconstrained online optimization.
problem Minimizing regret in dynamic online learning for strongly convex and smooth functions.
method Preconditioned OGD, Online Optimistic Newton (OON), multiple gradient queries.
result Achieves O ( C 2 , T ∗ ) O(C^*_{2,T}) O ( C 2 , T ∗ ) regret bound with one gradient query per round. A drawdown constraint forces the current wealth to remain above a given function of its maximum to date. We consider the portfolio optimisation problem of maximising the long-term growth rate of the expected utility of wealth subject to a drawdown constraint, as in the original setup of Grossman and Zhou (1993). We wor…
New methods solve complex optimization problems in machine learning.
problem Challenges in stochastic bilevel optimization with constraints and high variables.
method Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems.
result Comprehensive convergence theory for both unconstrained and constrained cases.