Paper introduces a new topological loss for better convergence.
problem Optimizing topological losses for model's desired topological behavior.
method Introduces a new regularized topology-aware loss function.
result Guarantees efficient optimization of the new loss function.
This work improves diffusion models by estimating the optimal loss value for better training diagnostics.
problem The optimal loss value of diffusion models is unknown and not indicative of absolute data-fitting quality.
method Derive the optimal loss in closed form and develop effective estimators, including a stochastic variant.
result Unlocking the optimal loss as a metric for diagnosing training quality of diffusion models.
We analyze the optimization landscape of α-loss in logistic models.
problem Optimization landscape of α-loss in logistic models.
method Tools from strictly-locally-quasi-convex functions and geometric techniques.
result Evolution of optimization landscape with respect to α.
As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions…
Optimizing proper loss yields calibrated models under specific conditions.
problem Understanding when optimizing proper loss functions leads to calibrated predictions.
method Local optimality condition and Lipschitz functions.
result Predictors with local optimality are nearly calibrated and nearly locally optimal.
New PG losses improve decision optimization in misspecified models.
problem Improving decision optimization in models that are not perfectly specified.
method Introducing Perturbation Gradient (PG) losses to connect decision loss with directional derivatives and optimizing using gradient techniques.
result PG losses yield best-in-class policies asymptotically, even in misspecified settings.
Study visualizes actor-critic loss landscapes for inventory optimization.
problem Difficulties in solving multi-store dynamic inventory control problems.
method Low-dimensional visualizations of actor loss function.
result Loss landscapes favor optimal policies in reinforcement learning.
Study on optimal fees in hedge funds with first-loss compensation.
problem Determining the best fee structure for hedge funds with first-loss compensation.
method Solved the manager's non-concave utility maximization problem, calculated Pareto optimal first-loss schemes, and maximized a decision criterion on this set.
result Traditional fees are not Pareto optimal, and the preferred first-loss coverage guarantee varies with investor and market factors.
This work introduces a new loss function to improve the efficiency of optimization-based PDE solvers.
problem Optimization-based PDE solvers converge slowly and are inefficient compared to classical iterative solvers.
method Proposes a novel Stabilized Gradient Residual (SGR) loss function to modulate the condition number.
result The SGR loss achieves orders-of-magnitude faster convergence than the MSE loss in both ODIL and PINNs frameworks.
We study differentially private (DP) algorithms for stochastic convex optimization (SCO). In this problem the goal is to approximately minimize the population loss given i.i.d. samples from a distribution over convex and Lipschitz loss functions. A long line of existing work on private convex optimization focuses on th…
Study proposes a differentiable surrogate loss function for optimizing Fβ score in binary classification with imbalanced data.
problem Non-differentiability of Fβ score makes it unsuitable for optimization by gradient-based learning. method Investigated relationship between Fβ score and loss functions, proposed a differentiable surrogate loss function. result Gradient paths of the proposed surrogate Fβ loss function approximate the gradient paths of the Fβ score. Optimizes hybrid insurance contracts for heavy-tailed losses.
problem Providing insurance against heavy-tailed losses with finite expected loss.
method Combines traditional and parametric insurance, using a Pareto-type criterion for optimization.
result The hybrid contract outperforms traditional contracts in simulations and real data.
New method uses FY loss for better inverse optimization.
problem Estimating unknown parameters from noisy and suboptimal solutions.
method Fenchel-Young loss approach for efficient gradient-based optimization.
result Significant improvement in parameter estimation accuracy and computational speed.
In this paper, we study two classes of optimal reinsurance models from perspectives of both insurers and reinsurers by minimizing their convex combination where the risk is measured by a distortion risk measure and the premium is given by a distortion premium principle. Firstly, we show that how optimal reinsurance mod…
This work classifies SOC loss functions based on their gradient properties.
problem Optimizing noisy systems in stochastic optimal control.
method Grouping loss functions into classes with the same gradient expectation.
result Different loss functions have the same optimization landscape but differ in gradient variance.
MTAdam optimizes multiple loss terms in neural models, balancing gradients dynamically.
problem Balancing multiple loss terms in neural model training is challenging and computationally demanding.
method Generalized Adam algorithm that computes separate derivatives and balances gradients across layers dynamically.
result Training with MTAdam leads to faster recovery from suboptimal initial loss weighting and matches conventional training outcomes.
A new method approximates expected empirical loss for stochastic deep learning tasks.
problem Determining optimal step sizes for stochastic gradient descent in deep learning.
method Applying one-dimensional function fitting to noisy losses of vertical cross sections to approximate expected empirical loss.
result The method leads to a robust and straightforward optimization method that performs well across datasets and architectures.
Optimal unimodal fitting for linear loss functions in a sequential, efficient manner.
problem Optimal unimodal transformation of univariate model scores under linear loss functions.
method Proposes a sequential approach to estimate the optimal rectangular fit for observed samples with each new sample.
result Sequential approach achieves optimal efficiency with logarithmic time complexity per iteration.
Many real-world analytics problems involve two significant challenges: prediction and optimization. Due to the typically complex nature of each challenge, the standard paradigm is predict-then-optimize. By and large, machine learning tools are intended to minimize prediction error and do not account for how the predict…
This paper improves risk bounds and calibration for smart predict-then-optimize method.
problem Improving risk bounds and calibration for smart predict-then-optimize method.
method Develops risk bounds and uniform calibration results for the SPO+ loss relative to the SPO loss.
result Empirical minimizer of the SPO+ loss achieves low excess true risk with high probability.
Near-optimal algorithms for predicting across multiple loss functions efficiently.
problem Predicting optimally across various loss functions simultaneously.
method Developed near-optimal online and offline learning algorithms for omniprediction.
result Achieved near-optimal complexity for both online and offline settings.
Optimizing full likelihoods adapts loss scales and shapes for robust modeling.
problem Rigid loss functions limit model adaptability and robustness.
method Optimize full likelihoods with adjustable parameters.
result Adaptive tuning of loss scales and shapes improves model robustness.
KOALA optimizes neural networks by treating loss as noisy measurements.
problem Training neural networks with changing loss functions.
method Adopting Kalman filtering for stochastic optimization.
result KOALA yields estimates on par with state-of-the-art methods.
Motivated by the industry practice of pairs trading, we study the optimal timing strategies for trading a mean-reverting price spread. An optimal double stopping problem is formulated to analyze the timing to start and subsequently liquidate the position subject to transaction costs. Modeling the price spread by an Orn…
A new convex loss function optimizes set predictions with balanced size and coverage.
problem Optimizing set predictions with balanced size and coverage.
method Proposes a convex loss function using Choquet integrals for nondecreasing subset-valued functions.
result Optimal trade-offs between conditional probabilistic coverage and set size.
DLM for BNNs fails to improve over ELBO optimization.
problem Performance of DLM for Bayesian Neural Networks (BNNs).
method Direct Loss Minimization (DLM) compared to ELBO optimization.
result DLM does not significantly improve over ELBO optimization for BNNs.
The paper examines optimal insurance design using Lambda-Value-at-Risk.
problem Optimal insurance design based on Lambda-Value-at-Risk.
method Analyzes optimal insurance solutions using Lambda-Value-at-Risk and closed-form expressions.
result Truncated stop-loss indemnity is optimal under certain conditions.
We consider the problem of estimating a low-rank matrix from a noisy observed matrix. Previous work has shown that the optimal method depends crucially on the choice of loss function. In this paper, we use a family of weighted loss functions, which arise naturally for problems such as submatrix denoising, denoising wit…
Efficient algorithms find optimal monotone transforms for calibration under strictly convex losses.
problem Calibrating estimations to improve performance with monotone transforms.
method Proposed linear-time and space algorithm for finding optimal monotone transforms for specific loss functions. Also proposed an anytime algorithm with linear space and pseudo-linearithmic time complexity.
result Optimal monotone transforms are unique and can be found efficiently for various strictly convex loss functions.
We address the problem of aggregating an ensemble of predictors with known loss bounds in a semi-supervised binary classification setting, to minimize prediction loss incurred on the unlabeled data. We find the minimax optimal predictions for a very general class of loss functions including all convex and many non-conv…
Paper explores challenges in training PINNs and loss landscape effects.
problem Challenges in training Physics-Informed Neural Networks (PINNs) due to loss landscape issues.
method Examined gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, and introduced NysNewton-CG (NNCG).
result Adam+L-BFGS outperforms other optimizers, and NysNewton-CG significantly improves PINN performance.
Boosting can efficiently optimize any loss function without requiring first-order information.
problem Boosting's efficiency in optimizing loss functions without first-order information.
method Extending gradient-based optimization to use only zeroth-order information.
result Boosting can optimize any loss function efficiently, including non-convex, non-differentiable, and non-continuous ones.
Much of the focus in machine learning research is placed in creating new architectures and optimization methods, but the overall loss function is seldom questioned. This paper interprets machine learning from a multi-objective optimization perspective, showing the limitations of the default linear combination of loss f…
In this paper we study a class of insurance products where the policy holder has the option to insure k of its annual Operational Risk losses in a horizon of T years. This involves a choice of k out of T years in which to apply the insurance policy coverage by making claims against losses in the given year. The…
Paper introduces a new G⋆ regret measure for online convex optimization with smooth losses.
problem Online convex optimization with smooth losses.
method Introduces a new G⋆ regret measure that depends on the cumulative squared gradient norm. result The G⋆ regret can be arbitrarily sharper than existing measures when losses have vanishing curvature. We propose a robust adversarial prediction framework for general multiclass classification. Our method seeks predictive distributions that robustly optimize non-convex and non-continuous multiclass loss metrics against the worst-case conditional label distributions (the adversarial distributions) that (approximately) m…
The paper explores conditions for predicting optimization performance.
problem Lack of formal theoretical guarantees linking prediction and optimization performance.
method Exploring conditions for asymptotic convergence and exact quantification of optimization performance.
result Explicit theoretical relationship between prediction and optimization performance.
We find a convex model for traditional nonlinear regression under L2 loss.
problem Nonlinear regression under L2 loss with non-convex optimization.
method Showed a convex nonlinear regression model for least squares problem.
result Existence of a convex model simplifies training complex systems.
Hill-ADAM optimizes loss landscapes by exploring state space deterministically.
problem Escaping local minima in loss landscapes.
method Hill-ADAM alternates between minimizing and maximizing error to explore the loss space.
result Hill-ADAM finds the global minimum state in loss landscapes.
New α-divergence loss function improves neural density ratio estimation.
problem Optimization challenges in existing DRE methods, especially overfitting and high sample requirements.
method Derived α-divergence loss function (α-Div) for neural density ratio estimation. result The α-divergence loss function (α-Div) offers stable and effective optimization for DRE. Decision trees improve decision-making by optimizing predictions of unknown parameters.
problem Optimizing decisions based on predicted unknown parameters.
method SPO Trees (SPOTs) for training decision trees under the SPO loss function.
result SPOTs provide higher quality decisions and significantly lower model complexity compared to other machine learning approaches.
The minimization of loss functions is the heart and soul of Machine Learning. In this paper, we propose an off-the-shelf optimization approach that can minimize virtually any non-differentiable and non-decomposable loss function (e.g. Miss-classification Rate, AUC, F1, Jaccard Index, Mathew Correlation Coefficient, etc…
We establish linear regret bounds for convex smooth losses using Fenchel-Young losses.
problem Establishing linear regret bounds for convex smooth losses.
method Constructing a convex smooth surrogate loss using Fenchel-Young losses generated by the convolutional negentropy.
result We derive a smooth loss with a linear surrogate regret bound.
This research analyzes the consistency of convex and nonconvex surrogate losses for adversarially robust classification.
problem Ensuring classifiers are robust to adversarial perturbations.
method Analysis of convex and nonconvex surrogate losses through the lens of calibration.
result No convex surrogate loss is calibrated with respect to the adversarial 0-1 loss for linear models, but nonconvex losses can be calibrated under certain conditions.
We propose an online convex optimization algorithm (RescaledExp) that achieves optimal regret in the unconstrained setting without prior knowledge of any bounds on the loss functions. We prove a lower bound showing an exponential separation between the regret of existing algorithms that require a known bound on the los…
We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
Optimizes exp-concave losses with a new risk bound.
problem Optimizing exp-concave losses with stochastic convex optimization.
method Empirical Risk Minimization with a unified geometric assumption and local norms.
result Provides an O(d/n+log(1/δ)/n) excess risk bound. This work justifies neural collapse under MSE loss and analyzes the optimization landscape.
problem Understanding neural collapse in deep neural networks under MSE loss.
method Global landscape analysis of vanilla nonconvex MSE loss.
result The only global minimizers are neural collapse solutions.