Supervised learning is an active research area, with numerous applications in diverse fields such as data analytics, computer vision, speech and audio processing, and image understanding. In most cases, the loss functions used in machine learning assume symmetric noise models, and seek to estimate the unknown function …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “parameter loss”
Paper introduces a new robust loss function for RL.
problem Heuristic selection of threshold parameters in quantile Huber loss.
method Derived from Wasserstein distance, captures noise in quantile values.
result Enhances robustness against outliers and enables parameter adjustment.
Optimizing full likelihoods adapts loss scales and shapes for robust modeling.
problem Rigid loss functions limit model adaptability and robustness.
method Optimize full likelihoods with adjustable parameters.
result Adaptive tuning of loss scales and shapes improves model robustness.
New loss function improves accuracy of MRI parameter estimation.
problem Systematic errors in parameter estimates at low SNR.
method Developed and implemented negative log Rician likelihood (NLR) loss.
result NLR loss shows higher accuracy in parameter estimation than MSE loss at low SNR.
Paper improves PAC-Bayes bounds for various loss types.
problem Improving PAC-Bayes bounds for different types of losses.
method Introducing new high-probability PAC-Bayes bounds for bounded and general tail behaviors losses, and extending to anytime-valid bounds.
result New fast-rate and mixed-rate bounds for losses with bounded ranges, and parameter-free bounds for losses with general tail behaviors.
Proposes a generalized XGBoost method for nonconvex loss functions.
problem Limited to convex loss functions in XGBoost.
method Extends XGBoost to use nonconvex loss functions and multivariate loss functions.
result Generalized XGBoost method can model multiple parameters in various distributions.
New risk control method for non-monotonic losses in complex parameters.
problem Controlling risk for non-monotonic losses with multidimensional parameters.
method Stability-based guarantees for generic algorithms applied to non-monotonic losses.
result Guarantees depend on algorithm stability, with looser guarantees for unstable algorithms.
A new method optimizes neural sequence models for better task performance.
problem Training neural sequence models with maximum likelihood estimation ignores task losses.
method Maximum likelihood guided parameter search (MGS) in the parameter space.
result MGS optimizes sequence-level losses, reducing repetition and non-termination.
We analyze impermanent loss in AMMs and show G3Ms are simplest.
problem Understanding impermanent loss in automated market makers.
method Developed a general framework and analyzed Geometric Mean Market Makers (G3Ms).
result G3Ms have the simplest impermanent loss characteristics.
In order to quantize the gate parameters of the LSTM (Long Short-Term Memory) neural network model with almost no recognition performance degraded, a new quantization method named Quantization Loss Re-Learn Method is proposed in this paper. The method does lossy quantization on gate parameters during training iteration…
The Huber loss is a robust loss function used for a wide range of regression tasks. To utilize the Huber loss, a parameter that controls the transitions from a quadratic function to an absolute value function needs to be selected. We believe the standard probabilistic interpretation that relates the Huber loss to the H…
We analyze the semi-hard triplet loss using Edgeworth expansion for better understanding of its behavior.
problem Understanding the behavior of the semi-hard triplet loss function.
method Developed a higher-order asymptotic analysis using the Edgeworth expansion.
result Derived explicit Edgeworth expansions revealing first-order corrections in terms of the third cumulant.
Relative Loss Bounds for On-line Density Estimation with the Exponential Family of Distributionscs.LG
We consider on-line density estimation with a parameterized density from the exponential family. The on-line algorithm receives one example at a time and maintains a parameter that is essentially an average of the past examples. After receiving an example the algorithm incurs a loss which is the negative log-likelihood…
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce mi…
New method uses FY loss for better inverse optimization.
problem Estimating unknown parameters from noisy and suboptimal solutions.
method Fenchel-Young loss approach for efficient gradient-based optimization.
result Significant improvement in parameter estimation accuracy and computational speed.
Decision trees improve decision-making by optimizing predictions of unknown parameters.
problem Optimizing decisions based on predicted unknown parameters.
method SPO Trees (SPOTs) for training decision trees under the SPO loss function.
result SPOTs provide higher quality decisions and significantly lower model complexity compared to other machine learning approaches.
We present a generalization of the Cauchy/Lorentzian, Geman-McClure, Welsch/Leclerc, generalized Charbonnier, Charbonnier/pseudo-Huber/L1-L2, and L2 loss functions. By introducing robustness as a continuous parameter, our loss function allows algorithms built around robust loss minimization to be generalized, which imp…
Improved DP SO with large Lipschitz parameters, handling outliers and heavy-tailed data.
problem Differential privacy in stochastic optimization with large Lipschitz parameters.
method Assumes bounded k-th order moments, provides linear-time algorithms for smooth convex and non-smooth convex losses.
result Improved risk bounds scaling with k-th moment, not uniform Lipschitz parameter.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
New loss function connects learning rate and momentum.
problem Finding optimal learning rate and momentum empirically.
method Proposes a new information-theoretical loss function.
result Loss, learning rate, and momentum are closely connected.
A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictorsstat.ML
The study finds a trade-off between model size, test loss, and training loss for linear predictors.
problem Finding the optimal balance between model size, test loss, and training loss for linear predictors.
method Established an algorithm and distribution-independent trade-off using non-asymptotic analysis.
result Models with low test loss are either classical (close to noise level training loss) or modern (large number of parameters).
Smoothness analysis of adversarial training reveals constraints cause more non-smoothness.
problem Non-smoothness of adversarial training loss function.
method Analyzed the smoothness of adversarial training loss function using optimal attacks for model parameters.
result The constraint causes more non-smoothness than constraint.
Investigates ways to train larger models with fewer resources, finding that test loss depends only on the actual number of trainable parameters.
problem Training larger models for cheaper under hardware constraints.
method Emulates an increase in effective parameters using frozen random parameters or fast structured transforms.
result Scaling laws cannot be deceived by spurious parameters; test loss depends only on the actual number of trainable parameters.
Deep ReLU networks with extra parameters have mostly good loss landscapes.
problem Finding good local minima in the loss landscape of deep neural networks.
method Analyzing shallow and deep ReLU networks with extra parameters on a generic dataset.
result Most activation patterns correspond to regions with no bad local minima.
Neural networks enjoy widespread use, but many aspects of their training, representation, and operation are poorly understood. In particular, our view into the training process is limited, with a single scalar loss being the most common viewport into this high-dimensional, dynamic process. We propose a new window into …
Incorporating Expert Opinion on Observable Quantities into Statistical Models -- A General Frameworkstat.ML
The paper introduces a method to incorporate expert opinion on observable quantities into statistical models.
problem Tackling the challenge of integrating expert knowledge on observable quantities into statistical models.
method The approach involves updating a prior belief using a loss function that reflects expert opinion on observable quantities.
result The method allows for a flexible specification of expert opinion and is straightforward to implement.
Uncertainty sampling, a popular active learning algorithm, is used to reduce the amount of data required to learn a classifier, but it has been observed in practice to converge to different parameters depending on the initialization and sometimes to even better parameters than standard training on all the data. In this…
The predict-then-optimize framework is fundamental in many practical settings: predict the unknown parameters of an optimization problem, and then solve the problem using the predicted values of the parameters. A natural loss function in this environment is to consider the cost of the decisions induced by the predicted…
Many banks adopt the Loss Distribution Approach to quantify the operational risk capital charge under Basel II requirements. It is common practice to estimate the capital charge using the 0.999 quantile of the annual loss distribution, calculated using point estimators of the frequency and severity distribution paramet…
New loss function helps learn unstable dynamical systems.
problem Gradient descent fails to learn unstable dynamical systems.
method Introduced a time-weighted logarithmic loss function.
result Time-weighted loss function effectively learns unstable systems.
KOALA optimizes neural networks by treating loss as noisy measurements.
problem Training neural networks with changing loss functions.
method Adopting Kalman filtering for stochastic optimization.
result KOALA yields estimates on par with state-of-the-art methods.
The Calculus of Expected Loss: Backtesting Parameter-Based Expected Loss in a Basel II Frameworkq-fin.RM
The dependency structure of credit risk parameters is a key driver for capital consumption and receives regulatory and scientific attention. The impact of parameter imperfections on the quality of expected loss (EL) in the sense of a fair, unbiased estimate of risk expenses however is barely covered. So far there are n…
New loss function handles uncertain constraints in CSLO problems.
problem Handling uncertain inequality constraints in CSLO with machine learning predictions.
method Introduces SPO-RC loss and SPO-RC+ surrogate, trains on truncated datasets, corrects bias.
result SPO-RC+ effectively manages constraint uncertainty and improves performance.
Estimates parameters in a deviated Gaussian mixture model.
problem Testing goodness-of-fit between a known function and a mixture of experts.
method Constructs novel Voronoi-based loss functions to estimate parameters.
result Characterizes local convergence rates of parameter estimation more accurately.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
Gating is a key feature in modern neural networks including LSTMs, GRUs and sparsely-gated deep neural networks. The backbone of such gated networks is a mixture-of-experts layer, where several experts make regression decisions and gating controls how to weigh the decisions in an input-dependent manner. Despite having …
The L1 loss landscape of neural nets near local minima behaves differently, revealing exponential decay and increased vertex density.
problem Understanding the L1 loss landscape of neural nets near local minima.
method Iterative minimization of the loss function on adjacent vertices of the Deep ReLU Simplex algorithm.
result Exponential decay of loss levels and increased vertex density around local minima.
Large learning rates cause parameter instability, leading to better generalization.
problem Understanding why deep neural networks perform well despite operating outside the traditional stability regime.
method Analyzing the effect of large learning rates on the orientation of Hessian eigenvectors and parameter exploration.
result Large learning rates induce parameter instability, leading to better generalization through exploration of flatter regions of the loss landscape.
In this paper we revisit the weighted likelihood bootstrap, a method that generates samples from an approximate Bayesian posterior of a parametric model. We show that the same method can be derived, without approximation, under a Bayesian nonparametric model with the parameter of interest defined as minimising an expec…
We find a convex model for traditional nonlinear regression under L2 loss.
problem Nonlinear regression under L2 loss with non-convex optimization.
method Showed a convex nonlinear regression model for least squares problem.
result Existence of a convex model simplifies training complex systems.
Estimating Quantile Families of Loss Distributions for Non-Life Insurance Modelling via L-momentsq-fin.RM
This paper discusses different classes of loss models in non-life insurance settings. It then overviews the class Tukey transform loss models that have not yet been widely considered in non-life insurance modelling, but offer opportunities to produce flexible skewness and kurtosis features often required in loss modell…
Most of the banks' operational risk internal models are based on loss pooling in risk and business line categories. The parameters and outputs of operational risk models are sensitive to the pooling of the data and the choice of the risk classification. In a simple model, we establish the link between the number of ris…
Regulatory requirements dictate that financial institutions must calculate risk capital (funds that must be retained to cover future losses) at least annually. Procedures for doing this have been well-established for many years, but recent developments in the treatment of conduct risk (the risk of loss due to the relat…
A surrogate loss function for optimization of score in binary classification with imbalanced datacs.LG
Study proposes a differentiable surrogate loss function for optimizing score in binary classification with imbalanced data.
problem Non-differentiability of score makes it unsuitable for optimization by gradient-based learning.
method Investigated relationship between score and loss functions, proposed a differentiable surrogate loss function.
result Gradient paths of the proposed surrogate loss function approximate the gradient paths of the score.
The AdaBoost algorithm was designed to combine many "weak" hypotheses that perform slightly better than random guessing into a "strong" hypothesis that has very low error. We study the rate at which AdaBoost iteratively converges to the minimum of the "exponential loss." Unlike previous work, our proofs do not require …
New insights on pruning deep networks by preserving function locality.
problem Designing effective pruning methods for deep neural networks.
method Revisited loss modeling using first and second order Taylor expansions, emphasizing locality.
result Both first and second order Taylor expansions can achieve similar performance in pruning.
This paper explains how low-precision arithmetic causes loss spikes in deep learning models.
problem Loss spikes during long-term training of deep neural networks.
method Analyzes the impact of floating-point precision limits on gradient updates and feature means.
result Numerical Feature Inflation (NFI) explains loss spikes and rapid parameter norm growth.
LoRA-Curve connects independent LoRA optima through continuous low-loss valleys, improving Bayesian model averaging.
problem Challenges in estimating epistemic uncertainty in LoRA-based Bayesian inference.
method Introduces LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space, with free and anchored configurations.
result Empirically shows that connecting independent LoRA optima through continuous low-loss valleys improves mutual information of the predictive distribution.