Study shows label errors impact model disparity metrics, proposing mitigation methods.
problem Impact of label errors on model disparity metrics.
method Empirical study, characterizing label error effects; proposing estimation and relabeling methods.
result Label errors significantly affect model disparity metrics, particularly for minority groups.
Metric learning enhances combinatorial coverage metrics' ability to predict classification errors.
problem Dataset dependence of combinatorial coverage metrics in anticipating classification errors.
method Metric learning to improve latent space separation of data classes.
result Metric learning increases SDCCMs' ability to distinguish between correctly and incorrectly classified data.
Paper proposes a robust metric learning algorithm.
problem Robustness of metric learning against adversarial perturbations is insufficient.
method Proposes a novel Mahalanobis distance metric learning algorithm.
result Certifiable robustness improvement over Euclidean distance.
Unified NICEk metrics improve solar forecasting accuracy.
problem Lack of suitable error metrics for multidimensional solar irradiance forecasting.
method Introducing NICEk framework with Lk norms for evaluating forecasting models.
result NICESigma consistently outperforms traditional metrics in discriminative power and statistical significance.
Corrects errors in previous studies on metric measure spaces.
problem Identifies and corrects errors in previous research on metric measure spaces.
method Corrigendum to previous publications in Acta Math, JFA.
result Corrects errors in previous studies on metric measure spaces.
A new metric optimizes forecasts for lumpy, intermittent demand.
problem Inaccurate demand forecasts lead to suboptimal logistics and production.
method Developed a novel metric that considers both statistical and business aspects.
result The new metric yields more accurate predictions for lumpy and intermittent demand.
New decision-theoretic calibration error metric improves prediction reliability.
problem Improving the reliability of predictions for decision-making.
method Proposed Calibration Decision Loss (CDL) and an efficient algorithm to achieve near-optimal CDL.
result Near-optimal CDL guarantees vanishing payoff loss from miscalibration.
Paper proves polynomial equivalence of quantum complexity metrics.
problem Quantum complexity metrics equivalence.
method Study of right-invariant metrics on unitary group.
result All metrics in the equivalence class have polynomial slowdown in approximation.
The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been…
Performance metrics (error measures) are vital components of the evaluation frameworks in various fields. The intention of this study was to overview of a variety of performance metrics and approaches to their classification. The main goal of the study was to develop a typology that will help to improve our knowledge a…
A visualization aids in comparing regression models by highlighting errors and correlations.
problem Comparing regression models is difficult due to varying hyper-parameters and metrics.
method Introduces a novel visualization approach using 2D residual space, Mahalanobis distance, and colormaps.
result Enhanced understanding of regression model performance differences and error distributions.
This paper tightens the generalization error bound for graph embedding in non-Euclidean spaces.
problem High generalization error in non-Euclidean graph embedding, preventing practical applications.
method Novel upper bound of graph embedding's generalization error using local Rademacher complexity.
result The new bound is tighter and faster, allowing better performance in non-Euclidean spaces.
Paper extends Bayesian Cramér-Rao bound with geometric considerations.
problem Estimation of covariance matrices with geometric structures.
method Intrinsic Bayesian Cramér-Rao bound with Riemannian geometry.
result Performance bounds for covariance matrix estimation.
TCE measures calibration error with a test-based approach.
problem Measuring calibration error of probabilistic binary classifiers.
method TCE uses a novel loss function based on a statistical test.
result TCE offers clear interpretation, consistent scale, and enhanced visual representation.
Synthetic data augmentation can improve imbalanced classification metrics.
problem Improving imbalanced classification metrics
method Developing a framework for analyzing the effects of synthetic data augmentation on score-based classification
result Augmentation can improve AUROC, AUPRC, balanced accuracy, and F1 score
New metrics improve scRNA-seq perturbation modeling by reducing mode collapse.
problem Outperformed by simple mean prediction in scRNA-seq perturbation modeling.
method Introduce DEG-aware metrics (WMSE, Rw2(Δ)) and negative/positive baselines. result WMSE loss function reduces mode collapse and improves model performance.
We develop coreset techniques for noisy clustering with provable guarantees.
problem Clustering with stochastic noise in datasets.
method Surrogate error metrics and coreset construction algorithm.
result Improved coreset size and better guarantees on true clustering cost.
This paper studies clustering of data sequences using the k-medoids algorithm. All the data sequences are assumed to be generated from \emph{unknown} continuous distributions, which form clusters with each cluster containing a composite set of closely located distributions (based on a certain distance metric between di…
Lagrangian data assimilation is a complex problem in oceanic and atmospheric modeling. Tracking drifters in large-scale geophysical flows can involve uncertainty in drifter location, complex inertial effects, and other factors which make comparing them to simulated Lagrangian trajectories from numerical models extremel…
Random forests are among the most popular classification and regression methods used in industrial applications. To be effective, the parameters of random forests must be carefully tuned. This is usually done by choosing values that minimize the prediction error on a held out dataset. We argue that error reduction is o…
Non-volatile memory, such as resistive RAM (RRAM), is an emerging energy-efficient storage, especially for low-power machine learning models on the edge. It is reported, however, that the bit error rate of RRAMs can be up to 3.3% in the ultra low-power setting, which might be crucial for many use cases. Binary neural n…
The study categorizes reward errors in reinforcement learning, finding some can be beneficial.
problem Training language models with imperfect proxy rewards.
method Theoretical analysis of policy gradient optimization and categorization of reward errors.
result Reward errors can be benign or even beneficial, preventing policy from stalling.
Ordinal regression is aimed at predicting an ordinal class label. In this paper, we consider its semi-supervised formulation, in which we have unlabeled data along with ordinal-labeled data to train an ordinal regressor. There are several metrics to evaluate the performance of ordinal regression, such as the mean absol…
In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking,…
Reference metrics are used to define the differential structure on multicube representations of manifolds, i.e., they provide a simple and practical way to define what it means globally for tensor fields and their derivatives to be continuous. This paper introduces a general procedure for constructing reference metrics…
This paper corrects errors in UMAP's derivation and explains its properties.
problem Errors in UMAP's derivation by McInnes et al.
method Full derivation of Spivak's functors and McInnes et al.'s finite variant.
result Corrected errors and provided an explicit description of the metric realization.
This study analyzes how well GANs approximate distributions from small samples.
problem Understanding how well GANs approximate distributions from limited data.
method Analysis of GANs using integral probability metrics and Hölder classes.
result GANs can adaptively learn low-dimensional structures or Hölder densities.
Algorithms learn and test variable partitions in various groups and error metrics.
problem Learning and testing variable partitions in different groups and error metrics.
method Algorithms for agnostically learning and testing k-partitionability over various groups and error metrics. result Learning algorithms for k-partitionability with polynomial time complexity and testing with adaptive queries. K-Medoids(KM) is a standard clustering method, used extensively on semi-metric data.Error analyses of KM have traditionally used an in-sample notion of error,which can be far from the true error and suffer from generalization gap. We formalize the true K-Medoid error based on the underlying data distribution.We decompo…
New metrics boost A/B-test power by up to 210%.
problem High cost and type-II errors in A/B-tests.
method Learn metrics from short-term signals to maximize power.
result Statistical power increased by up to 210%.
Metrics assess uncertainty structure and distribution for regression models.
problem Quantifying uncertainty in high-dimensional and nonlinear regression tasks.
method Two bounded comparison metrics for uncertainty structure and distribution.
result DNNs and DNOs provide encouraging uncertainty metric values in high dimensions.
Establishes upper bounds on generalization error in active learning.
problem Improving query algorithms in active learning.
method Derives upper bounds on generalization error using informativeness and representativeness query strategies.
result Validates the use of regularization techniques to ensure bounds' validity.
Reassesses calibration metrics in machine learning models.
problem Inconsistent reporting of calibration metrics in recent literature.
method Calibration-based decomposition of Bregman divergences, visualization of calibration and generalization error.
result New visualization technique for detecting trade-offs between calibration and generalization.
A new metric predicts model performance on unseen data.
problem Predicting performance on out-of-distribution data without labels.
method Uses model predictions to pseudo-label data, trains a new model, and measures difference from in-distribution models.
result Empirically outperforms existing methods on image and text classification tasks.
Paper introduces a new framework to improve sample efficiency in POMDPs learning.
problem Challenges in off-policy evaluation for POMDPs, especially with hidden states.
method Exploits the metric structure of belief space to relax coverage assumptions.
result Unified analysis technique yields tighter error bounds and sample efficiency improvements.
Normalizing Flows improve prediction interval efficiency in CP.
problem Inefficient prediction intervals in CP due to non-uniform error distribution.
method Train a Normalizing Flow to optimize the distance metric between errors and inputs.
result Optimized prediction intervals are more efficient and valid.
New metric reduces estimation error in survival model evaluation.
problem Dependent censoring complicates survival model evaluation.
method Dependent Brier score based on Archimedean copula and Copula-Graphic estimator.
result Reduces estimation error by 12-16% on average.
Paper proposes a new metric to evaluate survival models, especially for censored data.
problem Challenges in evaluating survival prediction models due to censored data.
method Developed a novel approach to estimate Mean Absolute Error (MAE) for survival datasets with censored data.
result The proposed MAE metric using pseudo-observations accurately ranks model performance and closely matches true MAE.
Improved robustness in multivariate regression and classification with DRO under Wasserstein metric.
problem Outliers in covariates and responses.
method Distributionally Robust Optimization (DRO) with Wasserstein metric ambiguity set and regularization.
result Significant improvement in predictive error and robustness.
New pruning method breaks power law scaling, potentially reducing error to exponential.
problem Improving neural network performance through scaling alone is costly.
method Developed a new data pruning metric to break power law scaling.
result Pruned datasets show better than power law scaling on various image datasets.
We show that on every compact spin manifold admitting a Riemannian metric of positive scalar curvature Friedrich's eigenvalue estimate for the Dirac operator can be made sharp up to an arbitrarily small given error by choosing the metric suitably.
We study the Lipschitz metric on Teichmuller space (defined by Thurston) and compare it with the Teichmuller metric. We show that in the thin part of Teichmuller space the Lipschitz metric is approximated up to bounded additive distortion by the sup metric on a product of lower-dimensional spaces (similar to the Teichm…
New error bounds for GANs with nonlinear objective functions derived.
problem Statistical consistency of GANs with nonlinear objective functions.
method Derivation of statistical error bounds for (f,Γ)-GANs using Rademacher complexity. result Proves the statistical consistency of (f,Γ)-GANs. New method for better initial centers in clustering with improved accuracy and privacy.
problem Improving the quality of clustering centers in metric spaces.
method HST initialization based on metric embedding tree structure, combined with efficient search algorithm and DP extension.
result HST initialization produces better initial centers than k-median++ with comparable efficiency and improved privacy. Paper explores generalization of minimax learners, proposing a new metric.
problem Understanding how minimax learners perform on unseen data.
method Proposes a new metric, the primal gap, to study generalization of minimax learners.
result Derives generalization error bounds for the primal gap in nonconvex-concave settings.
Study complex-valued VAEs for radar OOD detection.
problem Detecting out-of-distribution signals in complex radar environments.
method Proposed and compared several detection metrics for CVAE.
result CVAE-MSE and latent-based scores outperform ANMF-Tyler.
New research shows calibration error is flawed when dealing with model uncertainty.
problem Current model evaluation techniques conflate model uncertainty with aleatoric uncertainty.
method Posterior predictive checks to evaluate deep learning models.
result Calibration error and variants are incorrect when model uncertainty is present.
The paper analyzes the generalization of deep neural networks for metric and similarity learning.
problem Lack of rigorous understanding of generalization performance in metric and similarity learning.
method Derive explicit form of true metric, construct structured deep ReLU neural network, establish excess risk bounds.
result Explicit excess risk bounds for metric and similarity learning are derived.