This paper tackles unbiased loss functions for multilabel classification with missing labels.
problem Missing labels in multilabel classification tasks, especially in extreme multi-label classification (XMC).
method Derives unbiased estimators for multilabel reductions, including non-decomposable ones, and addresses increased variance with convex upper-bounds.
result Switching to unbiased estimators can alter the bias-variance trade-off and may require stronger regularization.
New loss functions improve extreme classification with missing labels.
problem Large number of infrequent labels and missing labels in XMC.
method Derive unbiased loss functions for XMC, incorporating them into existing algorithms.
result Significant improvement in extreme classification performance (up to 20%) over existing methods.
Proposes a new framework for learning from labeled and unlabeled data.
problem Learning from unlabeled and multi-label samples with arbitrary loss functions.
method Multi-complementary and unlabeled learning framework.
result Effective estimation of classification risk with optimal convergence rate.
ENSURE framework trains deep image recon algorithms without clean data.
problem Lack of clean, fully sampled ground-truth data for deep learning image reconstruction.
method Introduces ENSURE framework, a generalization of SURE and GSURE to random sampling patterns.
result ENSURE loss function is an unbiased estimate for true mean-square error.
UREs lead to overfitting in complex models, especially in complementary label learning.
problem Overfitting in weakly supervised learning with complementary labels.
method Proposed a surrogate complementary loss (SCL) framework to reduce gradient variance.
result SCL mitigates overfitting and improves URE-based methods.
Recommender systems widely use implicit feedback such as click data because of its general availability. Although the presence of clicks signals the users' preference to some extent, the lack of such clicks does not necessarily indicate a negative response from the users, as it is possible that the users were not expos…
New framework for learning with class-conditional multi-label noise.
problem Class labels corrupted with conditional probabilities for multiple labels.
method Formalized as CCMN framework, established unbiased estimators, proved consistency with multi-label loss functions, implemented partial multi-label learning method.
result Effectiveness validated on multiple datasets and metrics.
New method improves model robustness to biased data.
problem Learning unbiased models from biased datasets.
method Developed epsilon-SupInfoNCE and FairKL losses.
result Improved performance on biased datasets.
Generalized R2R handles non-Gaussian noise for deep network training.
problem Training deep networks from noisy data alone.
method Extending R2R to handle various noise distributions.
result GR2R loss is an unbiased estimator of supervised loss.
RS-PINN uses randomized smoothing to speed up high-dimensional PDE simulations without sacrificing accuracy.
problem High computational cost and bias in PINNs for high-dimensional PDEs.
method Introduces Gaussian noise for stochastic smoothing of PINNs, enabling Monte Carlo derivative approximation.
result Proposes bias correction techniques and a hybrid method to optimize the bias-variance trade-off.
In linear regression we wish to estimate the optimum linear least squares predictor for a distribution over d-dimensional input points and real-valued responses, based on a small sample. Under standard random design analysis, where the sample is drawn i.i.d. from the input distribution, the least squares solution for…
In this paper we address the problem of pool based active learning, and provide an algorithm, called UPAL, that works by minimizing the unbiased estimator of the risk of a hypothesis in a given hypothesis space. For the space of linear classifiers and the squared loss we show that UPAL is equivalent to an exponentially…
New framework for weakly supervised learning from label proportions.
problem Lack of consistent learning procedure and theoretical training criterion for LLP.
method Pose LLP as mutual contamination models (MCMs) and establish unbiased losses and generalization error bounds.
result Established novel technical results for MCMs and proposed a new experimental setting.
Paper proposes a method to estimate counterfactual outcomes without a known SCM.
problem Estimating counterfactual outcomes without a known structural causal model.
method Introduces rank preservation assumption and a novel ideal loss for unbiased learning of counterfactual outcomes.
result The proposed method is effective and unbiased, as shown by theoretical analysis and experiments.
An econometric or statistical model may undergo a marginal gain if we admit a new variable to the model, and a marginal loss if we remove an existing variable from the model. Assuming equality of opportunity among all candidate variables, we derive a valuation framework by the expected marginal gain and marginal loss i…
We present online boosting algorithms for multiclass classification with bandit feedback, where the learner only receives feedback about the correctness of its prediction. We propose an unbiased estimate of the loss using a randomized prediction, allowing the model to update its weak learners with limited information. …
This paper tackles selection bias in recommender systems by considering the neighborhood effect.
problem Selection bias in recommender systems due to filtering and user selection.
method Formalizes neighborhood effect as interference problem, introduces treatment representation, and proposes ideal loss.
result Proposed methods achieve unbiased learning when both selection bias and neighborhood effect are present.
The impact of a stress scenario of default events on the loss distribution of a credit portfolio can be assessed by determining the loss distribution conditional on these events. While it is conceptually easy to estimate loss distributions conditional on default events by means of Monte Carlo simulation, it becomes imp…
Paper compares DML methods fairly, finds older ones perform better.
problem Fair comparison of deep metric learning methods.
method Unified implementation of 12 DML algorithms under same conditions.
result Several older DML methods perform significantly better than previously believed.
Stochastic gradient descent (SGD), which dates back to the 1950s, is one of the most popular and effective approaches for performing stochastic optimization. Research on SGD resurged recently in machine learning for optimizing convex loss functions and training nonconvex deep neural networks. The theory assumes that on…
The paper improves sparse Gaussian processes by optimizing predictive loss.
problem Optimizing predictive loss in sparse Gaussian processes.
method Direct loss minimization (DLM) for log-loss and square loss, with product sampling (uPS) and biased Monte Carlo (bMC) for non-conjugate cases.
result DLM shows significant performance improvement in both log-loss and square loss cases.
Study shows the corrected Akaike criterion is inadmissible for estimating Kullback-Leibler discrepancy.
problem Inadmissibility of the corrected Akaike information criterion for estimating Kullback-Leibler discrepancy.
method Loss estimation framework to demonstrate inadmissibility and provide improved estimators.
result Improved estimators of Kullback-Leibler discrepancy are provided and perform well in reduced-rank situations.
Estimates conversion probabilities from click sequences with privacy constraints.
problem Training models in advertising with limited direct click-conversion links.
method Formalizes learning from attribution sets, constructs unbiased estimator, applies Empirical Risk Minimization.
result Empirical Risk Minimization achieves generalization guarantees and robustness against prior errors.
Proposes a simple solution to Gini importance bias in random forests.
problem Gini importance measure in random forests is biased and unreliable.
method Computes loss reduction on out-of-bag samples instead of in-bag.
result Solves the misleading/untrustworthy Gini importance issue.
We propose a new stochastic optimization framework for empirical risk minimization problems such as those that arise in machine learning. The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss. In contrast, we develop a co…
MeanFlow training is unstable due to misusing conditional velocity, leading to variance issues.
problem Unstable training of MeanFlow due to variance problems.
method Theoretical analysis and derivation of optimal coefficient in closed form.
result The optimal coefficient in MeanFlow training minimizes variance but not necessarily quality.
Paper proposes an unbiased risk estimator for PLLAC, handling unseen classes.
problem Handling unseen classes in PLLAC where some classes are not present in the training set.
method Proposes an unbiased risk estimator that estimates the distribution of augmented classes by differentiating known classes from unlabeled data.
result The estimator provides theoretical guarantees and converges to true risk minimizer as data increases.
NCV uses neural networks to improve Monte Carlo integration.
problem Improving variance reduction in parametric Monte Carlo integration.
method NCV combines a normalizing flow and a neural network to approximate the integrand and solve the integral equation, with a neural importance sampler to estimate the difference.
result NCV achieves state-of-the-art performance in light transport simulation with reduced noise and negligible bias.
PLUMAGE improves large model training efficiency and stability.
problem Accelerator memory and networking constraints during large model training.
method Probabilistic Low rank Unbiased Minimum Variance Gradient Estimator (PLUMAGE) that resolves bias and variance issues.
result PLUMAGE reduces training loss by 28% on average across the GLUE benchmark.
In contrast to the standard classification paradigm where the true class is given to each training pattern, complementary-label learning only uses training patterns each equipped with a complementary label, which only specifies one of the classes that the pattern does not belong to. The goal of this paper is to derive …
We propose a new model for supervised learning to rank. In our model, the relevance labels are assumed to follow a categorical distribution whose probabilities are constructed based on a scoring function. We optimize the training objective with respect to the multivariate categorical variables with an unbiased and low-…
Linear-Core Surrogates combine fast optimization and statistical efficiency in classification and structured prediction.
problem The trade-off between smoothness and margin-based losses in classification and structured prediction.
method Linear-Core (LC) Surrogates, a family of convex loss functions that stitch a linear core to a smooth tail.
result LC Surrogates achieve fast linear consistency rates while maintaining differentiability and strict H-consistency bounds. This paper presents a methodology and workflow that overcome the limitations of the conventional Generative Adversarial Networks (GANs) for geological facies modeling. It attempts to improve the training stability and guarantee the diversity of the generated geology through interpretable latent vectors. The resulting s…
OEUVRE estimates online loss with constant time and memory, outperforming other methods.
problem Accurately estimating expected loss in online learning.
method Recursive evaluation of each sample on current and previous models, using algorithmic stability for updates.
result Consistency, convergence rates, and concentration bounds proved for OEUVRE.
K-fold cross-validation (CV) with squared error loss is widely used for evaluating predictive models, especially when strong distributional assumptions cannot be taken. However, CV with squared error loss is not free from distributional assumptions, in particular in cases involving non-i.i.d. data. This paper analyzes …
Synthetic construction of 3D complex bases.
problem Creating a complete set of unbiased bases in 3D complex space.
method Synthetic construction using complex projective trigonometry.
result Synthetic construction of mutually unbiased bases in C^3.
Private statistics estimation faces a bias, accuracy, and privacy trilemma.
problem Balancing privacy, accuracy, and bias in statistical estimation.
method Use differential privacy (DP) for private statistics, but clip samples to control sensitivity and add noise for privacy, introducing bias.
result No algorithm can simultaneously have low bias, low error, and low privacy loss for arbitrary distributions.
Unbiased wealth exchanges always lead to inequality.
problem Understanding wealth distribution in unbiased binary exchange systems.
method Analytical demonstration of unbiased binary exchanges leading to perfect inequality.
result Any system driven by unbiased binary exchanges will reach perfect inequality and zero mobility.
VarGrad reduces variance in ELBO gradient estimation for variational inference.
problem Improving the variance of gradient estimators in variational inference.
method VarGrad uses a new log-variance loss to estimate the ELBO gradient, achieving lower variance than the score function method.
result VarGrad offers a lower variance gradient estimator compared to other methods.
The paper develops loss functions for pricing models using observational data.
problem Evaluating pricing policies directly from observational data with historical biases.
method Adapting machine learning techniques for corrupted labels to derive unbiased loss functions.
result Identifies minimum variance and robust estimators for contextual pricing.
MUSE provides unbiased stopping estimates for optimal problems.
problem Estimating the utility of optimal stopping problems.
method Backward recursive construction of the Multilevel Unbiased Stopping Estimator (MUSE).
result MUSE achieves ε-accuracy with O(1/ε^2) computational cost.
New loss function restores importance weighting in overparameterized models.
problem Restoring importance weighting in overparameterized neural networks.
method Introduced polynomially-tailed losses to restore effects of importance weighting.
result Polynomially-tailed losses improve performance in correcting distribution shift.
Randomized trials, also known as A/B tests, are used to select between two policies: a control and a treatment. Given a corresponding set of features, we can ideally learn an optimized policy P that maps the A/B test data features to action space and optimizes reward. However, although A/B testing provides an unbiased …
Estimates proper calibration errors and refinement terms in probabilistic predictions.
problem Lack of a general estimator for proper calibration errors and refinement terms with known statistical properties.
method Proposes a method for consistent, asymptotically unbiased estimation of proper calibration errors and refinement terms.
result Proves the relation between refinement and f-divergences, implying information monotonicity in neural networks.
Quantum circuit models learn better with specific initialization strategies.
problem Understanding and improving the optimization landscape of IQP-based generative models.
method Proved barren plateaus for random initialization, established lower bounds, and developed data-dependent initialization.
result Data-dependent initialization leads to faster convergence and better minimums.
New algorithm finds unbiased subnetworks in biased datasets.
problem Finding unbiased subnetworks in biased neural networks.
method Debiased Contrastive Weight Pruning (DCWP) algorithm.
result DCWP significantly outperforms state-of-the-art debiasing methods.
Proposes unbiased estimators for training mixture of experts models.
problem Efficiently training large-scale mixture of experts models on modern hardware.
method Two unbiased estimators based on principled stochastic assignment procedures.
result Both estimators are more effective and robust than biased alternatives.
The problem of estimating a high-dimensional sparse vector θ∈Rn from an observation in i.i.d. Gaussian noise is considered. The performance is measured using squared-error loss. An empirical Bayes shrinkage estimator, derived using a Bernoulli-Gaussian prior, is analyzed and compared with the…