Study reveals issues with neural autoregressive models and proposes mode recovery cost.
problem Unreasonable affinity of neural autoregressive models to short and long sequences.
method Investigates modes of ground-truth, empirical, and decoding-induced distributions via mode recovery cost.
result Mode recovery cost varies depending on ground-truth distribution and impacts decoding-induced distribution.
SAMPLR optimizes for ground truth in aleatoric parameters to avoid curriculum-induced covariate shift.
problem Curriculum learning shifts training distribution, leading to suboptimal policies in aleatoric settings.
method SAMPLR optimizes ground-truth utility function, avoiding curriculum-induced covariate shift.
result SAMPLR preserves optimality under ground-truth distribution, promoting robustness across various environments.
New method falsifies causal discovery results without ground truth.
problem Evaluation of causal discovery algorithms without ground truth data.
method Detects incompatibilities between causal graphs learned on different subsets of variables.
result Detection of incompatibilities can falsify wrongly inferred causal relations.
PoGDiff improves diffusion models on imbalanced datasets.
problem Diffusion models struggle with imbalanced text-image pairs.
method PoGDiff replaces ground-truth with a Product of Gaussians.
result PoGDiff enhances generation accuracy and quality on imbalanced datasets.
We analyze a new robust method for the reconstruction of probability distributions of observed data in the presence of output outliers. It is based on a so-called gradient conjugate prior (GCP) network which outputs the parameters of a prior. By rigorously studying the dynamics of the GCP learning process, we derive an…
Paper introduces metrics to evaluate missing data imputation without ground truth.
problem Handling missing data in time series without ground truth.
method Introduces Wasserstein distance (WD) and Jensen-Shannon divergence (JSD) as metrics to evaluate imputation quality.
result WD and JSD are effective metrics for assessing missing data imputation quality.
A new score measures data reliability without ground truth.
problem Assessing reliability of datasets without access to ground truth.
method Define ground-truth-based orderings and propose Gram determinant score.
result Gram determinant score effectively captures data quality across diverse observation processes.
Improved DLG extracts accurate labels from gradients, overcoming DLG's convergence issues.
problem Private training data leakage from shared gradients in distributed learning systems.
method Proposes iDLG, a simple approach to synthesize accurate labels from gradients.
result iDLG reliably extracts ground-truth labels from gradients, unlike DLG.
We give a new algorithm for learning a two-layer neural network under a general class of input distributions. Assuming there is a ground-truth two-layer network y=Aσ(Wx)+ξ, where A,W are weight matrices, ξ represents noise, and the number of neurons in the hidden layer is no larger than the input or outp…
This paper analyzes the difficulty of unsupervised domain adaptation using information theory.
problem The challenge of unsupervised domain adaptation under covariate shift.
method Formulates the problem using a distribution π in the ground-truth triples (p, q, f), defines optimal learner performance, and introduces PTLU for quantifying difficulty.
result Characterizes the optimal learner and introduces PTLU as a measure of UDA difficulty.
NM-VQTSG improves synthetic time series fidelity by aligning distributions.
problem Fidelity challenges in VQ-based time series generation.
method Neural mapping model using U-Net to refine synthetic data.
result Significant improvements in FID, IS, and conditional FID metrics.
New method accounts for uncertainty in medical AI evaluations.
problem Uncertainty in ground truth affects AI model performance estimates.
method Statistical aggregation approach to infer probabilities of medical conditions.
result Performance estimates are significantly lower when uncertainty is accounted for.
It has been noticed that some external CVIs exhibit a preferential bias towards a larger or smaller number of clusters which is monotonic (directly or inversely) in the number of clusters in candidate partitions. This type of bias is caused by the functional form of the CVI model. For example, the popular Rand index (R…
Unsupervised recalibration (URC) is a general way to improve the accuracy of an already trained probabilistic classification or regression model upon encountering new data while deployed in the field. URC does not require any ground truth associated with the new field data. URC merely observes the model's predictions a…
Proposes hinge-Wasserstein to improve uncertainty estimation in regression tasks.
problem Estimating multimodal aleatoric uncertainty in regression tasks from images.
method Regression-by-classification paradigm with hinge-Wasserstein loss.
result Hinge-Wasserstein loss improves uncertainty estimation on challenging tasks.
TOO optimizes stochastic epidemiological models by finding both parameter settings and random seeds.
problem Calibrating stochastic epidemiological models to match empirical observations.
method Gaussian process surrogates and Thompson sampling for optimization.
result Produces actual trajectories consistent with ground truth.
Bayesian meta learning improves uncertainty quantification in regression.
problem Trusting uncertainty quantification in Bayesian regression.
method Trust-Bayes framework for Bayesian meta learning, optimizing for trustworthy uncertainty quantification.
result Lower bounds and sample complexity for trustworthy uncertainty quantification are characterized.
The paper analyzes matrix completion with unlabeled implicit feedback and provides error bounds.
problem Matrix completion with shared low-rank ground truth and sampling distribution.
method Combining subspace recovery theory and matrix completion bounds.
result Error bounds showing contributions from estimating the sampling distribution and ground truth.
Pairwise Label Smoothing improves deep model generalization by reducing overconfidence.
problem Improving deep model generalization through regularization.
method PLS smooths labels for pairs of samples, learning distribution mass during training.
result PLS significantly outperforms LS and baseline models, reducing up to 30% classification error.
Universally valid ground truth is almost impossible to obtain or would come at a very high cost. For supervised learning without universally valid ground truth, a recommended approach is applying crowdsourcing: Gathering a large data set annotated by multiple individuals of varying possibly expertise levels and inferri…
Paper proposes a method to recover accurate labels from partially valid data in multi-label learning.
problem Tackles noisy supervision in multi-label learning with partially valid labels.
method Develops a two-stage method that estimates label enrichment and ground-truth confidences.
result Demonstrates improved performance over state-of-the-art PML methods.
Tests for classifier independence without ground truth labels.
problem Validation of classifier independence without ground truth labels.
method Exact solution for independent binary classifiers using algebraic geometry.
result Self-consistent test for classifier independence without ground truth labels.
Study identifies regions where scoring rules reliably detect forecast errors.
problem Insufficient reliability of scoring rules in evaluating multivariate probabilistic forecasts.
method Systematic finite-sample analysis of proper scoring rules on synthetic and real-world data.
result Identified regions of reliability for scoring rules in time-series forecasting.
Agent-based model compares different COVID-19 testing policies and their effectiveness.
problem Understanding how different testing policies reveal the true number of infected cases.
method Developed an agent-based simulation framework in Python to model various testing policies and interventions.
result Contact Tracing consistently captures more positive cases than Random Symptomatic Testing, and LBT performs similarly.
RealCause provides a realistic benchmark for causal inference.
problem Lack of a reliable benchmark for comparing causal effect estimators.
method Flexible generative models to create a benchmark that is both ground-truth and realistic.
result Evaluation of over 1500 causal estimators provides evidence for choosing hyperparameters using predictive metrics.
The Huber loss is a robust loss function used for a wide range of regression tasks. To utilize the Huber loss, a parameter that controls the transitions from a quadratic function to an absolute value function needs to be selected. We believe the standard probabilistic interpretation that relates the Huber loss to the H…
Generates synthetic manufacturing data for causal discovery benchmarking.
problem Lack of suitable real data for validating causal discovery algorithms.
method Distributional random forests for estimating conditional distributions.
result Semisynthetic manufacturing data adheres to a causal model.
A new method for estimating uncertainty intervals in regression.
problem Lack of effective methods to estimate uncertainty intervals in regression.
method Collaborating Networks (CN) approach using two neural networks with distinct loss functions.
result CN method improves performance on various real-world datasets, including forecasting A1c values in diabetic patients.
Normalizing flows model atomic solids without needing ground-truth samples.
problem Modeling atomic solids without ground-truth samples.
method Normalizing flows to transform a base distribution into the target solid.
result Excellent agreement between model estimates and literature values for Helmholtz free energy.
Recently developed deep-learning-based denoisers often outperform state-of-the-art conventional denoisers such as the BM3D. They are typically trained to minimize the mean squared error (MSE) between the output image of a deep neural network (DNN) and a ground truth image. Thus, it is important for deep-learning-based …
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
Estimates classifier errors without ground truth using algebraic geometry.
problem Lack of ground truth in real-world production systems.
method Non-parametric estimation using algebraic geometry to solve the self-assessment problem.
result Accuracy estimators are better than one part in a hundred.
Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…
Develops a goodness-of-fit test for self-exciting processes.
problem Quantifying how well generative models capture self-exciting point processes.
method Connects to Quasi-maximum-likelihood estimator (QMLE) theory and develops a non-parametric self-normalizing statistic, the Generalized Score (GS) statistics.
result Validates the proposed GS test's good performance through numerical simulation and real-data experiments.
New causal distances improve evaluation of causal discovery algorithms.
problem Evaluating causal discovery algorithms using graphical distances is limited.
method Defined causal distances based on causal distributions rather than graphical structure.
result Improved evaluation of causal discovery algorithms on synthetic and real-world datasets.
This paper is the first work to propose a network to predict a structured uncertainty distribution for a synthesized image. Previous approaches have been mostly limited to predicting diagonal covariance matrices. Our novel model learns to predict a full Gaussian covariance matrix for each reconstruction, which permits …
QUACKIE creates a new benchmark for NLP interpretability.
problem Evaluating NLP interpretability methods is challenging due to biased ground truths.
method Formulated a custom classification task from question-answering datasets, generating unbiased ground truths.
result Demonstrated the effectiveness of current interpretability methods on the new benchmark.
Stochastic gradient descent converges to universal limits in high dimensions.
problem Statistical tasks in high dimensions with specific data projections.
method Stochastic gradient descent applied to mixture distributions, proving universality of limits.
result The ODE limits are universal for mixtures of arbitrary product distributions.
Unsupervised clustering can reproduce categorization systems if features and metrics are correctly selected.
problem Reproducing expert-provided categorization systems using unsupervised clustering.
method Investigated using toy datasets and real-world fund categorization. Used appropriate feature selection and a supervised Random Forest-based distance metric.
result Unsupervised clustering can reproduce ground truth classes if features and metrics are correctly selected.
This paper proposes a set of criteria to evaluate the objectiveness of explanation methods of neural networks, which is crucial for the development of explainable AI, but it also presents significant challenges. The core challenge is that people usually cannot obtain ground-truth explanations of the neural network. To …
Paper proposes using unlabeled data for fair decision-making.
problem Bias in decision-making algorithms due to biased labels and selective labeling.
method Variational autoencoder for learning unbiased data representations from both labeled and unlabeled data.
result Method learns fair and stable decision policies with high utility.
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Autonomy and adaptation of machines requires that they be able to measure their own errors. We consider the advantages and limitations of such an approach when a machine has to measure the error in a regression task. How can a machine measure the error of regression sub-components when it does not have the ground truth…
Self-supervised methods learn from noisy data alone, useful for imaging problems.
problem Inferring signals from noisy and incomplete observations.
method Learning a solver from measurement data alone, without ground-truth references.
result Self-supervised methods can learn meaningful estimates from noisy data.
Preconditioned SGD accelerates convergence for ill-conditioned huge-scale matrix completion.
problem Recovering a low-rank matrix from incomplete data with high condition number.
method Preconditioned Stochastic Gradient Descent (SGD) for huge-scale online optimization.
result Preconditioned SGD converges to ε-accuracy in O(log(1/ε)) iterations, compared to O(κlog(1/ε)) for unpreconditioned SGD.
New method improves model calibration by adjusting confidence based on prediction correctness.
problem Improving model confidence alignment with true class probabilities.
method Post-hoc calibration objective using transformed samples for training.
result Competitive calibration performance on in-distribution and out-of-distribution test sets.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
New method estimates model performance bounds without ground truth labels.
problem Evaluation of weakly supervised models without direct access to ground truth labels.
method Formulates model evaluation as a partial identification problem and uses Fréchet bounds for performance estimation.
result Derives accurate and computationally efficient bounds for key metrics like accuracy, precision, recall, and F1-score.