New method evaluates language model forecasters by checking consistency of predictions.
problem Evaluating the performance of language model forecasters is difficult due to lack of ground truth.
method Developed a consistency check framework based on arbitrage to evaluate forecasters.
result Consistency metrics correlate with ground truth performance of LLM forecasters.
Measures consistency of tabular LLM predictions under fine-tuning multiplicity.
problem Conflicting predictions from fine-tuned tabular LLMs.
method Local stability measure in embedding space.
result Probabilistic guarantees on prediction consistency under multiplicity.
New framework improves generative models with prediction and consistency constraints.
problem Improving generative models with sparse labeled data.
method Optimizes variational autoencoders with prediction and consistency constraints.
result Promising image classification performance, especially in semi-supervised scenarios.
The nearest neighbor rule is proven consistent in a broad setting.
problem Proving consistency of the nearest neighbor rule in various settings.
method Proving online consistency for all measurable functions in doubling metric spaces under mild assumptions.
result The nearest neighbor rule is online consistent in all measurable functions in doubling metric spaces.
Separable losses are inconsistent for structured prediction models.
problem Inconsistency of separable losses in structured prediction models.
method Analysis of separable negative log-likelihood losses for structured prediction.
result Separable losses are not Bayes consistent and may not predict the most probable structure.
Max-min margin Markov networks improve consistency in structured prediction.
problem Statistical inconsistency in max-margin methods for structured prediction.
method Defining a max-min margin formulation to overcome statistical inconsistency.
result Proves consistency and provides an explicit algorithm with finite sample generalization bounds.
Self-consistent models improve reinforcement learning by aligning predictions with future values.
problem Improving reinforcement learning by aligning model predictions with future values.
method Proposes multiple self-consistency updates to encourage a learned model and value function to be consistent with each other.
result Self-consistency helps both policy evaluation and control in both tabular and function approximation settings.
Benchmark assesses fairness in algorithmic uncertainty, revealing consistent and calibrated estimates improve fairness.
problem Challenges in managing uncertainty in fairness evaluations for predictive algorithms.
method Introduces FairlyUncertain, an axiomatic benchmark for evaluating uncertainty in fairness.
result Consistent and calibrated uncertainty estimates improve fairness without explicit fairness interventions.
New method learns output embeddings for structured prediction.
problem Structured prediction with output embeddings.
method Jointly learns output embedding and regression function.
result Structured predictor is a consistent estimator with smaller complexity.
More and more AI services are provided through APIs on cloud where predictive models are hidden behind APIs. To build trust with users and reduce potential application risk, it is important to interpret how such predictive models hidden behind APIs make their decisions. The biggest challenge of interpreting such predic…
New bin-wise scaling methods improve prediction uncertainty calibration for machine learning.
problem Improving prediction uncertainty calibration for machine learning regression.
method Adaptations of Binwise Variance Scaling (BVS) with alternative loss functions and feature-based binning.
result Improved adaptivity and consistency in prediction uncertainty calibration.
Neural networks have become very popular in surrogate modeling because of their ability to characterize arbitrary, high dimensional functions in a data driven fashion. This paper advocates for the training of surrogates that are consistent with the physical manifold -- i.e., predictions are always physically meaningful…
ε-Consistent Mixup improves semi-supervised classification accuracy.
problem Improving semi-supervised classification accuracy with limited labeled data.
method Combines Mixup's linear interpolation with consistency regularization, using an adaptive tradeoff between the two.
result ε-Consistent Mixup yields the largest gains in low label-availability scenarios.
The paper explores conditions for predicting optimization performance.
problem Lack of formal theoretical guarantees linking prediction and optimization performance.
method Exploring conditions for asymptotic convergence and exact quantification of optimization performance.
result Explicit theoretical relationship between prediction and optimization performance.
Study on top-k classification with new loss functions and algorithms.
problem Improving multi-class classification accuracy and cardinality trade-off.
method Introducing cardinality-aware loss functions and deriving their consistency bounds.
result New cardinality-aware algorithms for top-k classification. Improves GAN-based semi-supervised learning with consistency regularization.
problem Lack of consistency in class probability predictions under local perturbations.
method Introduces consistency regularization to GANs, leveraging both local and interpolation consistency.
result Significantly improves performance and achieves new state-of-the-art results.
This paper examines how different decoding algorithms for LLMs align with various goals.
problem Consistency of decoding algorithms with different goals in LLMs.
method Analysis of greedy, lookahead, random sampling, and temperature-scaled random sampling algorithms.
result Random sampling is consistent with the true probability distribution, but other goals require optimal algorithms for specific probability distributions.
In many real-world prediction tasks, class labels include information about the relative ordering between labels, which is not captured by commonly-used loss functions such as multi-category cross-entropy. Recently, the deep learning community adopted ordinal regression frameworks to take such ordering information into…
Improved incremental sequence classification with temporal consistency.
problem Updating predictions as new sequence elements are revealed.
method Temporal-difference learning and a temporal-consistency condition for successive predictions.
result Optimizing a novel loss function improves data efficiency and predictive accuracy.
We improve prediction risk estimation for large datasets using sketching and ridge regression.
problem Estimating prediction risks for large datasets efficiently and accurately.
method Random matrix theory, generalized cross validation, sketched ridge regression ensembles, and ensemble trick.
result Consistent risk estimation and prediction intervals for large-scale datasets.
Analysis of cross-validation for early-stopped gradient descent in high-dimensional regression.
problem Inconsistency of GCV for early-stopped GD in high-dimensional least squares regression.
method Theoretical analysis of GCV and LOOCV applied to early-stopped GD in high-dimensional least squares regression.
result LOOCV converges uniformly to the prediction risk of early-stopped GD, while GCV is generically inconsistent.
CCE improves anomaly detection metrics by measuring both confidence and consistency.
problem Existing anomaly detection metrics lack discriminative power, hyperparameter dependency, and robustness to perturbations.
method CCE uses Bayesian estimation to quantify uncertainty and constructs global and event-level confidence and consistency scores.
result CCE demonstrates strict boundedness, robustness, and linear time complexity.
Chronologically consistent models maintain accuracy with time-restricted data.
problem Training data introduces lookahead bias and training leakage in large language models.
method ChronoBERT and ChronoGPT trained with only available data at each time point.
result Models achieve strong performance and competitive with larger models, mitigating lookahead bias.
Paper corrects Max-Margin loss for multi-label tasks.
problem Max-Margin loss inconsistency in multi-label classification.
method Introduced Restricted-Max-Margin loss.
result Consistent loss for multi-label tasks under milder conditions.
Proposes CRA framework for certifying fair predictive models.
problem Certifying fairness of predictive models trained on biased data.
method Formulates CRA for fairness queries, uses background knowledge and limited target population stats.
result Builds certifiably fair predictive models on target population.
Paper provides statistical guarantees for GNNs in link prediction.
problem Link prediction accuracy in graph neural networks.
method Proposes a linear GNN architecture (LG-GNN) and derives statistical guarantees.
result LG-GNN produces consistent estimators for edge probabilities and has better detection of high-probability edges.
We propose and analyze a regularization approach for structured prediction problems. We characterize a large class of loss functions that allows to naturally embed structured outputs in a linear space. We exploit this fact to design learning algorithms using a surrogate loss approach and regularization techniques. We p…
New framework predicts time series with missing values without imputation.
problem Predicting time series with missing values, especially when there's no ground truth for missing data.
method CRIB framework, combining attention mechanism and consistency regularization.
result CRIB framework predicts accurately even under high missing rates.
Develops a framework for consistent loss functions with variable transformations.
problem Lack of theoretical understanding of variable transformations in consistent loss functions.
method Formal characterizations of consistency for transformed loss functions in two cases: realization and prediction variables.
result Establishes new identifiable and elicitable functionals for complex predictive tasks.
A reject option improves partial-label learning's accuracy.
problem Ambiguously labeled data in real-world applications.
method Risk-consistent nearest-neighbor algorithm with a reject option.
result Our method provides the best trade-off between non-rejected predictions' number and accuracy.
New method learns functions without paired data using mediating variables.
problem Learning functions without paired input-output data.
method Mediated Uncoupled Learning: Predicting h(U) to approximate Y. result Statistical consistency and error bounds of the proposed method.
We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm. ICT encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points…
New methods for better uncertainty prediction in ML.
problem Insufficient calibration in machine learning regression.
method Conditional calibration with respect to input features (adaptivity).
result Consistency and adaptivity are complementary, and good consistency does not guarantee good adaptivity.
Study tightens bounds for interpolating noisy data using minimum l1-norm.
problem Predicting noisy data with minimum l1-norm interpolation.
method Provided matching upper and lower bounds for prediction error.
result Tight consistency up to negligible terms for d≫n. In many structured prediction problems, complex relationships between variables are compactly defined using graphical structures. The most prevalent graphical prediction methods---probabilistic graphical models and large margin methods---have their own distinct strengths but also possess significant drawbacks. Conditio…
New methods lift weak supervision to structured prediction, providing robustness guarantees.
problem Applying weak supervision techniques to structured prediction problems.
method Introducing pseudo-Euclidean embeddings, tensor decompositions, and invariants for consistent noise rate estimation.
result Generalization guarantees nearly identical to those for models trained on clean data.
A new autoregressive SPO method improves decision-making for dependent data.
problem Improving decision-making for dependent data in stochastic optimization.
method An autoregressive Smart Predict-then-Optimize (SPO) method for time series data.
result Generalization bounds and uniform calibration results for the SPO loss in autoregressive models.
We formalize and study the natural approach of designing convex surrogate loss functions via embeddings, for problems such as classification, ranking, or structured prediction. In this approach, one embeds each of the finitely many predictions (e.g.\ rankings) as a point in Rd, assigns the original loss val…
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems tha…
CPA framework assesses conditional validity of conformal prediction.
problem Challenges in evaluating conditional validity of conformal prediction.
method Reframes conditional coverage evaluation as a supervised learning task.
result Establishes convergence rates and proves CVI consistency.
Study finds no consistent return predictability using payout ratios across 16 countries.
problem Return predictability using payout ratios in various markets.
method Analysis of 16 developed countries' bond, equity, and housing markets using payout-price ratios.
result No consistent in-sample and out-of-sample performance with positive utility gain.
Random forests have proven to be reliable predictive algorithms in many application areas. Not much is known, however, about the statistical properties of random forests. Several authors have established conditions under which their predictions are consistent, but these results do not provide practical estimates of ran…
Diffusion models' consistency across splits explained by random matrix theory.
problem Consistency of diffusion models trained on non-overlapping subsets.
method Random matrix theory framework to quantify dataset effects on denoiser and sampling map.
result The theory explains and predicts cross-split disagreement in diffusion models.
ML weather forecasts lack physical consistency, but add value.
problem Lack of physical consistency in ML weather forecasts.
method Examined Pangu-Weather forecasts for fidelity and physical consistency.
result ML forecasts lack physical consistency and accuracy can be partly due to this.
Multi-step passenger demand forecasting is a crucial task in on-demand vehicle sharing services. However, predicting passenger demand over multiple time horizons is generally challenging due to the nonlinear and dynamic spatial-temporal dependencies. In this work, we propose to model multi-step citywide passenger deman…
New approach optimizes decisions based on uncertainty in predictions.
problem Mismatch between prediction accuracy and decision loss in sequential design.
method Directional uncertainty-guided approach to sequential experimental design.
result Directional uncertainty-based design stops earlier and performs better.
The paper proposes a method for predicting equity premium using penalized quantile regression.
problem Heteroscedasticity and heavy-tails in equity premium prediction.
method Penalized quantile regression with consistent variable selection across multiple quantiles.
result The proposed method outperforms benchmark methods and reveals interesting predictor relationships.
New metric reduces arbitrariness in fair binary classification predictions.
problem Variance in predictions leads to arbitrary decisions in fair classification.
method Developed a self-consistency metric and an abstention algorithm.
result Fair binary classification is often close to fair due to variance, not interventions.