Performance of investment managers are evaluated in comparison with benchmarks, such as financial indices. Due to the operational constraint that most professional databases do not track the change of constitution of benchmark portfolios, standard tests of performance suffer from the "look-ahead benchmark bias," when t…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Benchmark Bias”
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Financecs.AI
Look-Ahead-Bench evaluates financial LLMs for lookahead bias, revealing significant differences in model performance.
problem Measuring and mitigating lookahead bias in financial LLMs.
method Standardized benchmark evaluating model behavior in practical financial scenarios, analyzing performance decay across market regimes.
result Standard LLMs exhibit significant lookahead bias, while Pitinf models show improved generalization and reasoning abilities.
Semiparametric method removes bias in functional bilevel gradient estimation.
problem First-order bias in plug-in hypergradient when lower-level problem is nonparametric.
method Semiparametric debiasing theory based on efficient influence function leads to cross-fitted orthogonal hypergradient estimator.
result Asymptotic normality and uniform control over outer parameter established for the estimator.
Novel strategy benchmarks observational studies against randomized trials.
problem Benchmarking observational studies for treatment effect bias.
method Statistical test for null hypothesis of treatment effect difference.
result Valid lower bound on maximum bias strength for any subgroup.
FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.
problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.
Study evaluates bias mitigation methods in deep learning, finds they often exploit hidden biases.
problem Deep learning systems learn biases, affecting performance on minority groups.
method Improved evaluation protocol, new dataset, robustness across different tuning distributions.
result Bias mitigation methods often exploit hidden biases, are not robust to multiple forms of bias, and are sensitive to tuning set choice.
CausalGame benchmarks LLM agents' causal thinking in games.
problem Evaluating causal thinking in AI Scientists with LLMs.
method Interactive games with 14 scenarios incorporating selection bias, measurement error, and hidden confounders.
result None of the 30 LLM agents demonstrated reliable causal thinking, with the best model achieving only 68.0% survival.
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscs.LG
Paper tackles overestimation bias in continuous control, improving performance by 25%.
problem Overestimation bias in off-policy learning.
method Truncated Quantile Critics (TQC) combines distributional representation, truncation, and ensembling of critics.
result TQC outperforms state-of-the-art methods by 25% on the Humanoid environment.
Debiasing LLMs by Fine-tuningq-fin.GN
Fine-tunes LLMs to correct bias in predictions.
problem LLMs exhibit bias in predictions from data.
method Supervised fine-tuning with Low-Rank Adaptation (LoRA).
result Fine-tuning corrects bias in both controlled and real-world settings.
Excessive reuse of holdout data can lead to overfitting. However, there is little concrete evidence of significant overfitting due to holdout reuse in popular multiclass benchmarks today. Known results show that, in the worst-case, revealing the accuracy of adaptively chosen classifiers on a data set of size al…
SSMs have a built-in bias towards low-frequency components, which can be adjusted.
problem Frequency bias in SSMs affects their performance on long-range sequences.
method Proposed two mechanisms to tune frequency bias: scaling initialization or applying a Sobolev-norm-based filter.
result Tuning frequency bias improves SSMs' performance on long-range sequence learning tasks.
Reducing Selection Bias in Counterfactual Reasoning for Individual Treatment Effects Estimationcs.LG
Reduces selection bias in estimating individual treatment effects.
problem Selection bias in counterfactual reasoning.
method Auto-encoder with regularized loss based on Pearson Correlation Coefficient.
result Improves performance in estimating individual treatment effects.
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE ex…
Mitigates spurious correlations without bias labels.
problem Spurious correlations bias model performance.
method Introduces a novel training objective and debiasing method DPR.
result DPR achieves state-of-the-art performance.
Study reveals statistical bias in dataset replication, reducing accuracy drop from 11-14% to 3.6%.
problem Statistical bias in dataset replication affects model generalization accuracy.
method Analyzed ImageNet-v2, identified and corrected for bias, and compared results.
result Correcting bias reduces accuracy drop from 11-14% to 3.6%.
Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inferencecs.AI
Benchmark assesses LLMs' causal inference skills, revealing significant limitations.
problem Lack of rigorous evaluation of LLMs' causal inference capabilities.
method CausalPitfalls benchmark with structured challenges and grading rubrics.
result Significant limitations in current LLMs' statistical causal inference.
Paper connects sampling and labeling biases in large-output spaces.
problem Efficient training in large-output spaces with label imbalance.
method Unified approach to address sampling and labeling biases.
result Different negative sampling schemes trade-off performance on dominant and rare labels.
Study detects and explains positional bias in financial LLMs.
problem Positional bias in financial decision-making using LLMs.
method Unified framework and benchmark for detecting and quantifying bias in Qwen2.5 models.
result Positional bias is pervasive, scale-sensitive, and resurfaces under nuanced prompt designs.
Introduces recency bias to improve time-series forecasting.
problem Lack of recency bias in standard Transformer attention for time-series data.
method Reweights attention scores with a smooth heavy-tailed decay to emphasize nearby observations.
result Recency-biased attention consistently improves sequential modeling and achieves competitive performance on time-series forecasting benchmarks.
New active learning method uses combinatorial coverage to improve data transfer and reduce bias.
problem Inability to transfer sampled data to new models and sampling bias issues.
method Data-centric active learning methods utilizing combinatorial coverage.
result Sampling data with coverage leads to better data transfer and competitive sampling bias.
BEMA reduces bias in EMA, leading to faster convergence and better performance.
problem Stochasticity in language model fine-tuning destabilizes training.
method Bias-Corrected Exponential Moving Average (BEMA) augmentation of EMA.
result BEMA leads to significantly improved convergence rates and final performance.
LLMs show biases in investment analysis, leading to unreliable recommendations.
problem LLMs face conflicts between pre-trained knowledge and real-time market data, leading to biases in investment analysis.
method Experimental framework to investigate emergent behaviors in LLMs, analyzing sector, size, and momentum biases.
result Distinct, model-specific biases observed, including a tendency to prefer technology stocks, large-cap stocks, and contrarian strategies.
Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods …
Undetected overfitting can occur when there are significant redundancies between training and validation data. We describe AVE, a new measure of training-validation redundancy for ligand-based classification problems that accounts for the similarity amongst inactive molecules as well as active. We investigated seven wi…
DatedGPT prevents lookahead bias in financial forecasting models.
problem Lookahead bias in large language models trained on internet-scale data.
method Time-aware pretraining with annual data cutoffs and instruction fine-tuning.
result Models' knowledge is effectively bounded by their data cutoff year, improving forecasting validity.
The paper tackles biases in session-based recommender systems by modeling user interest as a stochastic process.
problem Data uncertainty, popularity bias, and exposure bias in session-based recommender systems.
method The paper proposes treating user interest as a stochastic process in the latent space, debiasing item embeddings, modeling dense user interest, and introducing fake targets to simulate extended exposure.
result The proposed approach mitigates challenges in session-based recommender systems, as shown by computational experiments on various datasets.
Machine Learning (ML) is increasingly applied in real-life scenarios, raising concerns about bias in automatic decision making. We focus on bias as a notion of opinion exclusion, that stems from the direct application of traditional ML pipelines to infer subjective properties. We argue that such ML systems should be ev…
Unified framework suppresses model bias in semi-supervised learning with decoupled sampling control.
problem Class imbalance in semi-supervised learning, especially with distributional mismatches.
method Unified framework SC-SSL with decoupled sampling control, explicit expansion capability, and adaptive sampling probabilities.
result Consistent and state-of-the-art performance across various benchmark datasets and distribution settings.
GeMA learns latent manifolds to benchmark complex systems.
problem Benchmarking complex systems like rail networks and economies with classical methods.
method Geometric Manifold Analysis (GeMA) using a productivity-manifold variational autoencoder (ProMan-VAE).
result GeMA provides more nuanced efficiency evaluations in complex systems.
Model predicts political ideology using context vectors to mitigate bias and scarcity.
problem Scarcity and selection bias in political ideology prediction.
method Proposes a statistical model decomposing embeddings into context and position vectors, training an end-to-end model for deployment.
result Model can predict ideological labels even with minimal biased data, outperforming state-of-the-art methods.
Improves transferability of representations from source to target domains with weights and invariant representations.
problem Label shift between source and target domains in unsupervised domain adaptation.
method Integrates weights and invariant representations to bound the target risk, highlighting the role of inductive bias.
result Empirical evidence shows that weak inductive bias makes adaptation more robust.
Exact Generalisation Error Exposes Benchmarks Skew Graph Neural Networks Success (or Failure)stat.ML
New method reveals why GNNs perform well on certain datasets.
problem Understanding why GNNs perform differently on similar datasets.
method Deriving exact generalization error for various GNN architectures.
result Benchmark datasets favor architectures that rely on graph structure.
The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cs.CL
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is un…
BIG-bench benchmarks language models, revealing their strengths and weaknesses.
problem Understanding and quantifying the capabilities of large language models.
method Developed the BIG-bench benchmark with 204 tasks from various domains, evaluated across multiple model sizes and types.
result Model performance improves with scale but remains poor, with breakthrough behaviors often involving multiple steps.
Study shows different trajectory prediction models generalize better under OoD conditions.
problem Comparing trajectory prediction models' robustness across different datasets.
method Training models on Argoverse 2 and testing on Waymo Open Motion, and vice versa, with various augmentation strategies.
result Smallest model with highest inductive bias performs best in OoD generalization.
A new algorithm reduces bias and variance in distributionally robust optimization.
problem Distributionally robust optimization with bias and variance issues.
method Prospect, a stochastic gradient-based algorithm that reduces hyperparameter tuning.
result Prospect achieves linear convergence and 2-3x faster convergence on various benchmarks.
Study characterizes harmful low-fidelity data sources for surrogate models.
problem Identifying which low-fidelity data sources to use in constructing surrogate models.
method Employed benchmark filtering techniques to assess harmful sources using limited data.
result Provided guidelines for using low-fidelity sources in an industrial setting.
QUACKIE creates a new benchmark for NLP interpretability.
problem Evaluating NLP interpretability methods is challenging due to biased ground truths.
method Formulated a custom classification task from question-answering datasets, generating unbiased ground truths.
result Demonstrated the effectiveness of current interpretability methods on the new benchmark.
FR-Train improves fair and robust AI training by detecting and reducing poisoned data.
problem Training AI models that are fair and robust in the presence of data bias and poisoning.
method Mutual information-based adversarial training with an additional discriminator.
result FR-Train maintains fairness and accuracy even in the presence of poisoned data.
The paper tackles sampling bias in credit scoring models and proposes methods to improve their training and evaluation.
problem Sampling bias in credit scoring models leads to an incomplete representation of the borrower population.
method Bias-aware self-learning framework and Bayesian evaluation method to correct for bias.
result Bayesian evaluation outperforms standard accuracy measures in predicting future performance.
DML-IV improves IV regression for learning decision policies by reducing bias.
problem Spurious correlations in offline datasets caused by hidden confounders.
method Double/debiased machine learning (DML) framework to reduce bias in two-stage IV regression.
result DML-IV outperforms state-of-the-art methods and learns high-performing policies.
Local Clustering improves semi-supervised learning models.
problem Improving semi-supervised learning models with limited labeled data.
method Local Clustering (LC) method to mitigate confirmation bias in Mean Teacher (MT) model.
result Adding LC loss to MT improves model performance on semi-supervised benchmark datasets.
FairVIC improves fairness in neural networks without sacrificing accuracy.
problem Mitigating bias in automated decision-making systems, particularly in deep learning models.
method Integrates variance, invariance, and covariance terms into the loss function during training to abstract fairness concepts.
result Significant improvements in fairness across all tested metrics without compromising accuracy.
DRAGON improves learning for rare classes in unbalanced datasets using class descriptions.
problem Learning rare classes in unbalanced datasets with deep models.
method DRAGON is a late-fusion architecture that corrects bias towards frequent classes and fuses class-descriptions to improve tail-class accuracy.
result DRAGON outperforms state-of-the-art models on new benchmarks for long-tail learning with class descriptors.
Local nonparametric meta-learning improves meta-generalization across tasks.
problem Meta-learning struggles with global inductive biases and out-of-distribution tasks.
method Proposes a local, nonparametric meta-learning algorithm using meta-trained local learning rules.
result Improved meta-generalization and state-of-the-art results in robotics benchmarks.
New algorithm reduces bias in trained models, near-optimal performance proven.
problem Reduction of bias in trained machine learning models.
method Scalable post-processing algorithm for debiasing trained models, including deep neural networks (DNNs).
result Proven to be near-optimal by bounding its excess Bayes risk.