Backward exploration reduces sample complexity in policy evaluation.
problem Empirical policy evaluation in reinforcement learning.
method Backward exploration algorithms from high-cost states.
result Reduced average-case sample complexity to .
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Empirical Evaluation”
AI agents improve forecast combination in empirical economics.
problem Hidden researcher degrees of freedom in AI-generated code.
method Adapted agent-loop architecture to empirical economics, added holdout evaluation.
result Independent agent searches find better forecast methods than benchmarks.
Study evaluates off-policy reinforcement learning methods.
problem Evaluating reinforcement learning policies without direct access to the behavior policy.
method Experimental benchmarking suite with diverse design parameters.
result Guidelines for using off-policy evaluation methods in practice.
Compensation methods correct overestimation of adversarial robustness in neural networks.
problem Overestimation of adversarial robustness using first-order attack methods.
method Proposed compensation methods address inaccurate gradient computation and reduce backpropagations.
result Empirical evaluation of adversarial robustness is improved with these methods.
AI agents improve forecast combination but require transparency.
problem AI coding agents increase flexibility in empirical economics, leading to hidden degrees of freedom.
method Adapted open-source agent-loop architecture to empirical economics workflow, adding post-search holdout evaluation.
result Multiple agent runs outperform standard benchmarks in rolling evaluation but not all on post-search holdout.
Empirical mode modeling improves state-space analysis of noisy data.
problem Analyzing nonlinear systems with noisy data.
method Combining empirical mode decomposition with empirical dynamic modeling.
result Empirical mode modeling enhances state-space representations in noisy data.
This research evaluates generalization measures in deep learning.
problem Understanding why deep learning models generalize well despite small training error.
method Empirical evaluation of generalization bounds and measures.
result Generalization measures should be evaluated using distributional robustness.
The (contextual) multi-armed bandit problem (MAB) provides a formalization of sequential decision-making which has many applications. However, validly evaluating MAB policies is challenging; we either resort to simulations which inherently include debatable assumptions, or we resort to expensive field trials. Recently …
Large language models correlate in errors, even with different architectures and providers.
problem Lack of empirical evidence on whether different large language models differ meaningfully.
method Empirical evaluation of over 350 large language models using two leaderboards and a resume-screening task.
result Large language models have substantial correlation in errors, even with distinct architectures and providers.
Synthetic experiments are crucial for assessing causal machine learning methods.
problem Current empirical evaluations of causal machine learning methods are insufficient and unreliable.
method Propose principles for conducting rigorous empirical analyses with synthetic data.
result Rigorous synthetic experiments are essential for building trust in causal machine learning methods.
The paper evaluates biased methods for alpha-divergence minimization.
problem The impact of bias on solutions found for alpha-divergence minimization.
method Empirical evaluation of biased methods for alpha-divergence minimization, focusing on bias effects and dimensionality.
result Solutions are biased towards KL-divergence minimizers and require impractical computation in high dimensions to minimize alpha-divergence.
Randomized control methods improve asset pricing and performance analysis.
problem Challenges in drawing inferences from traditional random portfolios in performance evaluation.
method Geometric random walks and Markov chain Monte Carlo methods to construct flexible control groups.
result Captured premia associated with size, value, quality, and momentum in a constrained setting.
Bayesian method infers contextual bandit policies robustly.
problem Inference of contextual bandit policies in small sample sizes.
method Empirical likelihood for Bayesian inference.
result Accurate uncertainty measurements and policy comparison.
Causal inference is central to many areas of artificial intelligence, including complex reasoning, planning, knowledge-base construction, robotics, explanation, and fairness. An active community of researchers develops and enhances algorithms that learn causal models from data, and this work has produced a series of im…
Paper introduces Modular Jets for diagnosing model decompositions in pipelines.
problem Evaluating model decompositions in pipelines for unique identification.
method Estimates empirical jets from module-level representations to diagnose mirage vs identifiable decompositions.
result Proves jet-identifiability theorem for two-module linear regression pipelines.
This paper evaluates how different imputation methods affect predictive models.
problem The impact of different imputation methods on predictive models' performance.
method Systematic evaluation of various imputation methods for different data sets and machine learning algorithms.
result Recommendation of a general method for empirical benchmarking of imputation methods.
A non-parametric method for evaluation of the aggregate loss distribution (ALD) by combining and numerically inverting the empirical characteristic functions (CFs) is presented and illustrated. This approach to evaluate ALD is based on purely non-parametric considerations, i.e., based on the empirical CFs of frequency …
We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In a…
Proposes a stability evaluation criterion for learning models using distributional perturbations.
problem Ensuring reliable deployment of learning models in out-of-sample environments.
method Uses optimal transport discrepancy with moment constraints to quantify minimal perturbation required for model deterioration.
result Validates the practical utility of the stability evaluation criterion across various real-world applications.
We present the first differentially private algorithms for reinforcement learning, which apply to the task of evaluating a fixed policy. We establish two approaches for achieving differential privacy, provide a theoretical analysis of the privacy and utility of the two algorithms, and show promising results on simple e…
Saliency methods can aid understanding of deep neural networks. Recent years have witnessed many improvements to saliency methods, as well as new ways for evaluating them. In this paper, we 1) present a novel region-based attribution method, XRAI, that builds upon integrated gradients (Sundararajan et al. 2017), 2) int…
Motivated by the Basel 3 regulations, recent studies have considered joint forecasts of Value-at-Risk and Expected Shortfall. A large family of scoring functions can be used to evaluate forecast performance in this context. However, little intuitive or empirical guidance is currently available, which renders the choice…
Stochastic variational inference for Bayesian deep neural network (DNN) requires specifying priors and approximate posterior distributions over neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dime…
PromptEval estimates LLM performance across many prompts, improving reproducibility.
problem Limited prompt templates affect LLM benchmark reproducibility.
method Estimates performance distribution across many prompts using borrowed strength.
result PromptEval accurately estimates performance quantiles with practical budget.
Develops a new framework for perpetual futures on binary prediction markets.
problem Lack of effective risk management in perpetual futures on binary prediction markets.
method PIRAP framework with six components: index estimator, margin sizing, leverage, funding rule, halt protocol, and eligibility framework.
result Mixed results from empirical evaluation, with some pre-registered floors passing and others failing.
New algorithms for model selection in off-policy evaluation of reinforcement learning.
problem Hyperparameter tuning for off-policy evaluation methods in reinforcement learning.
method Developed new model-free and model-based selectors with theoretical guarantees and a new experimental protocol.
result New model-free selector, LSTD-Tournament, demonstrates promising empirical performance.
LinMED is a new linear bandit algorithm with near-optimal regret bound.
problem Optimizing decision-making in linear bandit problems with sub-Gaussian distributions.
method LinMED is a randomized linear bandit algorithm with closed-form arm sampling probabilities.
result LinMED achieves a near-optimal regret bound of up to logarithmic factors.
CoinDICE estimates confidence intervals for unknown behavior policies in reinforcement learning.
problem Estimating value of a target policy using only behavior policy data.
method Function space embedding, generalized empirical likelihood method, Lagrangian optimization.
result Valid confidence intervals with tighter and more accurate estimates than existing methods.
LEEP measures transferability of learned representations efficiently.
problem Evaluating the transferability of learned representations in machine learning.
method LEEP: Log Expected Empirical Prediction, a simple measure requiring one pass through the target data set.
result LEEP predicts transfer and meta-transfer learning performance and convergence speed, outperforming existing measures.
Off-policy evaluation (OPE) in both contextual bandits and reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. The problem's importance has attracted many proposed solutions, including importance sampling (IS), self…
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
New approaches estimate recommendation metrics using sampling.
problem Understanding and resolving the use of sampling for recommendation evaluation.
method MLE and ME principles for empirical rank distribution recovery.
result Advantages of new approaches for top-k metrics estimation.
Suitability filter detects model performance degradation in real-world deployment.
problem Ensuring model reliability in safety-critical domains without access to ground truth labels.
method Uses suitability signals to evaluate classifier performance on unlabeled user data.
result The suitability filter reliably detects performance deviations due to covariate shift.
The study addresses biases in evaluating molecular optimization methods and proposes methods to reduce these biases.
problem Biases in in silico evaluation of molecular optimization methods.
method Discussion and empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
result Empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
Paper offers a fast convergence theory for offline decision making.
problem Offline decision making problems, including reinforcement learning and off-policy evaluation.
method Introduces a framework (DMOF) and algorithm (EDD) with a fast convergence guarantee.
result Demonstrates a fast convergence guarantee with a lower bound complement.
Method selects best estimator for off-policy evaluation.
problem Choosing the best estimator for off-policy evaluation.
method Generic data-driven method for estimator selection.
result Method is competitive with oracle estimator, up to a constant factor.
Proves accuracy guarantees for self-supervised learning with correlated positive pairs.
problem Lack of theoretical guarantees for self-supervised learning with correlated positive pairs.
method Novel augmentation graph concept and spectral decomposition loss.
result Provably accurate features under linear probe evaluation.
Unified approach for non-stationary and clustered bandits.
problem Solving non-stationary and clustered bandits with overlapping solutions.
method Test of homogeneity for seamless integration of non-stationary and clustered bandits.
result Unified solution framework for change detection and cluster identification.
Bayesian framework improves LLM evaluation stability and transparency.
problem Pass@k and avg@N are unstable and misleading for LLMs.
method Bayesian evaluation with posterior estimates and credible intervals.
result Posterior-based evaluation yields stable and transparent rankings.
OSIRIS reduces variance in off-policy evaluation by omitting irrelevant states.
problem High variance in importance sampling-based OPE estimators.
method OSIRIS reduces variance by omitting likelihood ratios associated with states irrelevant to return.
result OSIRIS is unbiased and has lower variance than ordinary importance sampling.
In this work, we evaluate adversarial robustness in the context of transfer learning from a source trained on CIFAR 100 to a target network trained on CIFAR 10. Specifically, we study the effects of using robust optimisation in the source and target networks. This allows us to identify transfer learning strategies unde…
Evaluating Company-specific Biases in Financial Sentiment Analysis using Large Language Modelsq-fin.CP
Study evaluates if LLMs have company-specific biases in financial sentiment analysis.
problem Evaluating if large language models exhibit company-specific biases in financial sentiment analysis.
method Comparing sentiment scores with and without company names, constructing economic models, and empirical analysis.
result LLMs show company-specific biases in sentiment analysis, impacting investor behavior and stock prices.
Doing well with less! On Sampling Techniques for Empirical Pairwise Loss Estimation/Minimizationstat.ML
Sampling a fraction of pairs can match full evaluation in machine learning losses.
problem High computational cost of full pairwise loss evaluation.
method Survey sampling techniques targeting informative pairs.
result Performance close to full pairwise evaluation achieved with frugal sampling.
EMPIR combines low and full precision DNNs to enhance robustness against adversarial attacks.
problem Vulnerability of DNNs to adversarial attacks that misclassify inputs with small perturbations.
method Ensemble of quantized DNN models with different numerical precisions.
result EMPIR ensembles increase adversarial robustness by 42.6% on average across different tasks.
Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
A new method for faster multi-objective optimization by evaluating objectives separately.
problem Finding the Pareto front of trade-offs between multiple objectives efficiently.
method Knowledge Gradient with decoupled evaluations, accounting for different costs.
result The method significantly outperforms existing approaches in terms of evaluation cost.
The paper evaluates and improves uncertainty estimates in neural networks for safety-critical applications.
problem Quantifying uncertainty in neural networks for safety-critical systems.
method Proposes a statistical test for evaluating uncertainty realism in neural networks and transfers a classification architecture to image-to-image tasks.
result The variational U-Net architecture significantly improves uncertainty realism in image-to-image tasks compared to a plain model.
The study improves VaR forecast accuracy by modeling conditional quantile dynamics.
problem Improving the accuracy of Value-at-Risk (VaR) forecasts for time-varying quantiles.
method Time-varying modeling of VaR, evaluation via simulation, asymmetric Mean Absolute Deviation loss function.
result Substantial improvements in forecasting conditional quantiles by maintaining predicted quantile unchanged.