Paper tackles efficient evaluation of natural stochastic policies in offline RL.
problem Efficiency issues in evaluating natural stochastic policies due to unknown evaluation policy.
method Derive efficiency bounds for tilting and modified treatment policies, propose nonparametric estimators.
result Proposed estimators attain efficiency bounds under lax conditions and enjoy partial double robustness.
ASEs use surrogate estimation to efficiently evaluate model performance with minimal labels.
problem Efficient model evaluation with limited labels.
method Surrogate-based estimation and active learning.
result ASEs offer greater label-efficiency than current methods for deep neural networks.
New method efficiently evaluates policies using trajectory data.
problem Statistically efficient policy evaluation with limited data.
method Trajectory-based approach for policy evaluation.
result Improved sample complexity for policy evaluation.
GRAPE improves RL policy evaluation in noisy environments.
problem Noise in real-world RL environments makes policy evaluation algorithms inefficient or prone to errors.
method GRAPE combines gap-increasing value update operators and off-policy eligibility trace.
result GRAPE is more efficient and noise-tolerant than existing methods.
New method improves evaluation of new policies in reinforcement learning.
problem Efficient off-policy evaluation in Markov decision processes.
method Double Reinforcement Learning (DRL) estimator for q-functions and marginalized density ratios. result DRL is efficient and doubly robust under certain conditions.
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
R-AutoEval+ improves model evaluation efficiency and reliability using adaptive synthetic data.
problem Accurate model selection from AI candidates using real-world data is costly and impractical at scale.
method R-AutoEval+ uses adaptive prediction-powered inference to correct bias in autoevaluators while maintaining or improving sample efficiency.
result R-AutoEval+ provides finite-sample reliability guarantees and enhanced sample efficiency compared to conventional methods.
Proposes a method for evaluating multiple dimensions of organizational effectiveness using DEA.
problem Evaluating multiple dimensions of organizational effectiveness in large data sets.
method Introduces two regularized DEA models (SBM and GP-SBM) to estimate both dimension-specific and aggregate efficiency scores.
result Demonstrates improved efficiency and validity compared to conventional methods.
Active testing reduces label costs for efficient model evaluation.
problem Real-world applications require expensive test labels, disconnecting from existing model evaluation methods.
method Derives acquisition strategies to select test points efficiently, addressing label bias and variance.
result Active testing improves model evaluation efficiency without sacrificing accuracy.
We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In a…
The paper develops efficient methods for estimating posterior distributions with few expensive likelihood evaluations.
problem Estimating posterior distributions with limited expensive likelihood evaluations.
method Active Bayesian regression using Gaussian processes for query efficiency.
result The proposed methods significantly reduce the number of likelihood evaluations needed for posterior estimation.
A computational theory reduces agent evaluation errors and speeds up processes.
problem Efficient evaluation of mini agents at reduced cost.
method Developed a computational theory and a meta-learner to handle heterogeneous agents.
result Reduced evaluation errors by 24.1% to 99.0% across various scenarios.
Cer-Eval saves LLM evaluation costs while maintaining accuracy.
problem Challenges in evaluating large language models due to large dataset requirements.
method Adapts to different evaluation objectives, uses test sample complexity, and develops a partition-based algorithm.
result Cer-Eval can save 20-40% test points with comparable accuracy and 95% confidence guarantee.
Paper tackles efficient off-policy evaluation in long-horizon settings.
problem Efficient off-policy evaluation in long-horizon settings with diminishing overlap.
method Derives efficiency bounds for OPE under Markovian and time-invariant structures, develops a new DRL estimator.
result DRL estimator provides efficient OPE even with just one dependent trajectory in time-invariant Markov decision processes.
Efficiently evaluates new ranking policies using click models.
problem Evaluate new ranking policies offline and optimize them before deployment.
method Proposes evaluation algorithms using click models to estimate expected clicks from logged data.
result Our estimators are more statistically efficient than those that do not use click models.
AlphaEval evaluates alpha mining models efficiently and comprehensively.
problem Lack of systematic evaluation for alpha mining models.
method Unified, parallelizable evaluation framework assessing predictive power, stability, robustness, financial logic, and diversity.
result AlphaEval achieves evaluation consistency comparable to comprehensive backtesting, providing more comprehensive insights and higher efficiency.
Normalizing Flows model tractable distributions for efficient sampling and evaluation.
problem Creating efficient generative models for sampling and density evaluation.
method Construct and use Normalizing Flows to learn distributions.
result Comprehensive review of current Normalizing Flow methods and future directions.
New method uses model comparison signals to improve LLM evaluation accuracy.
problem Limited benchmark sizes and model stochasticity in evaluating LLMs' mathematical reasoning.
method Combines standard labeled outcomes with model comparison signals to design a statistically efficient evaluation framework.
result Semiparametric estimator achieves the semiparametric efficiency bound and substantially improves ranking accuracy.
Efficiently evaluates and optimizes counterfactual policies from bandit feedback.
problem Statistical inefficiency in evaluating and optimizing policies from bandit feedback.
method Developed estimators with lowest variance for expected reward from counterfactual policies.
result Achieved significant variance reduction in estimators compared to standard methods.
New benchmark protocol evaluates neural network optimizers for efficiency and data shift sensitivity.
problem Benchmarking neural network optimizers with hyperparameter complexity and data shift sensitivity.
method Proposed a new evaluation protocol combining end-to-end and data-addition training efficiency, using bandit hyperparameter tuning and human study validation.
result No clear winner across all tasks, highlighting the complexity of optimizer performance.
Proposes a transformer-based approach for anomaly detection in time series data.
problem Inadequate evaluation metrics and inability to capture temporal features in time series anomaly detection.
method Introduces a proper evaluation metric and proposes a transformer-based approach for anomaly detection in time series data.
result Transformer-based approach outperforms state-of-the-art detectors in detecting sequential anomalies.
ProEval efficiently estimates AI performance and discovers failures using pre-trained Gaussian Processes.
problem Resource-intensive evaluation of generative AI models.
method ProEval uses pre-trained Gaussian Processes and Bayesian quadrature to estimate performance and discover failures.
result ProEval requires significantly fewer samples to achieve accurate performance estimates and reveals more diverse failure cases.
This paper improves auto-augment efficiency by sharing augmentation weights.
problem Efficient evaluation of augmentation policies for model training.
method Augmentation-Wise Weight Sharing (AWS) to create a fast yet accurate proxy task.
result Augmentation policies found achieve superior accuracies compared to existing methods.
mAIS improves free energy evaluation efficiency.
problem Computational infeasibility of exact free energy evaluation.
method mAIS, a marginalized version of AIS.
result mAIS is more efficient under certain conditions.
Active testing for large language models is made more efficient and accurate.
problem Efficient evaluation of large language models with limited labels.
method Cost-saving measures and in-context learning for constructing a surrogate model.
result Significantly more accurate evaluations of LLM performance compared to random data acquisition.
Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
New estimators improve OPE efficiency and stability.
problem Evaluate novel policies without exploration in RL.
method Empirical likelihood estimators that are more efficient and stable than existing methods.
result New estimators are more efficient and stable than IS, SNIS, and DR.
This paper improves sample efficiency for off-policy evaluation with preference data.
problem Improving sample efficiency for off-policy evaluation with preference data.
method Using a deep neural network to learn the value function and leveraging manifold structure.
result Established a provably efficient guarantee for off-policy evaluation with RLHF.
New method reduces variance in subpopulation model performance estimates.
problem High variance in subpopulation performance metrics for small groups.
method Using an evaluation model to form model-based metric (MBM) estimates.
result MBMs produce more accurate and lower variance estimates for small subpopulations.
Unified Bayesian framework for efficient off-policy evaluation and learning in large action spaces.
problem Efficient off-policy evaluation and learning in systems with correlated actions.
method Unified Bayesian framework with structured priors and sDM approach.
result sDM leverages action correlations without compromising computational efficiency.
This paper identifies critical cases for evaluating PV investment impacts on MV networks efficiently.
problem Challenges in maintaining and controlling voltages in MV distribution networks due to increasing PV generation.
method Clustering MV nodes based on electrical adjacency and time blocks, identifying critical cases for further study.
result A scalable method to time efficiently identify critical cases for PV investment evaluation.
Cramming method evaluates learned policies from contextual bandits efficiently.
problem Evaluating final learned policies from contextual bandit algorithms.
method On-policy evaluation using a single pass of data, ensuring consistency and asymptotic normality.
result Cramming method reduces evaluation standard error by approximately 40% compared to off-policy methods.
AlphaX uses MCTS and Meta-DNN to improve NAS efficiency and accuracy.
problem Improving sample efficiency and network evaluation cost in NAS.
method Adaptive MCTS with Meta-DNN for prediction and distributed rollouts for cost reduction.
result AlphaX finds architectures with high accuracy (97.84% on CIFAR-10, 75.5% on ImageNet) in fewer samples.
Paper addresses off-policy evaluation and learning with covariate shift.
problem Evaluating and training a new policy using historical data with a covariate shift.
method Derives efficiency bounds and proposes doubly robust estimators for OPE and OPL under covariate shift.
result Proposes estimators for off-policy evaluation and learning under covariate shift.
The paper evaluates functions of stable Lévy processes and their extrema efficiently.
problem Efficiently evaluating functions of stable Lévy processes and their extrema.
method Integral representations, conformal acceleration technique, simplified trapezoid rule.
result Efficient numerical procedures for cumulative probability distribution functions (cpdfs) are developed.
Paper explores BERT's efficiency on SQuAD2.0, freezing layers and using adapters.
problem Improving BERT's efficiency for SQuAD2.0 while maintaining performance.
method Freezing transformer layers, using adapters, and context-aware convolutional filters.
result Context-aware convolutional filters do not improve practical efficiency.
Researchers discuss connections and distinctions between DRITR and Kallus' work, focusing on policy evaluation and efficiency.
problem Policy evaluation and efficiency under distributionally robust individualized treatment rules.
method Efficient policy evaluation using combined data and DRITR under varying sample sizes.
result Efficient value function estimates can deliver competitive performance under certain conditions but not always.
Paper finds efficient OPE estimator for multiple logging policies with minimum variance.
problem Finding optimal importance sampling weights for multiple logging policies with varying variances.
method Established efficiency bound under stratified sampling and proposed an estimator achieving this bound.
result Proposed estimator achieves minimum variance for any instance.
This paper tackles label-efficient evaluation in extreme class imbalance.
problem Challenges in obtaining a sufficient sample for accurate evaluation in tasks with extreme class imbalance.
method Develops a framework for online evaluation based on adaptive importance sampling.
result Establishes strong consistency and a central limit theorem for performance estimates.
Efficiently evaluate generative models at the prompt level using tensor factorization.
problem Fine-grained evaluations of generative models are costly and often misaligned with human judgment.
method Tensor factorization model that merges cheap autorater data with a small set of human gold-standard labels.
result The method provides accurate and tight confidence intervals for model performance.
Abstractor enhances Transformers for relational reasoning, improving sample efficiency and performance.
problem Improving sample efficiency and performance in relational tasks.
method Introduces Abstractor module with relational cross-attention to enable explicit relational reasoning.
result Dramatic improvements in sample efficiency and performance on various relational tasks.
New method optimizes costly evaluations in Bayesian optimization.
problem Costly evaluations in BO methods, especially in hyperparameter tuning.
method Formulated as a CMDP, developed rollout approximation for optimal policy.
result Improved cost efficiency over standard BO methods.
AVATAR uses a surrogate model to quickly evaluate ML pipelines, saving time and resources.
problem Time-consuming evaluation of ML pipelines limits exploration of complex models.
method AVATAR employs a surrogate model to assess pipeline validity without execution.
result AVATAR accelerates ML pipeline evaluation, improving efficiency in complex scenarios.
New algorithm optimizes complex combinatorial structures efficiently.
problem Optimizing expensive functions over combinatorial structures.
method Adaptive, scalable model using semidefinite programming.
result Consistently outperforms other methods in evaluations.
BOSH optimizes functions with stochastic evaluations more efficiently and precisely.
problem Optimizing functions with noisy evaluations can lead to suboptimal solutions.
method BOSH uses a hierarchical Gaussian process to generate a growing pool of realizations.
result BOSH provides more efficient and higher-precision optimization than standard BO.
New method evaluates personalized treatment in critical care, robust to death.
problem Truncation by death in critical care makes traditional DTR evaluation ineffective.
method Principal stratification-based approach, focusing on always-survivor value function, with a semiparametrically efficient, multiply robust estimator.
result Demonstrates robustness and efficiency of the method for personalized treatment optimization.
Automates debiasing for large language model evaluations through Fisher random walk.
problem Rigorous and scalable evaluation of large language models.
method Semiparametric efficient estimator using Fisher random walk for weighted residual balancing.
result Efficient estimation of contextual preference scores for large language models.
PTSD improves neural samplers by combining diffusion models and PT, enhancing efficiency.
problem Efficiency and correlation issues in neural samplers compared to PT.
method Sequential training of diffusion models across temperatures, combining high-temperature models for approximate lower-temperature samples.
result Significantly improved target evaluation efficiency, outperforming diffusion-based samplers.