Experiments used in current continual learning research do not faithfully assess fundamental challenges of learning continually. Instead of assessing performance on challenging and representative experiment designs, recent research has focused on increased dataset difficulty, while still using flawed experiment set-ups…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “research evaluation”
This paper reviews off-policy evaluation methods in reinforcement learning.
problem Efficiency and accuracy of off-policy evaluation methods in reinforcement learning.
method Discussion of existing OPE methods, their statistical properties, and related research directions.
result Efficiency bounds and state-of-the-art OPE methods in reinforcement learning.
Machine learning suffers from poor design, data, and evaluation practices.
problem Ad hoc design, poor data hygiene, and lack of statistical rigor in model evaluation.
method Examines the entire machine learning process from design to evaluation, highlighting common pitfalls and providing recommendations.
result Common pitfalls in machine learning research and development are identified and actionable recommendations are provided.
The study examines dataset usage patterns in machine learning research.
problem Lack of attention to dataset dynamics in machine learning research.
method Analysis of dataset usage patterns across machine learning subcommunities and time periods (2015-2020).
result Increasing concentration on fewer and fewer datasets, significant adoption from other tasks, and concentration across the field on datasets introduced by elite institutions.
Open dataset and pipeline for realistic OPE research.
problem Lack of realistic and reproducible OPE experimental studies.
method Public logged bandit dataset and Python software.
result Enables experimental comparisons of OPE estimators.
FEET protocol evaluates foundation models across three scenarios.
problem Lack of standardized evaluation protocols for foundation models.
method Structured evaluation protocol across three use cases: frozen, few-shot, and fine-tuned embeddings.
result Demonstrates comprehensive assessment of foundation models' effectiveness.
This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and …
Experimental evaluation is a major research methodology for investigating clustering algorithms and many other machine learning algorithms. For this purpose, a number of benchmark datasets have been widely used in the literature and their quality plays a key role on the value of the research work. However, in most of t…
Paper benchmarks adversarial robustness methods on image classification.
problem Vulnerability of deep neural networks to adversarial examples.
method Established a comprehensive benchmark with robustness curves.
result Found important findings on adversarial attack and defense methods.
AlphaForgeBench evaluates LLMs as quantitative researchers, not trading agents, to address instability in financial decision-making.
problem Behavioral instability of LLMs in sequential decision-making under financial uncertainty.
method Proposes AlphaForgeBench, a framework that requires LLMs to generate executable alpha factors and compose factor-based trading strategies.
result Eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning.
EasyTime simplifies time series forecasting for researchers and practitioners.
problem Ease of use and accuracy in time series forecasting.
method One-click evaluation, automated ensemble, natural language Q&A.
result Superior forecasting accuracy compared to individual methods.
XAI-Bench releases synthetic datasets for evaluating feature attribution methods.
problem Evaluating and comparing feature attribution methods is challenging.
method Released synthetic datasets and benchmarking library.
result Efficiently evaluates feature attribution methods across various metrics.
This tutorial evaluates machine learning for healthcare applications.
problem Unrealistic expectations in healthcare AI development.
method Practical guidance on performance evaluation criteria and common mistakes.
result Better understanding and avoidance of common traps in AI healthcare.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
AI agents improve forecast combination in empirical economics.
problem Hidden researcher degrees of freedom in AI-generated code.
method Adapted agent-loop architecture to empirical economics, added holdout evaluation.
result Independent agent searches find better forecast methods than benchmarks.
Orpheus simplifies deep learning deployment on edge devices.
problem Optimizing deep learning inference on edge devices for efficiency.
method Orpheus is a new framework with a small codebase, minimal dependencies, and easy integration.
result Preliminary results show the effectiveness of Orpheus for inference optimisations.
Tool to estimate research impact for low-resource institutions.
problem Costly databases limit access for third-world institutions.
method Machine Learning for data analysis and panel regression.
result Approximation of SCOPUS Impact Factor for free.
Transparency, user trust, and human comprehension are popular ethical motivations for interpretable machine learning. In support of these goals, researchers evaluate model explanation performance using humans and real world applications. This alone presents a challenge in many areas of artificial intelligence. In this …
Study evaluates cryptocurrency markets, focusing on Bitcoin.
problem Evaluating financial markets, especially during crises.
method Multiple-bubble testing approach.
result Identifies Bitcoin's multiple bubbles during economic periods.
AI agents improve forecast combination but require transparency.
problem AI coding agents increase flexibility in empirical economics, leading to hidden degrees of freedom.
method Adapted open-source agent-loop architecture to empirical economics workflow, adding post-search holdout evaluation.
result Multiple agent runs outperform standard benchmarks in rolling evaluation but not all on post-search holdout.
Causal inference is central to many areas of artificial intelligence, including complex reasoning, planning, knowledge-base construction, robotics, explanation, and fairness. An active community of researchers develops and enhances algorithms that learn causal models from data, and this work has produced a series of im…
This paper critiques flawed MVTS anomaly detection evaluation methods and proposes a simple baseline.
problem Flawed evaluation methods in MVTS anomaly detection research.
method Robust evaluation protocols, including PCA-based baseline.
result Simple PCA-based baseline outperforms many DL approaches.
OGB provides diverse graph datasets for robust ML research.
problem Challenges in scalable and robust graph machine learning.
method Unified evaluation protocol, diverse datasets, and automated pipeline.
result Significant scalability and generalization challenges identified.
Automates research and development process by evaluating model capabilities.
problem Expanding experimental burden due to reading and verifying research directions.
method Proposes RD2Bench, a benchmark for evaluating data-centric automatic R&D.
result Demonstrates promising potential of LLMs in automating R&D process.
ISMCTS-BR learns best responses in large games, approximating worst-case performance.
problem Learning robustness to worst-case outcomes in large games.
method ISMCTS-BR, a scalable search-based algorithm for deep reinforcement learning.
result ISMCTS-BR approximates worst-case performance in large games.
In 2002, the UCR time series classification archive was first released with sixteen datasets. It gradually expanded, until 2015 when it increased in size from 45 datasets to 85 datasets. In October 2018 more datasets were added, bringing the total to 128. The new archive contains a wide range of problems, including var…
This paper evaluates metrics for graph generative models, addressing common pitfalls.
problem Evaluating and comparing graph generative models effectively.
method Systematic evaluation of MMD, analysis of synthetic and real graphs, practical recommendations.
result MMD can be problematic; practical solutions are provided.
This research evaluates the performance of an Artificial Neural Network based prediction system that was employed on the Shanghai Stock Exchange for the period 21-Sep-2016 to 11-Oct-2016. It is a follow-up to a previous paper in which the prices were predicted and published before September 21. Stock market price predi…
Our research extends the Bilingual Evaluation Understudy (BLEU) evaluation technique for statistical machine translation to make it more adjustable and robust. We intend to adapt it to resemble human evaluation more. We perform experiments to evaluate the performance of our technique against the primary existing evalua…
As reinforcement learning (RL) achieves more success in solving complex tasks, more care is needed to ensure that RL research is reproducible and that algorithms herein can be compared easily and fairly with minimal bias. RL results are, however, notoriously hard to reproduce due to the algorithms' intrinsic variance, …
Framework assesses autograders' reliability and biases.
problem Mixed reliability and biases in autograders for LLM evaluation.
method Bayesian GLMs to model evaluation outcomes.
result Explicit quantification of scoring differences and biases.
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public arc…
Benchmark study evaluates 8 clustering methods on 99 UCR time series datasets.
problem Assessing the performance of clustering methods on time series data.
method Examines 8 clustering methods across 3 categories and 3 distance measures on 99 UCR datasets.
result Provides a comprehensive dataset-level assessment of clustering methods.
Survey evaluates 20 anomaly detection methods for time-series data.
problem Comparing statistical, machine learning, and deep learning methods for anomaly detection.
method Comprehensive evaluation of 20 methods from three categories on public datasets.
result Provides insights into the performance and suitability of anomaly detection methods.
New framework assesses graph-learning datasets for better evaluation.
problem Insufficient evaluation of graph-learning datasets and methods.
method Introduces Rings framework for dataset ablations and proposes performance separability and mode complementarity measures.
result Demonstrates utility of Rings framework for graph-learning dataset evaluation.
Archive of 20 time series datasets for forecasting evaluation.
problem Lack of comprehensive time series forecasting datasets.
method Compilation and characterisation of 20 datasets from various domains.
result Characterisation and performance evaluation of datasets.
Automated classification of metadata of research data by their discipline(s) of research can be used in scientometric research, by repository service providers, and in the context of research data aggregation services. Openly available metadata of the DataCite index for research data were used to compile a large traini…
FedML aims to improve FL research by providing a library and benchmark.
problem Inconsistent FL algorithm development and performance comparison.
method FedML offers an open research library and benchmark supporting diverse computing paradigms and flexible API design.
result FedML facilitates fair algorithm comparison and development in federated learning.
MARS-Gym framework for marketplaces to train and evaluate recommender systems.
problem Challenges in designing, training, and evaluating recommender systems in marketplaces.
method Open-source framework for Reinforcement Learning agents in marketplaces.
result Empowers researchers and engineers to quickly build and evaluate agents for recommendations.
Evaluation metrics for prediction models don't fully reflect intervention impact.
problem Standard metrics don't accurately reflect reduction in patient outcomes from model use.
method Synthesized and discussed various evaluation methods, analyzed with simulated and real data.
result Evaluations without interventional data are limited or require strong assumptions.
GANs generate realistic financial data for research.
problem Creating realistic financial datasets for research and benchmarking.
method Proposed a novel type of GAN and methods for preprocessing and evaluation.
result GANs can replicate financial datasets with high fidelity.
A common problem in disciplines of applied Statistics research such as Astrostatistics is of estimating the posterior distribution of relevant parameters. Typically, the likelihoods for such models are computed via expensive experiments such as cosmological simulations of the universe. An urgent challenge in these rese…
Synthetic data can be used to ask more questions and accelerate discovery with provable validity guarantees.
problem Valid inference with synthetic data
method Task exchangeability
result Provable validity guarantees for synthetic data inference
Unlike major Western languages, most African languages are very low-resourced. Furthermore, the resources that do exist are often scattered and difficult to obtain and discover. As a result, the data and code for existing research has rarely been shared. This has lead a struggle to reproduce reported results, and few p…
Quantifying the Blockchain Trilemma: A Comparative Analysis of Algorand, Ethereum 2.0, and Beyondecon.GN
This study evaluates Algorand and Ethereum 2.0 for blockchain trilemma challenges.
problem Balancing decentralization, security, and scalability in blockchain systems.
method Comparative analysis of Algorand and Ethereum 2.0 using metrics for decentralization, scalability, and security.
result Each system has strengths in addressing the blockchain trilemma, providing insights for other blockchain technologies.
Paper uses LightGBM for mobile user credit assessment.
problem Improving credit evaluation methods for communication operators.
method Data preprocessing, feature engineering, multiple machine learning models integration.
result Established a suitable fusion model for operator user credit evaluation.
The appeal of metric evaluation of research impact has attracted considerable interest in recent times. Although the public at large and administrative bodies are much interested in the idea, scientists and other researchers are much more cautious, insisting that metrics are but an auxiliary instrument to the qualitati…
This study synthesizes stablecoin systems and develops a performance evaluation framework.
problem Fragmented academic research on stablecoins across economics, law, and computer science.
method Multi-method research design including literature synthesis, performance evaluation framework, and case study.
result Unified taxonomy and performance evaluation framework for stablecoin design.