Study constructs a Japanese financial LLM benchmark.
problem Need for domain-specific benchmarks for LLMs.
method Constructed a benchmark with multiple Japanese and financial domain tasks.
result GPT-4 outperforms other models in the benchmark.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Benchmark Study”
Benchmark study evaluates 8 clustering methods on 99 UCR time series datasets.
problem Assessing the performance of clustering methods on time series data.
method Examines 8 clustering methods across 3 categories and 3 distance measures on 99 UCR datasets.
result Provides a comprehensive dataset-level assessment of clustering methods.
This article provides a comprehensive study of different ways to make speed benchmarks of gradient boosted decision trees algorithm. We show main problems of several straight forward ways to make benchmarks, explain, why a speed benchmarking is a challenging task and provide a set of reasonable requirements for a bench…
The study initiates a theoretical analysis of dynamic benchmarking models.
problem Lack of theoretical foundation and empirical studies in dynamic benchmarks.
method Examined two realizations of dynamic benchmarking: sequential and hierarchical dependency models.
result Sequential dynamic benchmarks show initial performance improvement but can stall after three rounds due to label noise.
Study optimizes portfolio to minimize relative drawdown duration, penalizing unfavorable performance states.
problem Minimizing relative drawdown duration in portfolio optimization relative to a benchmark.
method Introduces a benchmark-relative drawdown-duration criterion penalizing unfavorable performance states. Uses a one-dimensional Markovian representation and Hamilton-Jacobi-Bellman equation.
result Derives explicit projection-based characterization of the optimal feedback control and identifies geometric settings for unique strong solutions.
Study proposes new methods to calculate probabilistic benchmarks in noisy data.
problem Identifying opportunities for improvement in comparable units with noisy data.
method 2-step methodology involving undersampling and relevance vector machine.
result Higher discrimination power achieved with macro-economic environment variables.
Recent studies have significantly improved the state-of-the-art on common-sense reasoning (CSR) benchmarks like the Winograd Schema Challenge (WSC) and SWAG. The question we ask in this paper is whether improved performance on these benchmarks represents genuine progress towards common-sense-enabled systems. We make ca…
We study the pricing and hedging of derivatives in incomplete financial markets by considering the local risk-minimization method in the context of the benchmark approach, which will be called benchmarked local risk-minimization. We show that the proposed benchmarked local risk-minimization allows to handle under extre…
OpenML Benchmarking Suitesstat.ML
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create an…
Study benchmarks TSC algorithms in distinguishing diffusions using the likelihood ratio test.
problem Benchmarking optimality of TSC algorithms in distinguishing diffusion processes.
method Proposes to benchmark TSC algorithms using the likelihood ratio test (LRT).
result LRT benchmarks are computationally efficient and can be applied to various time series types.
We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to …
Novel strategy benchmarks observational studies against randomized trials.
problem Benchmarking observational studies for treatment effect bias.
method Statistical test for null hypothesis of treatment effect difference.
result Valid lower bound on maximum bias strength for any subgroup.
Study benchmarks 19 survival models on 34 datasets, finding Cox model still best.
problem Quantitative comparison of survival models on low-dimensional data.
method Comprehensive benchmarking of 19 models on 34 datasets, tuning and evaluating using 6 metrics.
result Cox Proportional Hazards model remains best overall for low-dimensional, right-censored data.
Paper studies pricing and hedging of nonreplicable insurance contracts using benchmark-neutral approach.
problem Pricing and hedging of long-term insurance contracts like variable annuities.
method Benchmark-neutral pricing framework using stock growth optimal portfolio as numéraire.
result Prices can be significantly lower than risk-neutral ones, offering attractive long-term risk-management.
Study benchmarks embedding-based entity alignment methods for KGs.
problem Align entities across different KGs using embeddings.
method Survey and categorize 23 embedding-based methods, propose new KG sampling algorithm, develop open-source library.
result Understood strengths and limitations of embedding-based methods.
Simulation-Based Benchmarking of Reinforcement Learning Agents for Personalized Retail Promotionscs.AI
This study benchmarks AI agents for personalized retail promotions using simulations.
problem Optimizing coupon targeting for sparse customer purchase events.
method Comprehensive simulations of customer shopping behaviors; training RL agents on batch data.
result Contextual bandit and deep RL methods outperform static policies in sparse reward environments.
Study benchmarks methods for learning non-Cartesian k-space trajectories and reconstruction.
problem Benchmarking methods for learning non-Cartesian k-space trajectories and reconstruction.
method Comparing PILOT, BJORK, and HybLearn schemes to learn non-Cartesian k-space trajectories and reconstruction.
result HybLearn scheme outperforms other methods in learning and comparing non-Cartesian k-space trajectories and reconstruction.
Paper benchmarks machine learning for detecting process curve drifts.
problem Detecting drifts in multivariate manufacturing process data.
method Synthetic data generation and evaluation score introduction.
result Existing algorithms often fail with complex drift scenarios.
MementoML: Performance of selected machine learning algorithm configurations on OpenML100 datasetscs.LG
Paper compiles ML algorithm performance benchmarks on OpenML datasets.
problem Finding optimal hyperparameters for ML algorithms efficiently.
method Generated benchmark data for 7 ML algorithms on 39 datasets, fixed hyperparameters before testing.
result Comprehensive dataset of ML algorithm performance sensitivity.
Study introduces KorFinMTEB for Korean financial texts, revealing model limitations.
problem Limited evaluation benchmarks for low-resource domains, especially Korean.
method Developed KorFinMTEB, a tailored benchmark for Korean financial texts.
result Models perform better on translated benchmarks than on domain-specific ones.
This study provides benchmarks for different implementations of LSTM units between the deep learning frameworks PyTorch, TensorFlow, Lasagne and Keras. The comparison includes cuDNN LSTMs, fused LSTM variants and less optimized, but more flexible LSTM implementations. The benchmarks reflect two typical scenarios for au…
Performance of investment managers are evaluated in comparison with benchmarks, such as financial indices. Due to the operational constraint that most professional databases do not track the change of constitution of benchmark portfolios, standard tests of performance suffer from the "look-ahead benchmark bias," when t…
Study improves recognition of long-tail visual relationships.
problem Improving recognition of structured visual relationships from long-tail classes.
method Developed two benchmarks, introduced VilHub loss, and applied RelMix augmentation.
result Simple techniques significantly improved performance on tail classes.
The paper solves a control problem using reflections to track a benchmark process.
problem Optimal consumption with a benchmark process that grows over time.
method Introduced two auxiliary state processes with reflections to transform the problem into a more tractable form.
result Established the existence of a unique classical solution to the dual PDE.
MLPerf, an emerging machine learning benchmark suite strives to cover a broad range of applications of machine learning. We present a study on its characteristics and how the MLPerf benchmarks differ from some of the previous deep learning benchmarks like DAWNBench and DeepBench. We find that application benchmarks suc…
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
Benchmarking Counterfactual Interpretability in Deep Learning Models for Time Series Classificationcs.LG
This work benchmarks counterfactual methods in time series classification.
problem Lack of benchmarking studies for counterfactual methods in time series classification.
method Redesign metrics for sparsity, plausibility, and consistency; systematically benchmark 6 CF methods on 30 datasets.
result Performance of CF methods varies across metrics and models.
Randomly guessing weights helps analyze RL benchmarks objectively.
problem Understanding the complexity of reinforcement learning benchmarks.
method Generate policy networks by randomly guessing their parameters, evaluate on benchmarks, and analyze results.
result Small untrained networks can provide a robust baseline for various RL tasks.
Study finds little progress in medical machine learning benchmarks over 3 years.
problem Lack of meaningful progress in medical machine learning benchmarks for structured healthcare data.
method Comprehensive review and meta-analysis of benchmarks in medical machine learning for structured data.
result Deep recurrent models perform only better than logistic regression on certain clinical prediction tasks.
A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasetsstat.ML
Study introduces a benchmark suite for evaluating neural MI estimators on real-world unstructured datasets.
problem Lack of comprehensive evaluation methods for neural MI estimators on real-world unstructured datasets.
method Developed a benchmark suite using same-class sampling and a binary symmetric channel trick.
result Showed accurate manipulation of true MI values of real-world datasets.
Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Modelsq-fin.PM
Study benchmarks LLMs in portfolio optimization tasks.
problem Evaluate financial decision-making of LLMs.
method Mathematically explicit portfolio optimization problems with multiple-choice questions.
result Distinct performance patterns among LLMs in different financial tasks.
Study optimal consumption with relaxed benchmarks and drawdown constraints.
problem Optimal consumption under relaxed benchmark tracking and consumption drawdown constraint.
method Transformed stochastic control problem into regular control problem with state-control constraints, then solved using dual transform and optimal consumption behavior.
result Closed-form solution for optimal investment and consumption in feedback form.
Benchmarking Minimax Linkagestat.ML
Minimax linkage was first introduced by Ao et al. [3] in 2004, as an alternative to standard linkage methods used in hierarchical clustering. Minimax linkage relies on distances to a prototype for each cluster; this prototype can be thought of as a representative object in the cluster, hence improving the interpretabil…
UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).
Study benchmarks 26 clustering validity measures.
problem Determining the best clustering solution from candidates.
method Enhanced revision of previous methodology with three sub-methodologies.
result Comprehensive evaluation of 26 internal validity indexes.
QuantBench benchmarks AI methods for quantitative investment.
problem Lack of a standardized benchmark for AI in quantitative investment.
method Developed an industrial-grade benchmark platform with standardization, flexibility, and full-pipeline coverage.
result Revealed critical research directions in AI for quantitative investment.
The paper clarifies conditions for using benchmark scores in machine learning.
problem Using benchmark scores to draw scientific inferences about learning problems.
method Developing conditions of construct validity inspired by psychological measurement theory.
result Clarifies conditions under which benchmark scores support diverse scientific claims.
Benchmark of Likelihood-Free Inference Methods based on Neural and Optimal Transport Approachesstat.ME
This study benchmarks likelihood-free inference methods for models with heavy-tailed or discrete data.
problem Comparing likelihood-free inference methods for models with structural features like heavy-tails or discreteness.
method Four approaches: MLE, NBE, EOT, and AW-NBE are evaluated using simulations.
result The choice of evaluation tools is crucial for models with extremes and discrete data.
New model reveals significant impact of data and parameter variations on machine learning benchmarks.
problem Variance in machine learning benchmarks due to data sampling, initialization, and hyperparameters.
method Modeling the benchmarking process, analyzing comparison methods, and studying error rates.
result Adding more sources of variation reduces compute cost while improving performance.
Deep learning models struggle with new data in stock price trend prediction.
problem Stock price trend prediction using Deep Learning models.
method Examination of fifteen state-of-the-art DL models on LOB data, using LOBCAST framework.
result All models show significant performance drop with new data, questioning their market applicability.
The paper extends Merton's problem by adding benchmark tracking, finding optimal strategies.
problem Maximizing consumption utility with a trade-off against benchmark performance.
method Developed a convex duality theorem and derived optimal strategies for specific cases.
result Found optimal portfolio and consumption strategies for CRRA utility and geometric Brownian motion benchmarks.
Paper establishes a comprehensive benchmark for ECG time-series analysis.
problem Incomplete understanding of ECG signal properties and limitations in evaluation metrics.
method Categorization of downstream applications, identification of limitations, introduction of a novel metric, benchmarking of time-series models.
result Validation of the effectiveness of the proposed metric and model architecture.
A benchmarking framework for studying data geometry.
problem Generalization and approximation error bounds in deep learning.
method Repurposing and extending dSprites and COIL-20 with additional transformation dimensions and dense, axis-aligned sampling.
result Near-ground-truth accuracy in curvature, reach, and volume estimation.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
Study characterizes harmful low-fidelity data sources for surrogate models.
problem Identifying which low-fidelity data sources to use in constructing surrogate models.
method Employed benchmark filtering techniques to assess harmful sources using limited data.
result Provided guidelines for using low-fidelity sources in an industrial setting.
Investigates long-term performance of multi-fidelity Bayesian optimization.
problem Potential long-term under-performance of multi-fidelity Bayesian optimization.
method Simple benchmark study to investigate long-term performance.
result Under-performance of multi-fidelity Bayesian optimization in certain scenarios.
We provide a benchmark dataset for hand gesture recognition using force myography.
problem Lack of publicly available benchmark data for force myography hand gesture recognition.
method Collected data from 20 persons covering 18 unique gestures using a commercially available sensor setup.
result Improved gesture recognition accuracy through transfer learning.
WOODS benchmarks improve understanding of time series OOD generalization.
problem Limited understanding of OOD generalization in time series.
method Presented eight open-source time series benchmarks and revised OOD algorithms.
result Large room for improvement in OOD generalization algorithms for time series.