Randomly guessing weights helps analyze RL benchmarks objectively.
problem Understanding the complexity of reinforcement learning benchmarks.
method Generate policy networks by randomly guessing their parameters, evaluate on benchmarks, and analyze results.
result Small untrained networks can provide a robust baseline for various RL tasks.
PINNs struggle with increasingly complex ODEs, especially when parameters control their complexity.
problem Evaluating physics-informed neural networks on complex coupled ODEs.
method Tuned benchmarks of partial differential equations and harmonic oscillators; varying network architecture and training method.
result PINNs fail to solve complex ODEs, revealing issues like insufficient capacity, poor conditioning, and high local curvature.
UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).
While current benchmark reinforcement learning (RL) tasks have been useful to drive progress in the field, they are in many ways poor substitutes for learning with real-world data. By testing increasingly complex RL algorithms on low-complexity simulation environments, we often end up with brittle RL policies that gene…
The study initiates a theoretical analysis of dynamic benchmarking models.
problem Lack of theoretical foundation and empirical studies in dynamic benchmarks.
method Examined two realizations of dynamic benchmarking: sequential and hierarchical dependency models.
result Sequential dynamic benchmarks show initial performance improvement but can stall after three rounds due to label noise.
WILD-SCAV benchmarks AI in complex 3D FPS environments.
problem Lack of complexity and diversity in RL environments.
method Developed a 3D open-world FPS game environment.
result Demonstrates effectiveness in benchmarking RL algorithms.
CausalMan simulates complex causality for fair benchmarking.
problem Lack of realistic causal models with known data generating processes.
method Developed a physics-based simulator, CausalMan, for large-scale causality.
result Demonstrated inadequacy of state-of-the-art approaches and analyzed their performance.
EDINET-Bench evaluates LLMs on complex financial tasks using Japanese financial statements.
problem Challenges in evaluating LLMs on financial tasks due to specialized expertise and scarce benchmarks.
method Developed EDINET-Bench, an open-source Japanese financial benchmark for LLMs on tasks like fraud detection and earnings forecasting.
result State-of-the-art LLMs perform only marginally better than logistic regression in financial tasks, highlighting the need for more realistic benchmarks.
AI benchmarks evaluate football team performance using generative models.
problem Evaluating human performance in complex interactive tasks is error-prone and unreliable.
method Trained Conditional VRNN Model on player and ball tracking data to imitate and predict team interactions.
result Trained model as a useful benchmark for evaluating team performance in football.
GeMA learns latent manifolds to benchmark complex systems.
problem Benchmarking complex systems like rail networks and economies with classical methods.
method Geometric Manifold Analysis (GeMA) using a productivity-manifold variational autoencoder (ProMan-VAE).
result GeMA provides more nuanced efficiency evaluations in complex systems.
PhysicsFormer improves TSF models for GSWF with WEATHER-5K dataset.
problem Lack of comprehensive datasets for GSWF.
method PhysicsFormer combines dynamic core and Transformer, enforcing physical consistency.
result PhysicsFormer outperforms TSF models in operational forecasting.
RETINA Benchmark evaluates Bayesian deep learning on diabetic retinopathy detection.
problem Reliable uncertainty quantification for deep learning models in medical applications.
method Design and evaluation of a real-world diabetic retinopathy dataset and tasks.
result Benchmarking of Bayesian deep learning methods on diabetic retinopathy detection tasks.
Complexity helps identify sparse risk factors in asset pricing.
problem Tension between feature richness and economic parsimony in high-dimensional asset pricing.
method Expanding feature space and using basis pursuit to discover sparse risk factors.
result Nonlinear feature expansions combined with basis pursuit yield superior out-of-sample performance.
New benchmark PVR tests neural network reasoning about indirection.
problem Understanding neural network generalization limits.
method Introducing Pointer Value Retrieval (PVR) benchmark.
result Large variations in performance across different conditions.
Learning to control linear systems is statistically hard, especially for underactuated systems.
problem Statistical difficulty of learning to control linear systems, especially underactuated ones.
method Utilized minimax lower bounds and structural assumptions to prove learning complexity can be exponential.
result Learning complexity can be at most exponential with the controllability index of the system.
Paper introduces OARF benchmark suite for federated learning systems.
problem Limited diversity in federated learning benchmarks.
method Characterizes OARF benchmark suite with diverse data and applications.
result Federated learning can effectively increase end-to-end throughput.
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
Benchmarking off-policy evaluation methods for complex policies.
problem Lack of comprehensive benchmarks for off-policy evaluation methods.
method Collection of challenging high-dimensional control tasks and datasets.
result Standardized measure of progress for OPE methods.
Generates synthetic manufacturing data for causal discovery benchmarking.
problem Lack of suitable real data for validating causal discovery algorithms.
method Distributional random forests for estimating conditional distributions.
result Semisynthetic manufacturing data adheres to a causal model.
Paper presents a new dataset for testing causal discovery methods in industrial systems.
problem Lack of real-world datasets for evaluating causal discovery methods on time series data.
method Develops a dataset from an industrial system and its known causal graph.
result Provides a benchmark for evaluating causal discovery methods in complex systems.
MiniHack simplifies creation of complex RL environments.
problem Limited availability of challenging RL benchmarks.
method Develops a sandbox framework for easy RL environment design.
result MiniHack enables rapid creation of diverse RL testbeds.
RainfallBench benchmarks GNSS-based precipitation nowcasting models, addressing complex meteorological challenges.
problem Evaluation of precipitation nowcasting models in meteorology is insufficient due to focus on periodic variables.
method RainfallBench dataset and specialized evaluation protocols for multi-scale, multi-resolution, and extreme rainfall events.
result Bi-Focus Precipitation Forecaster (BFPF) enhances rainfall time series forecasting by incorporating domain-specific priors.
Simpler GNNs with low-rank non-parametric aggregators perform well on graph benchmarks.
problem Over-engineering in GNN architectures for common semi-supervised node classification datasets.
method Replacing feature aggregation with a non-parametric learner to streamline GNN design.
result Non-parametric regression is effective for semi-supervised learning on sparse, directed networks.
SMEs provide a transparent testbed for RL evaluation.
problem Lack of precise, white-box diagnostics in RL environments.
method Synthetic Monitoring Environments (SMEs) with fully configurable task characteristics and known optimal policies.
result SMEs allow for precise evaluation of RL algorithms, revealing the impact of specific environmental properties.
Forward-prediction models enhance physical reasoning, but only for specific tasks.
problem Improving physical reasoning in complex tasks involving many objects.
method Incorporated forward-prediction models into simple physical-reasoning agents and evaluated their performance on the PHYRE benchmark.
result Forward-prediction models improve physical-reasoning performance, especially on complex tasks, but generalization to new task templates is challenging.
SubseasonalClimateUSA dataset improves subseasonal weather forecasting.
problem Challenges in subseasonal weather forecasting, especially skill of physics-based models and integration of local and global variables.
method Curated dataset for training and benchmarking subseasonal forecasting models, including various methods.
result Benchmarking suggests simple and effective ways to improve current operational models.
Recent literature on online learning has focused on developing adaptive algorithms that take advantage of a regularity of the sequence of observations, yet retain worst-case performance guarantees. A complementary direction is to develop prediction methods that perform well against complex benchmarks. In this paper, we…
Rapid intensification (RI) of tropical cyclones often causes major destruction to human civilization due to short response time. It is an important yet challenging task to accurately predict this kind of extreme weather event in advance. Traditionally, meteorologists tackle the task with human-driven feature extraction…
New method computes optimal fairness-performance trade-off without complex models.
problem Intrinsic trade-off between fairness and classifier performance.
method Computes optimal Pareto front without training complex models.
result Optimal fair representations have useful structural properties enabling efficient computation.
We use surrogate losses to obtain several new regret bounds and new algorithms for contextual bandit learning. Using the ramp loss, we derive new margin-based regret bounds in terms of standard sequential complexity measures of a benchmark class of real-valued regression functions. Using the hinge loss, we derive an ef…
A new benchmark task for evaluating policy learning in complex, high-dimensional action spaces.
problem Lack of a commonly accepted benchmark for evaluating policy learning in hierarchical tasks with high-dimensional action spaces.
method Proposed DinerDash Gym benchmark and Decomposed Policy Graph Modelling (DPGM) algorithm.
result DPGM achieves significant improvement over baselines and effectively injects domain knowledge.
Pythae is a Python library for benchmarking VAE models.
problem Improving variational autoencoders for various tasks.
method Unified implementation and framework for 19 generative autoencoder models.
result Benchmarking 19 VAE models across multiple tasks.
Molecular machine learning has been maturing rapidly over the last few years. Improved methods and the presence of larger datasets have enabled machine learning algorithms to make increasingly accurate predictions about molecular properties. However, algorithmic progress has been limited due to the lack of a standard b…
Study introduces a benchmark suite for evaluating neural MI estimators on real-world unstructured datasets.
problem Lack of comprehensive evaluation methods for neural MI estimators on real-world unstructured datasets.
method Developed a benchmark suite using same-class sampling and a binary symmetric channel trick.
result Showed accurate manipulation of true MI values of real-world datasets.
QGNN uses Quaternion space for better graph and node classification.
problem Existing GNN methods struggle with Euclidean vector space limitations.
method Proposes QGNN to learn graph representations in Quaternion space.
result Obtains state-of-the-art results on graph and node classification benchmarks.
C3 compresses images and videos with low complexity and high performance.
problem High complexity and low performance in neural compression models.
method Overfits a small model to each image or video separately, improving RD performance with low complexity.
result Matches the RD performance of state-of-the-art neural and video codecs with significantly lower decoding complexity.
Paper benchmarks quantum neural networks against classical ones for binary classification tasks.
problem Comparing quantum neural networks with classical ones for binary classification.
method Evaluated with two toy examples, focusing on model complexity and training data size.
result EQNN and QNN outperform ENN and DNN for smaller parameter sets and training data samples.
Benchmark evaluates financial misinformation detection models, revealing weaknesses without external context.
problem Detecting financial misinformation without external references.
method RFC Bench at paragraph level, two tasks: reference-free detection and comparison-based diagnosis.
result Performance improves with comparative context, revealing model weaknesses in reference-free settings.
OceanForecastBench offers a comprehensive benchmark for data-driven ocean forecasting models.
problem Lack of open-source, standardized benchmarks for data-driven ocean forecasting models.
method Proposes OceanForecastBench, a benchmark with high-quality data and evaluation pipeline.
result Offers the most comprehensive benchmarking framework for data-driven ocean forecasting.
SurvHTE-Bench benchmarks HTE estimation in survival analysis with diverse datasets.
problem Challenges in estimating HTEs from right-censored survival data.
method Modular synthetic datasets, semi-synthetic datasets, and real-world datasets.
result First rigorous comparison of survival HTE methods under diverse conditions.
Deep reinforcement learning is successful in decision making for sophisticated games, such as Atari, Go, etc. However, real-world decision making often requires reasoning with partial information extracted from complex visual observations. This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL…
New benchmark predicts cardiometabolic risk from accelerometer data, with varying accuracy.
problem Lack of accurate tabular benchmarks for cardiometabolic risk from accelerometer data.
method Tabular learning methods (ridge regression, XGBoost, TabPFN v2) applied to NHANES data.
result TabPFN v2 achieves best performance, but triglycerides remain largely unpredictable.
Sloth predicts LLM performance using latent skills across families.
problem Variations in benchmark performance due to differences in training configurations and data processing across model families.
method Sloth uses publicly available benchmark data and assumes LLM performance is driven by latent skills influenced by model size and training tokens. It exploits correlations across benchmarks to provide accurate predictions.
result Sloth predicts LLM performance accurately and offers insights into scaling behaviors for complex tasks.
This paper represents a preliminary (pre-reviewing) version of a sublinear variational algorithm for isotropic Gaussian mixture models (GMMs). Further developments of the algorithm for GMMs with diagonal covariance matrices (instead of isotropic clusters) and their corresponding benchmarking results have been published…
GNNs improve supply chain analytics with real-world benchmarks.
problem Limited research on applying GNNs to supply chain management.
method Conceptual discussions, detailed formulations, examples, mathematical definitions, and task guidelines.
result GNN-based models outperform other methods by 10-40% in various supply chain tasks.
A common belief in model-free reinforcement learning is that methods based on random search in the parameter space of policies exhibit significantly worse sample complexity than those that explore the space of actions. We dispel such beliefs by introducing a random search method for training static, linear policies for…
CTBench benchmarks cryptocurrency time series generation for trading applications.
problem Lack of comprehensive benchmarks for cryptocurrency time series generation.
method Developed a comprehensive benchmark extsf{CTBench} with 13 metrics across 5 dimensions.
result Uncovered trade-offs between statistical fidelity and real-world profitability.
BeGIN benchmarks GNNs for instance-dependent label noise in graphs.
problem Instance-dependent label noise in graph data.
method BeGIN introduces a benchmark with various noise types and evaluates noise-handling strategies across GNN architectures.
result Challenges of instance-dependent noise, especially LLM-based corruption, and the importance of node-specific parameterization.