New benchmark for non-rigid 3D human shape retrieval.
problem Distinguishing between body shapes of 3D human models.
method Extended benchmark with 145 new models and FAUST dataset.
result Improved comparison of 25 shape retrieval methods.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “human benchmark”
New benchmark for causal reasoning from human video descriptions.
problem Lack of diversity in event types and natural language descriptions, and differences from human judgments.
method Iterative event cloze task and data augmentation techniques.
result Improved data collection efficiency and diverse causal judgments.
Agent57 beats human average on all Atari games.
problem Atari game benchmark's poor performance in challenging games.
method Trained neural network with adaptive policy selection and novel architecture.
result First deep RL agent to outperform human average on all 57 Atari games.
UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).
AI benchmarks evaluate football team performance using generative models.
problem Evaluating human performance in complex interactive tasks is error-prone and unreliable.
method Trained Conditional VRNN Model on player and ball tracking data to imitate and predict team interactions.
result Trained model as a useful benchmark for evaluating team performance in football.
Benchmark for math reasoning models from human proofs.
problem Measuring and accelerating machine learning models in high-level mathematical reasoning.
method Built a non-synthetic dataset from theorem prover proofs, defined a task for model to fill in missing propositions, used hierarchical transformer to improve performance.
result Neural models can capture non-trivial mathematical reasoning, hierarchical transformer outperforms baseline.
Rapid intensification (RI) of tropical cyclones often causes major destruction to human civilization due to short response time. It is an important yet challenging task to accurately predict this kind of extreme weather event in advance. Traditionally, meteorologists tackle the task with human-driven feature extraction…
AutoML frameworks outperform human data scientists on 7 out of 12 OpenML tasks.
problem Evaluating if AutoML can outperform human data scientists.
method Comparison of four AutoML frameworks on 12 popular OpenML datasets (6 supervised classification, 6 supervised regression).
result AutoML frameworks perform better or equal to human data scientists in 7 out of 12 tasks.
Study compares DSPy teleprompter algorithms for aligning LLM evaluations with human annotations.
problem Aligning LLM evaluation metrics with human annotations.
method Comparative analysis of five teleprompter algorithms within the DSPy framework.
result Certain teleprompters outperform others in detecting hallucinations.
Human decision-making underlies all economic behavior. For the past four decades, human decision-making under uncertainty has continued to be explained by theoretical models based on prospect theory, a framework that was awarded the Nobel Prize in Economic Sciences. However, theoretical models of this kind have develop…
We present a novel human-aware navigation approach, where the robot learns to mimic humans to navigate safely in crowds. The presented model, referred to as DeepMoTIon, is trained with pedestrian surveillance data to predict human velocity in the environment. The robot processes LiDAR scans via the trained network to n…
SPIN converts weak LLMs to strong ones using self-play.
problem Growing strong LLMs without human-annotated data.
method Self-play fine-tuning (SPIN) starting from a supervised fine-tuned model.
result SPIN significantly improves LLM performance across benchmarks.
Dreem Open Datasets: Multi-Scored Sleep Datasets to compare Human and Automated sleep stagingq-bio.QM
Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a sin…
Machine learning workflow development is anecdotally regarded to be an iterative process of trial-and-error with humans-in-the-loop. However, we are not aware of quantitative evidence corroborating this popular belief. A quantitative characterization of iteration can serve as a benchmark for machine learning workflow d…
Unified framework to bridge human and LLM judgments.
problem Systematic discrepancies between human and LLM evaluations.
method Latent human preference score and linear transformations of covariates.
result Higher agreement with human ratings and exposure of systematic gaps.
New benchmark protocol evaluates neural network optimizers for efficiency and data shift sensitivity.
problem Benchmarking neural network optimizers with hyperparameter complexity and data shift sensitivity.
method Proposed a new evaluation protocol combining end-to-end and data-addition training efficiency, using bandit hyperparameter tuning and human study validation.
result No clear winner across all tasks, highlighting the complexity of optimizer performance.
Deep learning model classifies concurrent human interactions from WiFi data with high accuracy.
problem Classifying concurrent human interactions from WiFi data with high accuracy.
method Attention-BiGRU deep learning model using Multiple Input Multiple Output radio link.
result Maximum benchmark accuracy of 94% for a single subject-pair, 88% for ten subject pairs.
Introduces MLM dataset for multitask learning across multiple languages and modalities.
problem Training and evaluating multitask systems on diverse data.
method Develops a new benchmark dataset with semantic data and modalities.
result Demonstrates challenges in generalizing on diverse data.
For machine learning perception problems, human-level classification performance is used as an estimate of top algorithm performance. Thus, it is important to understand as precisely as possible the factors that impact human-level performance. Knowing this 1) provides a benchmark for model performance, 2) tells a proje…
New benchmarks for offline RL from diverse datasets.
problem Measuring progress in offline RL due to lack of suitable benchmarks.
method Developed benchmarks tailored for offline RL, focusing on diverse dataset properties.
result Revealed deficiencies in existing offline RL algorithms.
Transformer models improve financial sentiment measurement.
problem Capturing nuanced sentiment from financial news articles.
method Transformer-based language models for sentiment classification and aggregation.
result Transformer models outperform traditional dictionary-based methods in sentiment classification.
Direct Density Ratio Optimization aligns LLMs with human preferences without assuming specific models.
problem Statistical inconsistency in aligning LLMs with human preferences.
method Direct Density Ratio Optimization (DDRO) estimates density ratio directly.
result DDRO is statistically consistent, converging to true human preferences as data grows.
Study real-world noisy labels from human annotations for better understanding.
problem Understanding and modeling real-world label noise in machine learning.
method Developed two new benchmark datasets (CIFAR-10N, CIFAR-100N) with human-annotated real-world noisy labels.
result Real-world noisy labels exhibit instance-dependent patterns, not class-dependent as previously assumed.
New framework evaluates model explanations based on decision task improvement.
problem Evaluation of model explanations often misses practical value.
method Decision-theoretic framework quantifying three key values.
result Provides benchmarks and interprets human-AI decision support.
The paper benchmarks data stream classifiers for human activity recognition on connected devices.
problem Challenges in human activity recognition on connected devices, particularly high memory consumption and low F1 scores.
method Evaluation of five stream classification algorithms on real and synthetic datasets, measuring both performance and resource consumption.
result HT and MF classifiers show superior performance and resilience to concept drift compared to other algorithms.
The step of expert taxa recognition currently slows down the response time of many bioassessments. Shifting to quicker and cheaper state-of-the-art machine learning approaches is still met with expert scepticism towards the ability and logic of machines. In our study, we investigate both the differences in accuracy and…
Paper proposes a robust RLHF algorithm for LLMs, improving response preference over baselines.
problem Reward model misspecifications in RLHF for LLMs.
method Proposes a robust algorithm to reduce reward and policy estimator variance, theoretically and empirically validated.
result Consistently outperforms existing methods on LLM benchmark datasets, favoring 77-81% of responses over baselines.
EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statementsq-fin.ST
EDINET-Bench evaluates LLMs on complex financial tasks using Japanese financial statements.
problem Challenges in evaluating LLMs on financial tasks due to specialized expertise and scarce benchmarks.
method Developed EDINET-Bench, an open-source Japanese financial benchmark for LLMs on tasks like fraud detection and earnings forecasting.
result State-of-the-art LLMs perform only marginally better than logistic regression in financial tasks, highlighting the need for more realistic benchmarks.
Framework uses human feedback to safely set OOD detection thresholds, reducing false positives.
problem Challenges in setting OOD detection thresholds for safety-critical applications.
method Mathematically grounded framework leveraging expert feedback to dynamically update thresholds.
result Guaranteed to meet FPR constraint while minimizing human feedback, maintaining FPR at most 5%.
SVM with local features improves human action recognition.
problem Improving human action recognition in videos.
method Local appearance and motion features extracted using CNNs, concatenated, and used with SVM for classification.
result SVM with local features outperforms previous methods on benchmark datasets.
Large-scale public datasets have been shown to benefit research in multiple areas of modern artificial intelligence. For decision-making research that requires human data, high-quality datasets serve as important benchmarks to facilitate the development of new methods by providing a common reproducible standard. Many h…
New benchmark PVR tests neural network reasoning about indirection.
problem Understanding neural network generalization limits.
method Introducing Pointer Value Retrieval (PVR) benchmark.
result Large variations in performance across different conditions.
BREEDS benchmarks assess model robustness to subpopulation shifts.
problem Measuring model robustness to novel subpopulation shifts.
method Controlled synthesis of realistic distribution shifts using class structure.
result Validated model sensitivity and effectiveness of robustness interventions.
Human visual object recognition is typically rapid and seemingly effortless, as well as largely independent of viewpoint and object orientation. Until very recently, animate visual systems were the only ones capable of this remarkable computational feat. This has changed with the rise of a class of computer vision algo…
In this paper, we propose a gamification approach as a novel framework for smart building infrastructure with the goal of motivating human occupants to reconsider personal energy usage and to have positive effects on their environment. Human interaction in the context of cyber-physical systems is a core component and c…
We provide an online RLHF workflow for large language models.
problem Lack of online RLHF workflow for large language models.
method Construct preference models using open-source datasets, approximate human feedback, and implement online iterative RLHF.
result Our LLM achieves state-of-the-art performance on various benchmarks.
This paper proposes a method to use LLMs as auxiliary evaluators in place of human judges.
problem The need for cost-effective and scalable evaluation of AI systems.
method Formulates a two-stage sampling design with LLM evaluations and human ratings, using a doubly robust estimator.
result Proposes a method to determine optimal sample sizes for human and LLM ratings.
New feature representation improves HAR model accuracy.
problem Improving human activity recognition model performance.
method Proposed new feature representation and comparison with existing methods.
result Techniques based on the proposed representation outperform baselines.
Online symptom checkers have significant potential to improve patient care, however their reliability and accuracy remain variable. We hypothesised that an artificial intelligence (AI) powered triage and diagnostic system would compare favourably with human doctors with respect to triage and diagnostic accuracy. We per…
Detecting anomalous activity in human mobility data has a number of applications including road hazard sensing, telematic based insurance, and fraud detection in taxi services and ride sharing. In this paper we address two challenges that arise in the study of anomalous human trajectories: 1) a lack of ground truth dat…
Paper uses AI to improve medical diagnosis accuracy.
problem Improving accuracy of medical diagnoses.
method Heuristic frequentist and Bayesian approaches applied to a nationwide dataset.
result Algorithm outperforms human doctors in detecting abnormal births.
When we test a theory using data, it is common to focus on correctness: do the predictions of the theory match what we see in the data? But we also care about completeness: how much of the predictable variation in the data is captured by the theory? This question is difficult to answer, because in general we do not kno…
This thesis tackles learning reward functions from human comparative feedback.
problem Designing reward functions for complex tasks is challenging and humans often provide suboptimal demonstrations.
method Proposes learning reward functions from comparative feedback (pairwise comparisons, best-of-many choices, rankings, scaled comparisons) and active learning techniques.
result Demonstrates the effectiveness of learning reward functions from comparative feedback in various domains.
Deep neural networks solve Raven's Progressive Matrices with high accuracy.
problem Testing relational reasoning in machine learning systems.
method Combining Wild Relation Networks with Multi-Layer Relation Networks and introducing Magnitude Encoding.
result Deep neural networks achieve 98.0 percent accuracy, significantly improving over previous methods.
MallowsPO enhances LLM fine-tuning with a dispersion index of human preferences.
problem Lack of diversity in human preferences in DPO.
method Developed a dispersion index based on Mallows' theory to characterize preference diversity.
result Demonstrated improved performance in various tasks using the dispersion index.
Benchmark tests spoken language models for infant language learning.
problem Understanding how infants learn language from speech.
method Developed a language-acquisition-friendly benchmark.
result Benchmarking shows models' strengths and weaknesses.
Variational Proximal Policy Optimization improves reinforcement learning from human feedback.
problem Policy mode collapse and brittle exploration loops in reinforcement learning.
method Particle-based variational inference framework with Mixture-of-Experts architecture.
result Significant improvements in complex reasoning benchmarks.
AI-Trader benchmarks LLMs in live financial markets, revealing poor trading performance.
problem Challenges in real-time financial decision-making by autonomous agents.
method Fully automated, live evaluation benchmark with minimal human intervention.
result General intelligence does not translate to effective trading, highlighting limitations.