The paper introduces new metrics for evaluating generative models of behavior.
problem Lack of quantitative evaluation criteria for unsupervised behavior discovery.
method Proposed and investigated several metrics for generative models of behavior.
result The proposed metrics correspond with biologists' intuitions and allow for model evaluation and bias understanding.
We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account…
Introduces LoCA regret to evaluate model-based RL methods.
problem Lack of consistent metrics to evaluate model-based RL methods.
method Inspired by neuroscience, introduces LoCA regret to measure model-based behavior.
result LoCA regret can identify model-based behavior and assess how close methods are to optimal model-based behavior.
AIPS improves ranking policy evaluation by adapting to diverse user behavior.
problem Inaccurate Off-Policy Evaluation of ranking policies due to high variance under diverse user behavior.
method Developed Adaptive IPS (AIPS) that adapts to different user behaviors and minimizes MSE.
result AIPS achieves minimum variance among unbiased estimators and provides significant empirical accuracy improvement.
New framework evaluates model explanations based on decision task improvement.
problem Evaluation of model explanations often misses practical value.
method Decision-theoretic framework quantifying three key values.
result Provides benchmarks and interprets human-AI decision support.
Study evaluates five LLMs for financial report analysis, revealing performance differences and variability.
problem Lack of understanding in reliability, consistency, and transparency of LLMs in financial analysis.
method Human evaluation, automated similarity metrics, and behavioral diagnostics applied to five transformer-based LLMs over U.S. 10-K filings.
result No single LLM consistently dominates across all evaluation perspectives, highlighting variability and need for interpretability.
The paper explains why estimating a history-dependent policy can reduce MSE in reinforcement learning.
problem Understanding why history-dependent policies can improve MSE in off-policy evaluation.
method The paper derives a bias-variance decomposition of MSE for various OPE estimators, showing how history-dependent policies can decrease variance and increase bias.
result History-dependent policies can decrease the variance of importance sampling estimators, leading to lower MSE.
Study minimax off-policy evaluation in multi-armed bandits with known and unknown behavior policies.
problem Evaluate policies in multi-armed bandits with unknown behavior policies.
method Develop minimax rate-optimal procedures for known and unknown behavior policies, including the Switch estimator and Chebyshev polynomial-based estimator.
result Plug-in estimator achieves optimal competitive ratio up to a logarithmic factor when behavior policy is unknown.
A test measures artificial agents' human-like behavior in video games.
problem Measuring the believability of artificial agents' human-like behavior.
method Developed a non-parametric two-sample hypothesis test.
result The p-value correlates with human judgment of human-like behavior. The paper shows how ignoring temporal context in recommender systems evaluation leads to false confidence, proposing a method to embed temporal context.
problem The discrepancy between offline and online recommender system performance evaluation.
method Proposes a training procedure to embed temporal context into recommender systems and validates its advantage using multi-objective optimization.
result Including temporal context in recommender systems evaluation can improve recall@20 by up to 20%.
New estimator GMIPS reduces variance in ranking policy evaluation.
problem High variance in off-policy evaluation for ranking policies.
method GMIPS estimator with user behavior model on ranking embedding spaces.
result GMIPS achieves lowest MSE and balances bias-variance trade-off.
New method evaluates AI stock prediction systems based on decision-making processes.
problem Lack of evaluation for AI systems' decision-making processes.
method Scores intermediate decision process using large language models and closed-loop reinforcement learning feedback.
result Composite behavioral score correlates with Sharpe ratio and reduces prediction error.
MLDS dataset reveals hidden model behavior via weight-space analysis.
problem Neural networks' opacity makes them hard to evaluate.
method Presented MLDS dataset of trained neural networks.
result Weight-space analysis reveals meaningful divergence with small changes in training data.
Evaluation of deep reinforcement learning (RL) is inherently challenging. In particular, learned policies are largely opaque, and hypotheses about the behavior of deep RL agents are difficult to test in black-box environments. Considerable effort has gone into addressing opacity, but almost no effort has been devoted t…
Study improves off-policy evaluation from non-i.i.d. bandit samples.
problem Improving off-policy evaluation from non-independent bandit samples.
method Constructing an estimator from a standardized martingale difference sequence.
result Proposed estimator performs better than existing methods.
ALMANACS benchmarks explainability methods on simulatability.
problem Evaluating the effectiveness of explainability methods for language models.
method ALMANACS is a simulatability benchmark that evaluates explainability methods on twelve safety-relevant topics.
result No explainability method outperforms the explanation-free control across all topics.
Social media systems rely on user feedback and rating mechanisms for personalization, ranking, and content filtering. However, when users evaluate content contributed by fellow users (e.g., by liking a post or voting on a comment), these evaluations create complex social feedback effects. This paper investigates how ra…
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
CoinDICE estimates confidence intervals for unknown behavior policies in reinforcement learning.
problem Estimating value of a target policy using only behavior policy data.
method Function space embedding, generalized empirical likelihood method, Lagrangian optimization.
result Valid confidence intervals with tighter and more accurate estimates than existing methods.
FLARKO uses LLMs, KGs, and KTO to generate profitable, behaviorally aligned financial recommendations.
problem Financial recommendation systems often fail to account for behavioral and regulatory factors.
method FLARKO integrates LLMs, KGs, and KTO to generate profitable and behaviorally aligned recommendations.
result FLARKO consistently outperforms state-of-the-art recommendation baselines on behavioral alignment and joint profitability.
Strategy evaluation schemes are a crucial factor in any agent-based market model, as they determine the agents' strategy preferences and consequently their behavioral pattern. This study investigates how the strategy evaluation schemes adopted by agents affect their performance in conjunction with the market circumstan…
Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showc…
Unified DICE estimators as regularized Lagrangians for improved off-policy evaluation.
problem Improving off-policy evaluation from behavior-agnostic data.
method Unified derivation of DICE estimators as regularized Lagrangians of a linear program.
result Dual solutions offer greater flexibility and provide superior estimates in practice.
The paper analyzes the asymptotic behavior of a knot polynomial for a specific real number.
problem Understanding the asymptotic behavior of a knot polynomial for a real number.
method Examining the asymptotic behavior of the N-dimensional colored Jones polynomial evaluated at exp(ξ/N) for a real number ξ. result From the asymptotic behavior, the mSL(2;C) Chern--Simons invariant and the Reidemeister torsion twisted by the adjoint action can be extracted. This paper demonstrates the use of genetic algorithms for evolving a grandmaster-level evaluation function for a chess program. This is achieved by combining supervised and unsupervised learning. In the supervised learning phase the organisms are evolved to mimic the behavior of human grandmasters, and in the unsupervi…
GICDM corrects hubness in embedding spaces for better generative model evaluation.
problem Hubness phenomenon distorts distances in high-dimensional embedding spaces.
method Generative ICDM (GICDM) using multi-scale extension to correct neighborhood estimation.
result GICDM resolves hubness-induced failures and improves metric behavior.
Bayesian approach quantifies uncertainty in LLM evaluations.
problem Statistical uncertainty in evaluating LLM behavior.
method Bayesian evaluation of LLM behavior using probabilistic text generation strategies.
result Bayesian approach provides useful uncertainty quantification about LLM behavior.
AXE evaluates explanations to avoid misleading Rashomon set model selection.
problem Evaluating explanations for Rashomon set models to avoid false selection.
method Proposed AXE method to evaluate explanation quality.
result AXE detects adversarial fairwashing with 100% success rate.
Investigates the use of Information Coefficient as a stock selection model performance measure.
problem The adequacy and effectiveness of Information Coefficient (IC) for evaluating stock selection models is unclear.
method Simulation and simple statistical modeling to examine IC behavior statically and dynamically.
result Proposes two practical procedures for IC-based ongoing performance monitoring of stock selection models.
We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy …
Peer effects, in which the behavior of an individual is affected by the behavior of their peers, are posited by multiple theories in the social sciences. Other processes can also produce behaviors that are correlated in networks and groups, thereby generating debate about the credibility of observational (i.e. nonexper…
We propose a novel approach to train a multi-modal policy from mixed demonstrations without their behavior labels. We develop a method to discover the latent factors of variation in the demonstrations. Specifically, our method is based on the variational autoencoder with a categorical latent variable. The encoder infer…
Improved probabilistic forecasts using behavioral transformations.
problem Improving accuracy and consistency of probabilistic asset price forecasts.
method Behavioral transformation of fundamental expectations to disentangle sentiment-induced biases.
result Substantial forecast gains across various models and risk-preferences.
Logit models are usually applied when studying individual travel behavior, i.e., to predict travel mode choice and to gain behavioral insights on traveler preferences. Recently, some studies have applied machine learning to model travel mode choice and reported higher out-of-sample predictive accuracy than traditional …
We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. To address this challenge, we propose an algorithm that safely and …
This paper evaluates knowledge graph completion models under the open-world assumption, revealing unexpected behavior of metrics.
problem Evaluation of knowledge graph completion models often assumes a closed-world assumption, which can lead to misleading results.
method The paper studies KGC evaluation under the open-world assumption, analyzing the behavior of metrics like MRR and Hits@K.
result Metrics like MRR and Hits@K can show significant degradation under the open-world assumption, leading to incorrect model comparisons.
Despite the growing importance of multilingual aspect of web search, no appropriate offline metrics to evaluate its quality are proposed so far. At the same time, personal language preferences can be regarded as intents of a query. This approach translates the multilingual search problem into a particular task of searc…
New framework analyzes pre-stock jump trading behaviors using multivariate time series analysis.
problem Understanding micro-trading behaviors before stock price jumps.
method Multivariate time series analysis considering temporal information.
result Identifies highly informative attributes for predicting price jumps.
In this paper, we propose a new feature extraction technique for program execution logs. First, we automatically extract complex patterns from a program's behavior graph. Then, we embed these patterns into a continuous space by training an autoencoder. We evaluate the proposed features on a real-world malicious softwar…
COHORTNEY groups web users based on activity patterns.
problem Lack of academic discussion on cohort analysis for user behavior.
method Unsupervised non-parametric machine learning approach.
result COHORTNEY outperforms traditional methods in cohort analysis.
Study shows quantum modularity in figure-eight knot's colored Jones polynomial.
problem Asymptotic behavior of colored Jones polynomial of figure-eight knot.
method Analyzing polynomial evaluated at specific points and showing asymptotic equivalence.
result Quantum modularity demonstrated in the figure-eight knot's colored Jones polynomial.
Study reveals LLM personality patterns but lacks behavioral consistency.
problem Understanding and validating personality traits in LLMs.
method Characterized LLM personality across three dimensions: training dynamics, self-report validity, and intervention effects.
result Self-reported traits do not reliably predict behavior, and instructional alignment affects trait expression but not behavior.
BEACON optimizes discovery by efficiently finding novel behaviors.
problem Discovering diverse system behaviors without a scalar objective.
method Bayesian optimization inspired strategy using multi-output Gaussian processes.
result BEACON discovers broader sets of distinct behaviors than competing methods.
Study evaluates clustering methods for Google Trends data.
problem Clustering high-dimensional, noisy time series data.
method Symbolic Aggregate Approximation (SAX), Enhanced SAX (eSAX), and Topological Data Analysis (TDA).
result TDA provides more balanced and meaningful groupings than SAX and eSAX.
VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.
problem Estimating value function of a target policy from offline data collected by a behavior policy.
method Proposes VA-OPE, an algorithm that reweights Bellman residual using estimated variance of the value function.
result Achieves a tighter error bound than the best-known result.
We present a practical health-theme machine learning (ML) application concerning `AI for social good' domain for `Producing Good Outcomes' track. In particular, the solution is concerning the problem of a potential elderly adult dementia onset prediction in aging societies. The paper discusses our attempt and encouragi…
Study examines if LLMs' trading styles match real market behavior.
problem Lack of behavioral consistency in LLMs' trading strategies.
method Year-long simulations with LLMs, operationalizing behavioral finance drivers, and comparing with financial theory.
result LLMs' strategy switching is only partially consistent with behavioral finance theories.
Counterfactual policy evaluation improves autonomous driving policies' generalization.
problem Learnt policies often fail to generalize and handle novel situations.
method Introduces counterfactual policy evaluation using counterfactual worlds.
result Significantly decreases collision-rate while maintaining high success-rate.