Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

113227340453 · Jun 202019922001200920172026
48 results for user evaluation

AIPS improves ranking policy evaluation by adapting to diverse user behavior.

problem Inaccurate Off-Policy Evaluation of ranking policies due to high variance under diverse user behavior.
method Developed Adaptive IPS (AIPS) that adapts to different user behaviors and minimizes MSE.
result AIPS achieves minimum variance among unbiased estimators and provides significant empirical accuracy improvement.

Recommendation systems have been integrated into the majority of large online systems. They tailor those systems to individual users by filtering and ranking information according to user profiles. This adaptation process influences the way users interact with the system and, as a consequence, increases the difficulty …

2014-07-03abs ↗pdf ↗

Many successful applications of computer vision to image or video manipulation are interactive by nature. However, parameters of such systems are often trained neglecting the user. Traditionally, interactive systems have been treated in the same manner as their fully automatic counterparts. Their performance is evaluat…

2009-12-13abs ↗pdf ↗

UCFE benchmarks LLMs in financial tasks with human feedback.

problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).

Social media systems rely on user feedback and rating mechanisms for personalization, ranking, and content filtering. However, when users evaluate content contributed by fellow users (e.g., by liking a post or voting on a comment), these evaluations create complex social feedback effects. This paper investigates how ra…

2014-05-06abs ↗pdf ↗

Federated learning is a distributed, on-device computation framework that enables training global models without exporting sensitive user data to servers. In this work, we describe methods to extend the federation framework to evaluate strategies for personalization of global models. We present tools to analyze the eff…

2019-10-22abs ↗pdf ↗

We quantify content availability and user discovery opportunities in recommender systems.

problem Determining the maximum probability of recommending content to users.
method Stochastic reachability to compute upper bounds on recommendation likelihood.
result Reachability metrics can detect biases and diagnose user discovery limitations.

GUIDE-VAE generates user-guided data with improved realism and performance.

problem Generating data points for multi-user datasets while considering user information.
method Conditional generative model that integrates user embeddings and a pattern dictionary-based covariance composition.
result GUIDE-VAE outperforms conventional VAEs in multi-user settings, especially under data imbalance.

Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical…

2015-06-12abs ↗pdf ↗

Estimates users' preference for a site over others using engagement data.

problem Lack of data on users' interactions with other sites makes it hard to estimate preferences for a focal site.
method Uses Hierarchical Bayes Method with two estimation techniques: Markov Chain Monte Carlo and Stochastic Gradient with Langevin Dynamics.
result Good support found for the approach to computing personalized share of engagement.

New method adapts to user preferences dynamically, improving recommendation models.

problem Current recommendation models lack dynamic adaptation to changing user preferences.
method Preference Discerning with LLM-Enhanced Generative Retrieval
result Mender achieves state-of-the-art performance in adapting to evolving user preferences.

Today, large amounts of valuable data are distributed among millions of user-held devices, such as personal computers, phones, or Internet-of-things devices. Many companies collect such data with the goal of using it for training machine learning models allowing them to improve their services. User-held data is, howeve…

2019-07-08abs ↗pdf ↗

Defense against user shilling attacks in collaborative filtering using edge reweighting.

problem Vulnerability of collaborative filtering to profile injection attacks.
method Adversarial robustness based edge reweighting to attenuate non-robust edges.
result Effective defense against various types of attacks demonstrated through experiments.

The evolution of social media users' behavior over time complicates user-level comparison tasks such as verification, classification, clustering, and ranking. As a result, naïve approaches may fail to generalize to new users or even to future observations of previously known users. In this paper, we propose a novel pro…

2019-10-11abs ↗pdf ↗

Paper proposes a two-stage ranking for personalized TV recommendations.

problem Improving TV recommendation accuracy and efficiency.
method First, identifies potential candidates using user viewing patterns. Then, ranks them based on user preferences and program textual information.
result The proposed model outperforms in recommendation accuracy and efficiency.

Machine-generated interpretations do not improve users' guessing accuracy in image classifiers.

problem Determining the usefulness of machine-generated explanations for deep neural networks.
method Human evaluation of crowd workers guessing incorrectly predicted labels with and without visual interpretations.
result Showing machine-generated visual interpretations decreased average guessing accuracy by about 10%.

New model improves recommendation systems by analyzing user-item interactions.

problem Improving recommendation systems for better user-item interactions.
method Sliced Anti-symmetric Decomposition (SAD) model using tensor decomposition.
result SAD produces the most consistent personalized preferences compared to SOTA models.

The outcome of the explorative data analysis (EDA) phase is vital for successful data analysis. EDA is more effective when the user interacts with the system used to carry out the exploration. In the recently proposed paradigm of iterative data mining the user controls the exploration by inputting knowledge in the form…

2018-04-09abs ↗pdf ↗

We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. To address this challenge, we propose an algorithm that safely and …

2019-12-05abs ↗pdf ↗

Suitability filter detects model performance degradation in real-world deployment.

problem Ensuring model reliability in safety-critical domains without access to ground truth labels.
method Uses suitability signals to evaluate classifier performance on unlabeled user data.
result The suitability filter reliably detects performance deviations due to covariate shift.

Critical task and cognition-based environments, such as in military and defense operations, aviation user-technology interaction evaluation on UI, understanding intuitiveness of a hardware model or software toolkit, etc. require an assessment of how much a particular task is generating mental workload on a user. This i…

2019-11-07abs ↗pdf ↗

A new framework promotes trustworthy user-generated datasets by ensuring no user benefits from misreporting.

problem Incentivizing data misreporting in user-generated datasets.
method Proposes Licchavi, a global and personalized learning framework with provable strategyproofness guarantees.
result Proves that no user can gain much by replying to Licchavi's queries with deviated answers.

The need for diversification of recommendation lists manifests in a number of recommender systems use cases. However, an increase in diversity may undermine the utility of the recommendations, as relevant items in the list may be replaced by more diverse ones. In this work we propose a novel method for maximizing the u…

2014-11-13abs ↗pdf ↗

Hybrid approach combines user feedback and machine learning for predicting user satisfaction.

problem Measuring user satisfaction in large-scale conversational agent systems.
method Fusion of explicit user feedback and predictions from two machine-learned models trained on different data types.
result Hybrid approach significantly improves user satisfaction predictions.

Transparency, user trust, and human comprehension are popular ethical motivations for interpretable machine learning. In support of these goals, researchers evaluate model explanation performance using humans and real world applications. This alone presents a challenge in many areas of artificial intelligence. In this …

2017-11-20abs ↗pdf ↗

Model user preferences for conversational LLMs using weak rewards.

problem Lack of persistent user models in conversational LLMs leading to repeated user restatements.
method Vector-Adapted Retrieval Scoring (VARS) framework that updates user vectors online from weak scalar rewards.
result Full VARS agent achieves strongest overall performance, matches strong Reflection baseline in task success, and reduces user effort.

As more and more people shift their movie watching online, competition between movie viewing websites are getting more and more intense. Therefore, it has become incredibly important to accurately predict a given user's watching list to maximize the chances of keeping the user on the platform. Recent studies have sugge…

2019-09-02abs ↗pdf ↗