How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity b…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,786 papers · 148 categories
Trend · papers per month
48 results for “quality evaluation”
Do-AIQ: A Design-of-Experiment Approach to Quality Evaluation of AI Mislabel Detection Algorithmstat.ML
Do-AIQ framework evaluates AI algorithms' quality using DOE.
problem Quality evaluation of AI mislabel detection algorithms.
method Design-of-experiment approach with high-dimensional constraint space design and surrogate modeling.
result Established framework for evaluating AI algorithm quality robustly.
Automatically evaluates image quality based on human judgment.
problem Difficulty in rigorously evaluating generated image quality.
method Generative model embeddings, human labels regression, and statistical matching.
result 66% accuracy in predicting human scores of image realism.
Framework evaluates quality of synthetic data generated with differential privacy.
problem Ensuring synthetic data retains statistical quality after applying differential privacy.
method Developed a framework to evaluate synthetic data quality from a practical researcher's viewpoint.
result Synthetic data can be evaluated against training data or underlying populations, and for specific tasks like inference or prediction.
Paper explores how text generation quality and diversity metrics relate to distribution fitting.
problem Unclear relation between text generation quality and diversity metrics and distribution fitting.
method Theoretical approach to prove a linear combination of quality and diversity metrics can be a divergence metric.
result CR/NRR proposed as a better substitute for BLEU/Self-BLEU metrics.
Framework improves ML performance by identifying high-quality data.
problem Poor data quality hampers ML performance.
method Intelligent data-centric evaluation framework combining quality measurements and unsupervised learning.
result Framework improves ML system performance in real-world use case.
Evaluating the Effect of Credit Collection Policy on Portfolio Quality of Micro-Finance Bankq-fin.PM
Study finds strict collection policies improve portfolio quality of microfinance banks.
problem Improving portfolio quality of microfinance banks through better credit collection policies.
method Multi-stage sampling, regression analysis, descriptive statistics.
result Collection policy has a higher effect on portfolio quality.
Protein structure prediction has been a grand challenge problem in the structure biology over the last few decades. Protein quality assessment plays a very important role in protein structure prediction. In the paper, we propose a new protein quality assessment method which can predict both local and global quality of …
Generative models predict page quality without training, useful for low-resource settings.
problem Detecting low-quality content in web articles.
method Human evaluation and analysis of 500 million web articles.
result Generative models can predict page quality without training, useful for low-resource settings.
Proposes GM Score to evaluate GANs considering diversity, disentanglement, and discriminability.
problem Evaluation of GANs for sample quality and diversity.
method Integrates various factors including intra-class and inter-class diversity, disentanglement, and discriminability metrics.
result Demonstrates improved evaluation of GANs on MNIST dataset.
Automated dialogue quality evaluation using user satisfaction estimates across multiple domains.
problem Lack of automated and domain-independent dialogue quality evaluation metrics.
method Created a new Response Quality annotation scheme, introduced five domain-independent feature sets, and experimented with six machine learning models.
result Gradient Boosting Regression model achieved best prediction performance, with a 16% relative improvement in binary satisfaction class prediction accuracy.
Study evaluates quality of uncertainty estimates for neural networks.
problem Lack of principled assessment methods for evaluating uncertainty quality in deep learning.
method Statistical methods of frequentist interval coverage, interval width, and expected calibration error.
result Different UQ methods produce markedly different quality uncertainty estimates.
This research evaluates the quality of unsupervised embeddings using linear separability metrics.
problem Difficulty in evaluating unsupervised learning model performance in practice.
method Survey and introduction of three methods, including a novel one, to assess embedding quality.
result Metrics can robustly estimate embedding quality in an unsupervised way.
A framework assesses the quality of crowdsourced weather data.
problem Quality control and assessment of crowdsourced weather data from third-party stations.
method Proposes a simple, scalable, and interpretable AI/Stats/ML framework to assess TPAWS data.
result Demonstrates the performance of the framework using synthetic and real data.
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improv…
Enhanced metrics improve generative model evaluation reliability.
problem Lack of reliable quality metrics for generative models.
method Introduce Clipped Density and Clipped Coverage metrics.
result Metrics prevent out-of-distribution samples from biasing quality scores.
It Takes Two to Tango: Two Parallel Samplers Improve Quality in Diffusion Models for Limited Stepscs.CV
Two parallel samplers enhance image quality in limited denoising steps.
problem Limited denoising steps in diffusion models reduce image quality.
method Two parallel samplers denoise at successive times, integrating their information.
result Two parallel samplers improve image quality compared to a single sampler.
Text generation is an important Natural Language Processing task with various applications. Although several metrics have already been introduced to evaluate the text generation methods, each of them has its own shortcomings. The most widely used metrics such as BLEU only consider the quality of generated sentences and…
Stabilizes training of DNN for speech enhancement using PESQ scores.
problem Stability issues in training DNNs using non-differentiable OSQA scores.
method Approximate OSQA scores with a differentiable auxiliary DNN and stabilize training with reinforcement learning techniques.
result Stable training of DNN to achieve state-of-the-art PESQ scores and better sound quality.
The paper proposes using density ratio estimation to evaluate synthetic data quality.
problem Improving the quality and utility of synthetic data for analysis.
method Density ratio estimation to measure synthetic data quality.
result Density ratio estimation yields more accurate global utility estimates than existing methods.
With super-resolution optical microscopy, it is now possible to observe molecular interactions in living cells. The obtained images have a very high spatial precision but their overall quality can vary a lot depending on the structure of interest and the imaging parameters. Moreover, evaluating this quality is often di…
The paper introduces a method to assess machine translation quality with confidence intervals.
problem Evaluating the uncertainty and quality of machine translation.
method Utilizes conformal predictive distributions to produce prediction intervals with guaranteed coverage.
result The method outperforms a baseline on six language pairs in terms of coverage and sharpness.
Determining the quality of the results obtained by clustering techniques is a key issue in unsupervised machine learning. Many authors have discussed the desirable features of good clustering algorithms. However, Jon Kleinberg established an impossibility theorem for clustering. As a consequence, a wealth of studies ha…
The paper proposes a new method to evaluate LLM agent responses using ECDF clustering.
problem The standard evaluation of LLM agent responses via majority voting obscures response quality and distribution.
method The paper introduces a novel evaluation framework based on ECDF of cosine similarities and clustering of ECDFs using distances and -medoids algorithm.
result ECDF clustering reveals interpretable group structures in LLM responses, offering insights into agent settings.
New methods correct bias in LLM-as-a-Judge evaluations, but reliability depends on judge quality and model calibration.
problem Systematic bias in LLM-as-a-Judge evaluations using naive estimators.
method Analytical results, simulations, and real-data case study to diagnose reliability of corrected estimates.
result Corrected estimates, especially shared-calibration comparisons, can be unreliable under certain conditions.
FinSphere improves stock analysis quality with AI and expert-curated data.
problem Lack of objective evaluation metrics and depth in stock analysis by FinLLMs.
method Developed AnalyScore, curated Stocksis dataset, and FinSphere AI agent.
result FinSphere outperforms general and domain-specific LLMs in generating high-quality stock analysis reports.
Statistical and economic evaluation of forecasts in electricity markets: beyond RMSE and MAEq-fin.CP
This paper evaluates forecast quality in electricity markets beyond traditional accuracy measures.
problem Traditional accuracy measures fail to reflect the economic value of electricity price forecasts.
method Investigates four quality dimensions: accuracy, dispersion, association, and extremum identification.
result Dispersion- and association-based measures better capture forecast economic value.
Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routingcs.LG
Meta-Router optimizes LLM selection using gold-standard and preference-based data.
problem Training a high-quality LLM router with combined data sources is challenging due to bias and scarcity.
method Developed an integrative causal router training framework to correct bias and improve routing accuracy.
result Our approach delivers more accurate routing and improves the trade-off between cost and quality.
Callisto: Entropy based test generation and data quality assessment for Machine Learning Systemscs.LG
CALLISTO generates tests and assesses ML data quality using prediction entropy.
problem Validating ML systems for accuracy and data quality.
method Entropy-based test generation and data quality assessment framework.
result CALLISTO detects up to 20x more errors than traditional methods.
Generative Adversarial Networks (GANs) are an elegant mechanism for data generation. However, a key challenge when using GANs is how to best measure their ability to generate realistic data. In this paper, we demonstrate that an intrinsic dimensional characterization of the data space learned by a GAN model leads to an…
Spectral Overlap and a Comparison of Parameter-Free, Dimensionality Reduction Quality Metricsstat.ML
Nonlinear dimensionality reduction methods are a popular tool for data scientists and researchers to visualize complex, high dimensional data. However, while these methods continue to improve and grow in number, it is often difficult to evaluate the quality of a visualization due to a variety of factors such as lack of…
Study evaluates margin parameter effects on knowledge embedding quality.
problem Understanding margin parameter's impact on embedding quality.
method Examined margin parameter values for multi-relational categorized data.
result Lower margin values are insufficient, while larger values cause noise.
Enhances clustering quality evaluation in noisy data.
problem Reliable clustering quality assessment in noisy Gaussian mixtures.
method Feature Importance Rescaling (FIR) method.
result FIR improves correlation between cluster validity indices and ground truth.
Spinal cord stimulation has enabled humans with motor complete spinal cord injury (SCI) to independently stand and recover some lost autonomic function. Quantifying the quality of bipedal standing under spinal stimulation is important for spinal rehabilitation therapies and for new strategies that seek to combine spina…
Novel Orlicz regrets consistently bound environmental variable statistics.
problem Consistent evaluation of stochastic environmental variables like water quality indices.
method Proposed novel Orlicz regrets for upper and lower bounds.
result Explicit linkage between Orlicz regrets and divergence risk measures.
The paper improves QD policy ensembles using distribution ratio estimators.
problem Training diverse and high-quality reinforcement learning agents.
method Using Stein variational gradient descent and distribution ratio estimators.
result The method generates diverse and high-quality reinforcement learning agents.
The paper assesses quality measures for machine learning models using cross-validation.
problem Evaluating the accuracy and robustness of quality measures for machine learning models.
method Cross-validation approach to estimate prediction error and quantify explained variation. Confidence bounds and local quality measures derived from residuals.
result The reliability and robustness of quality measures are assessed through numerical examples and confidence bounds.
The study analyzes how probabilistic forecasts improve battery trading strategies in electricity markets.
problem Improvements in statistical forecast quality do not directly translate to economic value in battery trading strategies.
method The study frames battery optimization as a stochastic program based on fully probabilistic forecasts and examines decision quality under different uncertainty models.
result The study identifies two critical flaws in quantile-based trading strategies and provides theoretical justification and empirical evidence.
Tree Index evaluates cluster quality by creating decision trees from data.
problem Evaluating the quality of cluster results from various techniques.
method Tree Index creates a decision tree from clustered data, combining entropy and depth of leaves.
result Tree Index discriminates between sensible and non-sensible clusters on brain dataset.
This paper uses LLMs and cycle consistency for better machine translation evaluation.
problem Evaluating translation quality and LLM capabilities without ground truth.
method Generate translation candidates, back-translate, and evaluate cycle consistency.
result Larger LLMs or more inference passes improve cycle consistency.
We focus our attention on the link prediction problem for knowledge graphs, which is treated herein as a binary classification task on neural embeddings of the entities. By comparing, combining and extending different methodologies for link prediction on graph-based data coming from different domains, we formalize a un…
An automated metric to evaluate dialogue quality is vital for optimizing data driven dialogue management. The common approach of relying on explicit user feedback during a conversation is intrusive and sparse. Current models to estimate user satisfaction use limited feature sets and rely on annotation schemes with low …
Experimental evaluation is a major research methodology for investigating clustering algorithms and many other machine learning algorithms. For this purpose, a number of benchmark datasets have been widely used in the literature and their quality plays a key role on the value of the research work. However, in most of t…
DAISYnt evaluates synthetic data quality and privacy in regulated domains.
problem Balancing data quality and privacy in regulated domains.
method Developed a suite of advanced tests (DAISYnt) to evaluate synthetic data quality and privacy.
result DAISYnt sets a de facto standard for synthetic data evaluation in regulated domains.
VLM judges rank well but score poorly; task difficulty and annotation quality affect interval width.
problem VLMs as judges lack reliability indicators in multimodal evaluations.
method Conformal prediction using score-token log-probabilities.
result Evaluation uncertainty is task-dependent, affecting interval width and reliability.
A new first-order sampler improves diffusion probabilistic model sampling quality.
problem The belief that first-order methods are inherently slower for diffusion probabilistic model sampling.
method A novel training-free, first-order sampler that approximates the forward-value evaluation via a one-step lookahead predictor.
result The proposed sampler provably approximates the ideal forward-value trajectory while retaining first-order convergence and can improve sample quality under the same NFE budget.
Comparison and Adaptation of Automatic Evaluation Metrics for Quality Assessment of Re-Speakingcs.CL
Re-speaking is a mechanism for obtaining high quality subtitles for use in live broadcast and other public events. Because it relies on humans performing the actual re-speaking, the task of estimating the quality of the results is non-trivial. Most organisations rely on humans to perform the actual quality assessment, …
In this work we study an economic agent based model under different asymmetric information degrees. This model is quite simple and can be treated analytically since the buyers evaluate the quality of a certain good taking into account only the quality of the last good purchased plus her perceptive capacity β. As a cons…