Unified evaluation for both quality and diversity in NLP.
problem Measuring both quality and diversity in NLP models.
method Proposes HUSE, a metric combining human and statistical evaluation.
result HUSE detects both quality and diversity defects in NLP models.
New methods for evaluating clustering quality.
problem Evaluating unsupervised clustering results.
method Various techniques to evaluate clustering quality.
result No perfect clustering evaluation metric exists.
Do-AIQ framework evaluates AI algorithms' quality using DOE.
problem Quality evaluation of AI mislabel detection algorithms.
method Design-of-experiment approach with high-dimensional constraint space design and surrogate modeling.
result Established framework for evaluating AI algorithm quality robustly.
Automatically evaluates image quality based on human judgment.
problem Difficulty in rigorously evaluating generated image quality.
method Generative model embeddings, human labels regression, and statistical matching.
result 66% accuracy in predicting human scores of image realism.
Framework evaluates quality of synthetic data generated with differential privacy.
problem Ensuring synthetic data retains statistical quality after applying differential privacy.
method Developed a framework to evaluate synthetic data quality from a practical researcher's viewpoint.
result Synthetic data can be evaluated against training data or underlying populations, and for specific tasks like inference or prediction.
Paper explores how text generation quality and diversity metrics relate to distribution fitting.
problem Unclear relation between text generation quality and diversity metrics and distribution fitting.
method Theoretical approach to prove a linear combination of quality and diversity metrics can be a divergence metric.
result CR/NRR proposed as a better substitute for BLEU/Self-BLEU metrics.
New metric correlates local topic quality with human judgments.
problem Evaluation of topic models focuses on global metrics, ignoring token-level assignments.
method Proposed a human evaluation task and automated metrics to assess local topic quality.
result Consistency metric correlates best with human judgments of local topic quality.
The paper evaluates and compares dimensionality reduction quality metrics without tuning.
problem Evaluating the quality of nonlinear dimensionality reduction visualizations is challenging.
method Comparison of dimensionality reduction quality metrics on datasets with known ground truth manifolds.
result A few methods consistently perform well, with one proposed as a benchmark.
CrossLID evaluates GAN quality by comparing intrinsic dimensions.
problem Measuring GAN quality and detecting mode collapse.
method CrossLID measures local intrinsic dimensionality of real-world data vs. GAN-generated samples.
result CrossLID correlates with GAN training progress and is robust to noise.
Framework improves ML performance by identifying high-quality data.
problem Poor data quality hampers ML performance.
method Intelligent data-centric evaluation framework combining quality measurements and unsupervised learning.
result Framework improves ML system performance in real-world use case.
Study finds strict collection policies improve portfolio quality of microfinance banks.
problem Improving portfolio quality of microfinance banks through better credit collection policies.
method Multi-stage sampling, regression analysis, descriptive statistics.
result Collection policy has a higher effect on portfolio quality.
Paper proposes metrics to evaluate both quality and diversity in text generation models.
problem Existing metrics only evaluate quality or diversity, not both.
method Approximate the distance between generative model and real data distribution using n-gram and BERT features.
result Proposed metrics better evaluate both quality and diversity of text generation models.
Protein structure prediction has been a grand challenge problem in the structure biology over the last few decades. Protein quality assessment plays a very important role in protein structure prediction. In the paper, we propose a new protein quality assessment method which can predict both local and global quality of …
Generative models predict page quality without training, useful for low-resource settings.
problem Detecting low-quality content in web articles.
method Human evaluation and analysis of 500 million web articles.
result Generative models can predict page quality without training, useful for low-resource settings.
Proposes GM Score to evaluate GANs considering diversity, disentanglement, and discriminability.
problem Evaluation of GANs for sample quality and diversity.
method Integrates various factors including intra-class and inter-class diversity, disentanglement, and discriminability metrics.
result Demonstrates improved evaluation of GANs on MNIST dataset.
Automated dialogue quality evaluation using user satisfaction estimates across multiple domains.
problem Lack of automated and domain-independent dialogue quality evaluation metrics.
method Created a new Response Quality annotation scheme, introduced five domain-independent feature sets, and experimented with six machine learning models.
result Gradient Boosting Regression model achieved best prediction performance, with a 16% relative improvement in binary satisfaction class prediction accuracy.
New metric estimates user satisfaction for dialogue quality evaluation.
problem Sparse and intrusive explicit user feedback for dialogue quality evaluation.
method Created a new Response Quality annotation scheme and developed a turn-level User Satisfaction metric using five domain-independent feature sets and six machine learning models.
result Gradient boosting regression achieved best correlation of ~0.79 between predicted and annotated user satisfaction labels.
Study evaluates quality of uncertainty estimates for neural networks.
problem Lack of principled assessment methods for evaluating uncertainty quality in deep learning.
method Statistical methods of frequentist interval coverage, interval width, and expected calibration error.
result Different UQ methods produce markedly different quality uncertainty estimates.
This research evaluates the quality of unsupervised embeddings using linear separability metrics.
problem Difficulty in evaluating unsupervised learning model performance in practice.
method Survey and introduction of three methods, including a novel one, to assess embedding quality.
result Metrics can robustly estimate embedding quality in an unsupervised way.
A framework assesses the quality of crowdsourced weather data.
problem Quality control and assessment of crowdsourced weather data from third-party stations.
method Proposes a simple, scalable, and interpretable AI/Stats/ML framework to assess TPAWS data.
result Demonstrates the performance of the framework using synthetic and real data.
Enhanced metrics improve generative model evaluation reliability.
problem Lack of reliable quality metrics for generative models.
method Introduce Clipped Density and Clipped Coverage metrics.
result Metrics prevent out-of-distribution samples from biasing quality scores.
Two parallel samplers enhance image quality in limited denoising steps.
problem Limited denoising steps in diffusion models reduce image quality.
method Two parallel samplers denoise at successive times, integrating their information.
result Two parallel samplers improve image quality compared to a single sampler.
Stabilizes training of DNN for speech enhancement using PESQ scores.
problem Stability issues in training DNNs using non-differentiable OSQA scores.
method Approximate OSQA scores with a differentiable auxiliary DNN and stabilize training with reinforcement learning techniques.
result Stable training of DNN to achieve state-of-the-art PESQ scores and better sound quality.
The paper proposes using density ratio estimation to evaluate synthetic data quality.
problem Improving the quality and utility of synthetic data for analysis.
method Density ratio estimation to measure synthetic data quality.
result Density ratio estimation yields more accurate global utility estimates than existing methods.
With super-resolution optical microscopy, it is now possible to observe molecular interactions in living cells. The obtained images have a very high spatial precision but their overall quality can vary a lot depending on the structure of interest and the imaging parameters. Moreover, evaluating this quality is often di…
The paper introduces a method to assess machine translation quality with confidence intervals.
problem Evaluating the uncertainty and quality of machine translation.
method Utilizes conformal predictive distributions to produce prediction intervals with guaranteed coverage.
result The method outperforms a baseline on six language pairs in terms of coverage and sharpness.
The paper proposes a new method to evaluate LLM agent responses using ECDF clustering.
problem The standard evaluation of LLM agent responses via majority voting obscures response quality and distribution.
method The paper introduces a novel evaluation framework based on ECDF of cosine similarities and clustering of ECDFs using distances and k-medoids algorithm. result ECDF clustering reveals interpretable group structures in LLM responses, offering insights into agent settings.
New methods correct bias in LLM-as-a-Judge evaluations, but reliability depends on judge quality and model calibration.
problem Systematic bias in LLM-as-a-Judge evaluations using naive estimators.
method Analytical results, simulations, and real-data case study to diagnose reliability of corrected estimates.
result Corrected estimates, especially shared-calibration comparisons, can be unreliable under certain conditions.
FinSphere improves stock analysis quality with AI and expert-curated data.
problem Lack of objective evaluation metrics and depth in stock analysis by FinLLMs.
method Developed AnalyScore, curated Stocksis dataset, and FinSphere AI agent.
result FinSphere outperforms general and domain-specific LLMs in generating high-quality stock analysis reports.
This paper evaluates forecast quality in electricity markets beyond traditional accuracy measures.
problem Traditional accuracy measures fail to reflect the economic value of electricity price forecasts.
method Investigates four quality dimensions: accuracy, dispersion, association, and extremum identification.
result Dispersion- and association-based measures better capture forecast economic value.
Meta-Router optimizes LLM selection using gold-standard and preference-based data.
problem Training a high-quality LLM router with combined data sources is challenging due to bias and scarcity.
method Developed an integrative causal router training framework to correct bias and improve routing accuracy.
result Our approach delivers more accurate routing and improves the trade-off between cost and quality.
CALLISTO generates tests and assesses ML data quality using prediction entropy.
problem Validating ML systems for accuracy and data quality.
method Entropy-based test generation and data quality assessment framework.
result CALLISTO detects up to 20x more errors than traditional methods.
New research highlights flaws in evaluating clustering algorithms using classification datasets.
problem Flaws in evaluating clustering algorithms using classification datasets.
method Advanced visualization and dimension reduction techniques to expose flaws.
result Current practice of evaluating clustering algorithms may produce misleading results.
Unified metric c-Eval evaluates feature-based explanations by perturbation.
problem Lack of consensus on evaluating feature-based local explanations.
method Introduces c-Eval metric and framework to quantify explanation quality.
result c-Eval captures the importance of input features and is applicable in adversarial-robust models.
QANet estimates image segmentation quality without human inspection.
problem Quantitative evaluation of image segmentation quality.
method QANet is a Quality Assurance Network that solves a regression problem to estimate quality measures.
result QANet accurately estimates segmentation quality with respect to ground truth.
Study evaluates margin parameter effects on knowledge embedding quality.
problem Understanding margin parameter's impact on embedding quality.
method Examined margin parameter values for multi-relational categorized data.
result Lower margin values are insufficient, while larger values cause noise.
Enhances clustering quality evaluation in noisy data.
problem Reliable clustering quality assessment in noisy Gaussian mixtures.
method Feature Importance Rescaling (FIR) method.
result FIR improves correlation between cluster validity indices and ground truth.
Spinal cord stimulation has enabled humans with motor complete spinal cord injury (SCI) to independently stand and recover some lost autonomic function. Quantifying the quality of bipedal standing under spinal stimulation is important for spinal rehabilitation therapies and for new strategies that seek to combine spina…
Novel Orlicz regrets consistently bound environmental variable statistics.
problem Consistent evaluation of stochastic environmental variables like water quality indices.
method Proposed novel Orlicz regrets for upper and lower bounds.
result Explicit linkage between Orlicz regrets and divergence risk measures.
The paper improves QD policy ensembles using distribution ratio estimators.
problem Training diverse and high-quality reinforcement learning agents.
method Using Stein variational gradient descent and distribution ratio estimators.
result The method generates diverse and high-quality reinforcement learning agents.
The paper assesses quality measures for machine learning models using cross-validation.
problem Evaluating the accuracy and robustness of quality measures for machine learning models.
method Cross-validation approach to estimate prediction error and quantify explained variation. Confidence bounds and local quality measures derived from residuals.
result The reliability and robustness of quality measures are assessed through numerical examples and confidence bounds.
The study analyzes how probabilistic forecasts improve battery trading strategies in electricity markets.
problem Improvements in statistical forecast quality do not directly translate to economic value in battery trading strategies.
method The study frames battery optimization as a stochastic program based on fully probabilistic forecasts and examines decision quality under different uncertainty models.
result The study identifies two critical flaws in quantile-based trading strategies and provides theoretical justification and empirical evidence.
Tree Index evaluates cluster quality by creating decision trees from data.
problem Evaluating the quality of cluster results from various techniques.
method Tree Index creates a decision tree from clustered data, combining entropy and depth of leaves.
result Tree Index discriminates between sensible and non-sensible clusters on brain dataset.
This paper uses LLMs and cycle consistency for better machine translation evaluation.
problem Evaluating translation quality and LLM capabilities without ground truth.
method Generate translation candidates, back-translate, and evaluate cycle consistency.
result Larger LLMs or more inference passes improve cycle consistency.
We focus our attention on the link prediction problem for knowledge graphs, which is treated herein as a binary classification task on neural embeddings of the entities. By comparing, combining and extending different methodologies for link prediction on graph-based data coming from different domains, we formalize a un…
DAISYnt evaluates synthetic data quality and privacy in regulated domains.
problem Balancing data quality and privacy in regulated domains.
method Developed a suite of advanced tests (DAISYnt) to evaluate synthetic data quality and privacy.
result DAISYnt sets a de facto standard for synthetic data evaluation in regulated domains.
VLM judges rank well but score poorly; task difficulty and annotation quality affect interval width.
problem VLMs as judges lack reliability indicators in multimodal evaluations.
method Conformal prediction using score-token log-probabilities.
result Evaluation uncertainty is task-dependent, affecting interval width and reliability.
A new first-order sampler improves diffusion probabilistic model sampling quality.
problem The belief that first-order methods are inherently slower for diffusion probabilistic model sampling.
method A novel training-free, first-order sampler that approximates the forward-value evaluation via a one-step lookahead predictor.
result The proposed sampler provably approximates the ideal forward-value trajectory while retaining first-order convergence and can improve sample quality under the same NFE budget.