AtC combines human judgments and model scores for better human-centered assessments.
problem Lack of verifiable ground truth in human-centered assessments.
method Two-stage framework: aggregate judgments, then calibrate model scores.
result AtC outperforms human-only or model-only assessments across datasets.
HRL improves open-domain dialog models by optimizing long-term conversational goals.
problem Challenges in open-domain dialog generation, including repetitive outputs, difficulty tracking conversational goals, and inappropriate text.
method Proposes VHRL, a hierarchical reinforcement learning approach using policy gradients to tune utterance-level embeddings of a variational sequence model.
result Significant improvements in human evaluation and automatic metrics over state-of-the-art dialog models.
HCLM framework uses entropy regularization for open learning systems.
problem Real-world AI challenges and limitations of deep learning.
method Dynamical and information-theoretic framework with entropy regularization.
result Geometric entropy surrogates, especially log-determinant covariance entropy, induce stronger and more stable information forces.
The paper proposes a happiness-based fairness framework using linear programming.
problem Fairness in decision-making outcomes.
method Linear programming to compute optimal, fair post-processing strategies.
result Unified and extended fairness definitions with practical strengths across diverse scenarios.
With an eye towards human-centered automation, we contribute to the development of a systematic means to infer features of human decision-making from behavioral data. Motivated by the common use of softmax selection in models of human decision-making, we study the maximum likelihood parameter estimation problem for sof…
Paper proposes self-supervised method for accurate speaker diarization.
problem Speaker diarization without massive labeling effort.
method Introduces dynamic triplet loss and multinomial loss for audio-video synchronization.
result Best model yields +8% F1-score improvement and diarization error rate reduction.
A method for data encryption makes data look identical to humans but misleading to machine learning.
problem Data leakage in data sharing for medical purposes.
method Proposes a method inspired by adversarial attacks for data encryption.
result Encrypted data look identical to humans but misleading to machine learning methods.
LIMEADE improves AI advice for opaque models, enhancing accuracy and user satisfaction.
problem Lack of advice methods for opaque AI models.
method Develops a general framework to translate advice into model updates.
result Improves accuracy and user satisfaction compared to baselines.
Survey of visual analytics in deep learning for better model understanding.
problem Challenges in interpreting deep learning models due to their complexity.
method Human-centered interrogative framework focusing on Five W's and How.
result Standardized tools for model explanation, interpretation, debugging, and improvement.
Interactive EMA combines multiple explainability methods to improve model understanding.
problem Isolated explanations of machine learning models lead to misunderstandings and wrong reasoning.
method Interactive EMA (IEMA) combines multiple explainability methods sequentially.
result Interactive EMA increases the performance and confidence of human decision making.
Protein structure prediction has been a grand challenge problem in the structure biology over the last few decades. Protein quality assessment plays a very important role in protein structure prediction. In the paper, we propose a new protein quality assessment method which can predict both local and global quality of …
Paper introduces active Bayesian method for assessing black-box classifiers efficiently.
problem Need to assess performance of black-box classifiers reliably with limited labels.
method Develops inference strategies and proposes active Bayesian framework for efficient instance selection.
result Significant gains in performance assessment with fewer labels compared to traditional methods.
HodgeRank method improves fairness in online peer assessments.
problem Bias and heterogeneity in peer assessment leading to unfair scoring.
method Reference ranking method using HodgeRank for online peer assessment.
result Objective scoring reference for instructors based on mathematics.
AI could democratize education but risks inequality.
problem AI risks creating educational inequality.
method Speculation and posing questions on AI in education.
result AI could exacerbate educational inequality.
Study examines how different assessment formats affect student learning in a data communications course.
problem Understanding how various assessment formats impact student learning outcomes.
method Comparing student learning outcomes across multiple assessment formats in a core data communications course at George Mason University.
result Collective assessment formats enhance student knowledge demonstration.
New method assesses graph generators using graph classifiers.
problem Quantifying how well generative models create realistic graphs.
method Using graph classifiers to evaluate synthesized graphs against real ones.
result Inability of a classifier to distinguish real from synthetic graphs indicates poor model performance.
Paper explores physics-informed deep learning for system reliability assessment.
problem Limited study on deep learning for system reliability assessment.
method Physics-informed deep learning approach for system reliability assessment.
result Physics-informed deep learning can alleviate computational challenges and combine measurement data and mathematical models.
chemmodlab simplifies fitting and assessing machine learning models in cheminformatics.
problem Comparing the utility of new machine learning models in cheminformatics.
method Streamlines model fitting and assessment pipeline, using k-fold cross-validation and multiplicity adjustments.
result Ease of presenting statistically significant performance differences among models.
Bayesian networks improve product risk assessment by handling uncertainty and causality.
problem Limited handling of uncertainty and inability to incorporate causal explanations in existing methods.
method Bayesian Networks (BNs) for improved systematic product risk assessment.
result BN approach provides more powerful and flexible risk assessments.
Enhances early risk assessments for pediatric outcomes using contrastive learning.
problem Improving risk assessments in early stages of pediatric development.
method Contrastive multi-modal framework that treats each time window as a distinct modality, training on all available data.
result Consistent improvements in early-stage risk assessments validated on real-world tasks.
Improved assessment of knee osteoarthritis using geodesic B-score.
problem Need for automatic, reader-independent measures of osteoarthritis clinical outcomes.
method Derive a geodesic B-score for Riemannian shape spaces, develop efficient algorithm for large shape populations.
result Geodesic B-score exhibits improved discrimination ability over Euclidean B-score.
The paper discusses safety assessment for AI systems, focusing on machine learning models.
problem Safety assessment of AI systems, especially machine learning models, in safety-related applications.
method Analyzed AI models as statistical models and proposed a new budget allocation for AI safety.
result Safety assessment of AI systems requires a new approach focusing on the model used, not just the system.
Deep learning improves mammography assessment with high accuracy.
problem Challenges in mammography assessment due to noise, resolution, and lack of ground truths.
method Proposes a classification approach using multi-scale deep tissue classifiers.
result Highest AUC of 0.9 achieved in classifying suspicious tissue patches.
Unified framework for CVA sensitivities, hedging, and risk assessment.
problem Computing and managing Credit Value Adjustment (CVA) sensitivities and risks.
method Probabilistic machine learning and refined regression on simulated data, validated by Monte Carlo methods.
result Identification of optimal sensitivities for practical tasks like hedging and risk assessment.
Large corporate credit models may be adapted for small business risk assessment.
problem Limited data and lack of credit analysts for small businesses.
method Adapting large corporate credit risk models for small businesses.
result Adapted models can predict small business credit risk effectively.
DecoupleNets use neural networks to assess and select dependence models.
problem Assessing and selecting dependence models for multivariate data.
method Neural networks (DecoupleNets) transform data to uniformity, then assess and select models.
result DecoupleNets provide a novel, efficient method for dependence model assessment and selection.
C-FAR automates clustering assessment for neural tracking.
problem Manual assessment of clusters by humans is slow and impractical for large datasets.
method C-FAR uses automated feedback queries to select optimal clustering from multiple algorithms.
result C-FAR produces near-perfect clustering on simulated neural data.
R2DE assesses new exam questions quickly and accurately.
problem Assessing newly generated exam questions efficiently and accurately.
method R2DE uses NLP to estimate difficulty and discrimination of questions based on their text.
result R2DE can estimate question difficulty and discrimination quickly and accurately.
Automated fatigue assessment using ECG and actigraphy sensors.
problem Fatigue assessment based on self-reporting suffers from recall bias.
method Wearable sensing, machine learning, feature selection, self-attention model, consistency self-attention mechanism.
result Very promising results achieved in fatigue assessment.
Develops methods for AI self-assessment to improve trustworthiness.
problem Uncertainty in AI predictions and lack of trust in AI systems.
method Uncertainty estimation techniques considering practical impacts and costs.
result Guidelines for selecting and designing effective AI self-assessment methods.
Mindful active learning improves activity recognition using wearable sensors.
problem Activity recognition using wearable sensors with human cognitive and physical limitations.
method Introduces mindful active learning, a framework that considers human memory and query budget.
result EMMA framework achieves higher accuracy than traditional methods, especially with limited query budgets and weak human memory.
Report assesses Edinburgh tram project's risk management.
problem Projects often exceed costs and timelines, underperforming intended benefits.
method Describes and evaluates risk assessment and management for the Edinburgh tram.
result Careful consideration of cost, schedule, and benefit risks is crucial for megaprojects.
Model uses smartphone data to assess MS trajectories.
problem Personalized longitudinal MS assessment.
method Imputation, generalized estimation equation, ensemble learning, fine-tuning.
result Promising model for predicting MS over time.
Automated MRI image quality assessment framework using machine learning.
problem Manual quality assessment of MRI images is time-consuming and costly.
method Machine learning model trained on human observer labels without reference images.
result Framework achieves 93.7% accuracy in estimating image quality.
New method assesses financial and cyber risks under uncertainty.
problem Uncertainty in risk assessment for financial and cyber systems.
method Combines stochastic approximation and distorted mix method to compute worst case average value at risk.
result Efficient algorithm for tail uncertainty in multivariate distributions.
New method approximates CV for model assessment and selection.
problem Efficient model assessment and selection with large number of folds.
method Approximates expensive refitting with a single Newton step warm-started from full training set optimizer.
result Uniform non-asymptotic, deterministic model assessment guarantees for approximate CV.
Fast risk assessment for autonomous vehicles using learned agent futures.
problem Risk assessment for autonomous vehicles given probabilistic predictions of other agents' futures.
method Non-sampling based methods using deep neural networks for probabilistic predictions, with Gaussian and non-Gaussian mixture models for agent positions and controls.
result Effective risk assessment for low probability events using learned models of agent futures.
The paper assesses quality measures for machine learning models using cross-validation.
problem Evaluating the accuracy and robustness of quality measures for machine learning models.
method Cross-validation approach to estimate prediction error and quantify explained variation. Confidence bounds and local quality measures derived from residuals.
result The reliability and robustness of quality measures are assessed through numerical examples and confidence bounds.
PerSense assesses personality traits from text for commonsense reasoning.
problem Estimating human personality traits from text for mental health analysis.
method Aggregated Probability Density Functions (PDF) and Machine Learning (ML) models.
result PerSense algorithms achieve comparable results to ground truth data, with high accuracy for personality assessment and commonsense prediction.
A deep learning framework assesses physical rehabilitation exercises.
problem Lack of versatile, robust, and practical assessment methods for rehabilitation exercises.
method Deep learning framework with metrics, scoring functions, and neural networks.
result First implementation of deep neural networks for rehabilitation performance assessment.
Dual quality assessment method tackles adversarial robustness issues across various metrics.
problem Varying robustness levels and bias in adversarial attacks and defenses.
method Model agnostic dual quality assessment method, including robustness levels.
result Current networks and defenses are vulnerable at all robustness levels, highlighting the need for a dual approach.
Paper proposes a new model to assess risks in energy storage systems considering both exogenous and endogenous uncertainties.
problem Current risk assessment ignores the stochastic nature of energy storage availability.
method Data-driven unified model with exogenous and endogenous uncertainty description for four types of generic energy storage.
result Comparative results show more severe risks for endogenous uncertainty, suggesting new strategies for system operators.
Optimizes risk assessment tools using mixed-integer programming.
problem Challenges in healthcare risk assessment due to label scarcity and asymmetric misclassification costs.
method Jointly optimizes scoring weights and category thresholds via mixed-integer programming (MIP).
result Prevents label-scarce category collapse and achieves more accurate risk categorization.
TransCORALNet uses transformer and CORAL for supply chain credit assessment with cold start.
problem Supply chain credit assessment for new borrowers with limited data.
method Two-stream transformer CORAL networks with domain adaptation and LIME.
result TransCORALNet outperforms state-of-the-art models in accuracy.
Automatically assesses the quality of online health articles.
problem Lack of automated tools to evaluate the quality of online health information.
method Data mining approach using 10 quality criteria and feature selection.
result Classifier achieved 84%-90% accuracy on 10 criteria.
Study improves data quality assessment for structural monitoring data.
problem Ensuring reliability of structural health monitoring data.
method Probabilistic data quality assessment using a conditional diffusion model.
result Significantly improves accuracy of data quality assessment.
A new method assesses regression models' global optimality.
problem Challenges in evaluating regression models without access to true data.
method Information Teacher framework based on Shannon mutual information.
result Demonstrates capability to detect global optimality.
Generative models assess quality on time-series data using ITS and FITD.
problem Lack of consensus for quality assessment of class-conditional generative models on time-series data.
method Introduced InceptionTime Score (ITS) and Frechet InceptionTime Distance (FITD) to evaluate generative models.
result ITS and FITD combined with TSTR can accurately assess generative model performance on time-series data.