MCAL reduces labeling costs by 6x for auto-labeling data sets.
problem Expensive human annotation for ground-truth data sets.
method Iterative approach that trains a classifier to auto-label part of the data set, determining which samples to label using humans and which to label using the classifier at each step.
result 6x lower overall cost compared to human labeling the entire data set, always cheaper than competing strategies.
Study human-machine interaction with private info using offline RL.
problem Confounding bias and distributional mismatch in offline RL for human-guided interaction.
method Developed a novel identification result and OPE method to address confounding bias, and used pessimism to tackle distributional mismatch.
result Policy pair converges to optimal one at satisfactory rate under mild assumptions.
Proposes a graph-based method for active learning in regression.
problem Active learning for regression is understudied and faces challenges with noisy labels and parameter uncertainty.
method Formulates active regression as a bipartite graph optimization problem.
result Demonstrates the effectiveness of the proposed graph-based approach on benchmark data sets.
Affective computing has become a very important research area in human-machine interaction. However, affects are subjective, subtle, and uncertain. So, it is very difficult to obtain a large number of labeled training samples, compared with the number of possible features we could extract. Thus, dimensionality reductio…
The emerging paradigm of Human-Machine Inference Networks (HuMaINs) combines complementary cognitive strengths of humans and machines in an intelligent manner to tackle various inference tasks and achieves higher performance than either humans or machines by themselves. While inference performance optimization techniqu…
New AI assistant for power grid operators simplifies complex decision-making.
problem Complexity and uncertainty in power grid operations.
method Unified human-machine interface and AI integration.
result Development of a new assistant framework for power grid operators.
Improved speech emotion recognition using pre-trained language models.
problem Challenging task of speech emotion recognition for natural human-machine interaction.
method Fine-tuning pre-trained language models for text emotion recognition, combining with speech emotion recognition.
result 73.5% accuracy in speech emotion recognition on a subset of IEMOCAP dataset.
Due to concerns about human error in crowdsourcing, it is standard practice to collect labels for the same data point from multiple internet workers. We here show that the resulting budget can be used more effectively with a flexible worker assignment strategy that asks fewer workers to analyze easy-to-label data and m…
Study on predictable forward processes in trading without frequent evaluations.
problem Trading performance evaluation times not matching trading times.
method Solving a linear functional equation to construct predictable forward processes.
result Predictable forward processes are inherently myopic and optimal strategies do not use future information.
A model for human-machine decision-making with private info and opacity.
problem Optimizing decisions in a human-machine system with private info and opacity.
method Formulated as a two-player learning problem, proved lower and upper bounds on optimality.
result Simple coordination strategy is nearly minimax optimal, efficient learning possible under certain assumptions.
Framework ensures alignment between humans and machines in LLMs.
problem Human-machine misalignment in LLMs scoring mechanisms.
method Lightweight calibration framework for blackbox models.
result Provably guarantees alignment between humans and machines.
Develops methods for cooperative Bayesian inference.
problem Cooperation between learning agents.
method Sequential Bayesian inference approaches.
result Theoretical foundation for cooperative inference.
Artificial intelligence (AI) is intrinsically data-driven. It calls for the application of statistical concepts through human-machine collaboration during generation of data, development of algorithms, and evaluation of results. This paper discusses how such human-machine collaboration can be approached through the sta…
When might human input help (or not) when assessing risk in fairness domains? Dressel and Farid (2018) asked Mechanical Turk workers to evaluate a subset of defendants in the ProPublica COMPAS data for risk of recidivism, and concluded that COMPAS predictions were no more accurate or fair than predictions made by human…
We study the problem of designing AI agents that can robustly cooperate with people in human-machine partnerships. Our work is inspired by real-life scenarios in which an AI agent, e.g., a virtual assistant, has to cooperate with new users after its deployment. We model this problem via a parametric MDP framework where…
Deep architecture learns transferable features for robust speech emotion recognition.
problem Robust and discriminative features for diverse speech emotion domains.
method Jointly uses CNN for domain-shared features and LSTM for domain-specific emotion classification.
result Transferable features provide gains up to 18.4% in speech emotion recognition.
Combines human and model predictions for improved accuracy.
problem Improving classification accuracy when both human and model predictions are imperfect.
method Uses confusion matrices and calibration to combine probabilistic model outputs with human class-level predictions.
result Human-model combinations consistently outperform either alone, with accuracy gains even with limited human input.
As machine learning systems move from computer-science laboratories into the open world, their accountability becomes a high priority problem. Accountability requires deep understanding of system behavior and its failures. Current evaluation methods such as single-score error metrics and confusion matrices provide aggr…
Interfacing a kinetic action of a person to an action of a machine system is an important research topic in many application areas. One of the key factors for intimate human-machine interaction is the ability of the control algorithm to detect and classify different user commands with shortest possible latency, thus ma…
Cooperative communication plays a central role in theories of human cognition, language, development, culture, and human-robot interaction. Prior models of cooperative communication are algorithmic in nature and do not shed light on why cooperation may yield effective belief transmission and what limitations may arise …
Compared to humans, machine learning models generally require significantly more training examples and fail to extrapolate from experience to solve previously unseen challenges. To help close this performance gap, we augment single-task neural networks with a meta-recognition model which learns a succinct model code vi…
Paper tackles dynamic behavior of variable topology mechanisms, presenting new transition conditions.
problem Dynamic behavior of mechanisms with changing kinematic topology.
method Presented new transition conditions for variable topology mechanisms using projected motion equations and Voronets equations.
result Results show the dynamic behavior of joint locking in 3R and 6DOF mechanisms.
Autonomous systems can substantially enhance a human's efficiency and effectiveness in complex environments. Machines, however, are often unable to observe the preferences of the humans that they serve. Despite the fact that the human's and machine's objectives are aligned, asymmetric information, along with heterogene…
Dual-stage sEMG classification improves gesture recognition accuracy.
problem Improving accuracy in hand gesture recognition from sEMG signals.
method Dual-stage classification approach: first stage groups similar activities, second stage classifies within groups.
result Dual-stage classification yields significantly higher accuracy than single-stage approach.
In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human--machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interacti…
Bayesian topological learning improves EEG signal analysis for brain state classification.
problem Challenges in classifying and analyzing noisy, nonlinear, nonstationary EEG signals.
method Persistent homology with Bayesian framework to track topological features and incorporate prior knowledge.
result Bayesian topological learning outperforms existing methods for noisy EEG classification.
This paper evaluates deferring systems using causal inference.
problem Evaluating the impact of deferring systems on model accuracy is challenging.
method The paper uses a causal inference framework to evaluate deferring systems, distinguishing between scenarios with and without access to human predictions.
result The approach allows identifying causal effects of deferring strategies on predictive accuracy.
Deep neural networks improve sEMG-based hand gesture classification.
problem Accurate classification of hand gestures from sEMG signals.
method Master-slave architecture with DNNs and synthetic feature data.
result Up to 9% improvement in accuracy with synthetic data.
We present a systematic analysis on the performance of a phonetic recogniser when the window of input features is not symmetric with respect to the current frame. The recogniser is based on Context Dependent Deep Neural Networks (CD-DNNs) and Hidden Markov Models (HMMs). The objective is to reduce the latency of the sy…
Object tracking is an ubiquitous problem that appears in many applications such as remote sensing, audio processing, computer vision, human-machine interfaces, human-robot interaction, etc. Although thoroughly investigated in computer vision, tracking a time-varying number of persons remains a challenging open problem.…
Study shows current metrics for audio adversarial examples are unreliable for human perception.
problem The reliability of metrics for evaluating audio adversarial examples.
method Analytical framework and human evaluation experiment.
result Current metrics for audio adversarial examples are not reliable for human perception.
Bayesian CNN estimates uncertainty in COVID-19 detection.
problem Uncertainty in deep learning predictions for medical diagnosis.
method Drop-weight based Bayesian Convolutional Neural Networks (BCNN).
result Uncertainty correlates with prediction accuracy.
Despite showing state-of-the-art performance, deep learning for speech recognition remains challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Recently, there have been greater efforts in the design of small, low-footprint deep neural networks (DNNs) that are more appropriate fo…
CA connects visual concepts to neural network representations.
problem Global association of human interpretable concepts with neural network internals.
method Established a taxonomy for CA methods and reviewed state-of-the-art research.
result Unified understanding of CA methods and datasets.
ShapleyBO explains BO's decisions, enhancing human-AI collaboration in robotics.
problem Lack of explainability in BO's black box nature, especially in human-in-the-loop applications.
method Uses game-theoretic Shapley values to quantify and interpret BO's parameter contributions.
result ShapleyBO identifies exploration and exploitation contributions, disentangling aleatoric and epistemic uncertainty.
The paper offers a checklist for comparing human and machine visual perception.
problem Comparing human and machine visual perception.
method Designing, conducting, and interpreting experiments to investigate mechanisms.
result Feedback mechanisms may not be necessary for visual reasoning tasks.
Decoding language representations directly from the brain can enable new Brain-Computer Interfaces (BCI) for high bandwidth human-human and human-machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak. In this study, we prop…
PHASE dataset simulates complex social interactions in physical environments.
problem Lack of datasets for evaluating physically grounded perception of complex social interactions.
method Created PHASE dataset of 2D animations with procedural generation and physics engine.
result SIMPLE model outperforms neural networks in recognizing complex social interactions.
Society's drive toward ever faster socio-technical systems, means that there is an urgent need to understand the threat from 'black swan' extreme events that might emerge. On 6 May 2010, it took just five minutes for a spontaneous mix of human and machine interactions in the global trading cyberspace to generate an unp…
Proposes a new model for noisy labels considering multiple labelers and adversarial attacks.
problem Real-world noisy label models with multiple labelers and adversarial attacks.
method Labeler-dependent noise model with adversarial attack vectors.
result State-of-the-art approaches for learning from noisy labels are defeated by adversarial label attacks.
An assistant learns to mediate decisions between humans and experts, balancing risk and learning.
problem Learning to mediate decisions between imperfect humans and expert knowledge.
method Formalizes online decision mediation, proposes a policy to balance immediate loss and future generalization.
result Consistent gains over benchmarks in decision-making performance.
Label smoothing improves model performance even with noisy labels.
problem Mitigating label noise in deep learning models.
method Examined label smoothing as a technique to cope with label noise and compared it to loss-correction methods.
result Label smoothing is competitive with loss-correction techniques under label noise and beneficial for distillation from noisy data.
Paper proposes a method to recover accurate labels from partially valid data in multi-label learning.
problem Tackles noisy supervision in multi-label learning with partially valid labels.
method Develops a two-stage method that estimates label enrichment and ground-truth confidences.
result Demonstrates improved performance over state-of-the-art PML methods.
CbMLC improves multi-label classification with noisy labels.
problem Evaluating multi-label classifiers with noisy labels.
method Context-Based Multi-Label Classifier (CbMLC) that handles noisy labels without additional supervision.
result CbMLC yields substantial improvements over previous methods in noisy label settings.
Proposes ML-GCN for multi-label network node representation learning.
problem Complex multi-label networks with correlated labels.
method Two Siamese GCNs model node-label and label-label interactions, integrated under a unified objective function.
result Effective node representation learning with preserved label interactions.
Proposes MGPLL for PL learning with non-random noise.
problem Partial label learning with non-random label noise.
method Bi-directional mapping framework, conditional noise label generation, multi-class predictor, adversarial learning.
result Demonstrates state-of-the-art performance in partial label learning.
Logistic regression can handle noisy labels effectively when labels are imperfectly assigned by multiple experts.
problem Label noise in supervised classification due to manual labelling by multiple experts.
method Using approximate posterior probabilities of class membership from multiple experts to train logistic regression models.
result Logistic regression can be robust to label noise when classification difficulty is the only source of errors.
FLAME auto-labels mobile data efficiently on diverse processors.
problem Accurately and efficiently labeling mobile data with unknown labels on heterogeneous processors.
method Self-adaptive auto-labeling system Flame that schedules and executes workloads on mobile processors.
result Flame achieves high labeling accuracy and performance on heterogeneous mobile processors.