Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
This paper introduces a very challenging dataset of historic German documents and evaluates Fully Convolutional Neural Network (FCNN) based methods to locate handwritten annotations of any kind in these documents. The handwritten annotations can appear in form of underlines and text by using various writing instruments…
Singing voice separation attempts to separate the vocal and instrumental parts of a music recording, which is a fundamental problem in music information retrieval. Recent work on singing voice separation has shown that the low-rank representation and informed separation approaches are both able to improve separation qu…
This paper introduces a new large-scale music dataset, MusicNet, to serve as a source of supervision and evaluation of machine learning methods for music research. MusicNet consists of hundreds of freely-licensed classical music recordings by 10 composers, written for 11 instruments, together with instrument/note annot…
Entity resolution (ER) presents unique challenges for evaluation methodology. While crowdsourcing platforms acquire ground truth, sound approaches to sampling must drive labelling efforts. In ER, extreme class imbalance between matching and non-matching records can lead to enormous labelling requirements when seeking s…
Unsupervised learning classifies transient noise in gravitational wave detectors.
problem Transient noise interferes with gravitational wave signals, causing instability.
method Combines variational autoencoder and invariant information clustering.
result Consistent classification with Gravity Spy project labels.
Estimates long-term effects using past experiments as instruments with many weak instruments.
problem Estimating long-term causal effects with limited short-term outcomes and many weak instruments.
method Nonparametric instrumental variable inference with many weak instruments, using past experiments as instruments.
result Automatic debiased machine learning estimators for linear functionals of the structural function and its minimum-norm projection are efficient in the many-weak-instruments regime.
ZNet learns instrumental representations from covariates for causal inference.
problem Lack of valid instruments in observational studies.
method Representation learning approach that constructs instrumental representations from observed covariates.
result ZNet enables IV-based estimation without explicit instruments.
TSCI estimates treatment effects using machine learning and data-adaptive methods for invalid instruments.
problem Estimating treatment effects with invalid instruments.
method Two-stage algorithm: first stage uses machine learning for nonlinearities, second stage selects and projects out instrument violations.
result Effective treatment effect estimation even with invalid instruments.
Researchers develop methods for causal inference with imperfect instrumental variables.
problem Quantifying cause and effect relationships with imperfect instrumental variables.
method Established a quantitative relationship between violations of instrumental inequalities and minimal measurement dependence, providing adapted inequalities valid in the presence of relaxed measurement dependence.
result Adapted inequalities for average causal effect in instrumental scenarios with binary outcomes, addressing violations of instrumental inequalities.
Paper proposes CIV estimator for categorical instruments in small sample settings.
problem Estimation with categorical instruments in settings with few observations per category.
method CIV estimator leveraging regularization assumption for latent categorical variable.
result CIV estimator is asymptotically normal, efficient, and semiparametrically efficient under homoskedasticity.
Active learning selects both observations and annotation precision for Gaussian Processes.
problem Costly annotation in supervised learning.
method Proposes an active learning algorithm that selects observations and annotation precision, using a modified BALD objective.
result Empirically shows the benefits of adjusting annotation precision in active learning.
One of the problems on the way to successful implementation of neural networks is the quality of annotation. For instance, different annotators can annotate images in a different way and very often their decisions do not match exactly and in extreme cases are even mutually exclusive which results in noisy annotations a…
New method uses few instruments to estimate complex causal effects.
problem Estimating causal effects with limited instruments in high-dimensional settings.
method Sequentially selects and combines instruments to estimate the treatment effect.
result Can reliably recover the treatment effect's projection onto the instrumented subspace.
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
Accurate annotation of medical image is the crucial step for image AI clinical application. However, annotating medical image will incur a great deal of annotation effort and expense due to its high complexity and needing experienced doctors. To alleviate annotation cost, some active learning methods are proposed. But …
Paper develops a new estimator for panel data with endogenous treatments, improving causal inference.
problem Challenges in causal inference for static panel data with endogenous treatments and confounding variables.
method Develops Double Machine Learning (DML) estimator for static panel models with endogenous treatments (panel IV DML). Introduces weak-identification diagnostics.
result Panel IV DML estimator improves estimation accuracy and delivers more reliable inference under weak identification.
DFIV uses deep neural nets to learn nonlinear features in IV regression.
problem Learning causal relationships from observational data with nonlinear interactions.
method DFIV trains deep neural nets to define nonlinear features on instruments and treatments, alternating training to compose stages 1 and 2.
result DFIV outperforms state-of-the-art methods on IV benchmarks and off-policy policy evaluation.
PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.
problem Inaccurate and insufficient confusion matrices for annotators in multi-class classification tasks.
method PTBCC (ProtoType learning-driven Bayesian Classifier Combination) uses prototype confusion matrices to capture annotator expertise.
result PTBCC achieves up to 15% accuracy improvement and 3% higher average accuracy compared to existing methods.
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Bayesian methods improve text annotation quality.
problem Inconsistent and unreliable human annotations in natural language processing.
method Two semi-supervised Bayesian methods: a deep learning model and an ensemble method.
result Bayesian methods enhance the reliability and performance of BERT models.
Learning a causal effect from observational data is not straightforward, as this is not possible without further assumptions. If hidden common causes between treatment X and outcome Y cannot be blocked by other measurements, one possibility is to use an instrumental variable. In principle, it is possible under some…
Paper proposes an efficient method for bounding box annotation in object detection.
problem Manual annotation of bounding boxes is tedious and resource-intensive.
method Iterative training of object detector on small batches of labeled images, with human annotator correcting errors.
result Significant reduction in human annotation effort, up to 75%.
Human annotations serve an important role in computational models where the target constructs under study are hidden, such as dimensions of affect. This is especially relevant in machine learning, where subjective labels derived from related observable signals (e.g., audio, video, text) are needed to support model trai…
Paper proposes a weak supervision technique for CNN semantic segmentation of lung diseases using partially annotated data.
problem Creating annotated datasets for semantic segmentation of lung diseases is laborious and time-consuming.
method Proposes a weak supervision technique that utilizes partially annotated datasets to improve CNN semantic segmentation accuracy.
result Significantly improved segmentation accuracy using partially annotated datasets.
We study the problem of semi-supervised singing voice separation, in which the training data contains a set of samples of mixed music (singing and instrumental) and an unmatched set of instrumental music. Our solution employs a single mapping function g, which, applied to a mixed sample, recovers the underlying instrum…
Valid causal inference with invalid instruments using majority or modal valid relationships.
problem Estimating causal effects in the presence of unobserved confounding and invalid instruments.
method Ensemble of instrumental variable estimators to estimate the modal prediction, achieving accurate estimates of conditional average treatment effects.
result Valid causal inference can be achieved with a majority or modal valid instrument-response relationship.
Annotation guidelines used to guide the annotation of training and evaluation datasets can have a considerable impact on the quality of machine learning models. In this study, we explore the effects of annotation guidelines on the quality of app feature extraction models. As a main result, we propose several changes to…
Bayesian nonparametric machine learning improves instrumental variable inference.
problem Estimating causal effects with nonlinear relationships.
method Bayesian Additive Regression Trees (BART) for estimating functions and Dirichlet Process mixtures for error terms.
result Dramatic improvements in inference with nonlinear data, no manual tuning required.
In this paper, we learn disentangled representations of timbre and pitch for musical instrument sounds. We adapt a framework based on variational autoencoders with Gaussian mixture latent distributions. Specifically, we use two separate encoders to learn distinct latent spaces for timbre and pitch, which form Gaussian …
Improves off-policy evaluation with imperfect annotations.
problem Limited dataset coverage for evaluating new policies.
method Doubly robust estimators combining IS and DM, incorporating counterfactual annotations.
result Using annotations within the DM component yields the most desirable theoretical results.
Proposes TSCI method to infer causal effects with weak or invalid instruments using machine learning.
problem Causal inference with weak or invalid instrumental variables.
method Two-stage curvature identification (TSCI) using machine learning.
result Asymptotically unbiased and Gaussian estimator for causal effects.
The paper uses graph learning to detect valid instruments in high-dimensional data for house pricing.
problem Endogeneity bias and invalid instrument validation in high-dimensional data.
method Merge variable selection algorithms and probabilistic graphs to estimate house prices and causal structure.
result Efficient data-driven instrument selection and invalid instrument purge in high-dimensional data.
Simplifies IV regression for high-dimensional instruments.
problem Nonlinear instrumental variable regression with high-dimensional instruments.
method Combines kernelized IV methods with an adaptive regression algorithm.
result Faster convergence and adaptability to feature dimensionality.
A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal…
New method for personalized pricing using invalid instrumental variables.
problem Personalized pricing under endogeneity with limited standard methods.
method PRINT method for continuous treatment, solving conditional moment restrictions.
result Established optimal pricing strategy under endogeneity with invalid instrumental variables.
GIV methodology extends instrumental variable estimation for high-dimensional data.
problem Estimating structural parameters in high-dimensional models with endogeneity and latent factors.
method Extends GIV methodology to large N and T, treats factors and loadings as unknown, and uses additional instruments for efficiency.
result Efficiency gains and negligible sampling errors in estimated instrument and factors.
Crowdlab uses classifiers to estimate consensus labels and annotator quality.
problem Leveraging multiple annotators for data classification.
method Weighted ensemble approach using any trained classifier.
result Superior estimates for consensus labels and annotator quality.
Estimates linear model from noisy covariates and instruments using spectral regularization.
problem Estimating a linear model from many noisy covariates and instruments.
method Two-stage least squares with spectral regularization of canonical correlations.
result Upper and lower bounds on estimation error, proving optimality of the method with noisy data.
Recidivism prediction instruments (RPI's) provide decision makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time. While such instruments are gaining increasing popularity across the country, their use is attracting tremendous controversy. Much of the controversy c…
The increase in data collection has made data annotation an interesting and valuable task in the contemporary world. This paper presents a new methodology for quickly annotating data using click-supervision and hierarchical object detection. The proposed work is semi-automatic in nature where the task of annotations is…
New method improves IV estimation with many weak and invalid instruments.
problem Identification in linear IV models with unknown validity.
method Non-convex penalized approaches, surrogate sparsest penalty.
result Advantages over other IV estimators in selection consistency and weak IV strength conditions.
New method exploits independence in instrumental variable models for better causal inference.
problem Identify causal functions in the presence of unobserved confounders.
method HSIC-X method that exploits independence between response, hidden confounders, and instruments.
result The method provides better finite sample results and is invariant to distributional shifts.
This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Our framework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expert…
Develops methods to identify and estimate causal effects with instrumental variables.
problem Causal inference with confounded treatment assignment and unobserved variables.
method General nonparametric causal framework, debiased machine learning, semiparametric theory.
result Consistent and asymptotically normal estimators for average treatment effect.
Kernel method improves instrumental variable regression rates.
problem Nonparametric instrumental variable regression with weak instruments.
method Kernel-based two-stage least-squares method, strong L2 convergence analysis. result Minimax optimal rates for instrumental regression under standard assumptions.