source: arxiv statistics ml: design-based supervised learning with noisy human labels
level: research
researchers often use automated classifiers to label unstructured data for statistical analysis. existing methods can correct errors in these labels using a probability-sampled audit set, but they assume audit labels are correct. in reality, human audit labels are often noisy, and only some items get expert review. this paper introduces partially adjudicated design-based supervised learning (pa-dsl) to handle this situation.
pa-dsl works by first using adjudicated cases to correct the noisy human labels. then it uses the corrected audit information to debias analyses based on the full set of automated labels. the estimator is valid for many downstream analyses when the audit and adjudication probabilities are known. experiments on synthetic data and wikipedia detox show that pa-dsl maintains nominal coverage and reduces root mean squared error by 10 to 17 percent compared to using only adjudicated labels.
the method addresses a common problem in applied machine learning: human labelers make mistakes, and expert review is expensive. by combining automated labels, noisy human audits, and a small set of adjudicated examples, pa-dsl provides more accurate and reliable statistical estimates. this can improve the quality of research that relies on automatically labeled data, such as content moderation, sentiment analysis, and other text classification tasks.
why it matters: it allows data scientists to get reliable results from automated classifiers even when human audit labels contain errors, reducing the need for costly expert review.
source: arxiv statistics ml: design-based supervised learning with noisy human labels