Comparison Of Johnson Neyman Mantel Haenszel And Partial Correlation Procedures For Detecting Differential Item Performance For Moderate Sample Size

Comparison of Johnson-Neyman, Mantel-Haenszel and Partial Correlation Procedures for Detecting Differential Item Performance for Moderate Sample Size

A Comparison of the Reliability of Item Response Theory and Mantel-Haenszel Chi-square Procedures for Detecting Differential Item Functioning

The Effects of Matching Criterion Contamination on the Mantel-Haenszel Procedure

Modern bias detection procedures search for differences in item performance between demographic groups after conditioning on an estimate of the ability intended to be measured by the test. The estimate of ability is typically some function of the total test score. Since examinees with equal test scores are said to be matched on ability, the internal measure of ability is referred to as the matching criterion. When the test contains one or more biased items, the test score will not be a valid measure of ability. As a result, the matching criterion is said to be contaminated by the biased items. This study consists of a comprehensive examination of the Mantel-Haenszel (MH) procedure in the presence of a contaminated matching criterion. This examination focused on two primary issues: (1) assessing the effects of contamination on the MH procedure, and (2) developing alternative DIF detection procedures which are robust to contamination. The results indicate that the presence of contamination has minor effects on the MH procedure when contamination is small or moderate, but has substantial effects when contamination is large. The effects were related to the proportion of items containing contamination, and increased as the level of DIF in the contaminated items increased. Two solutions to the problem of matching criterion contamination solutions were proposed. First, a procedure was developed that adjusts the obtained MH value to correct for the effects of contamination. The results of a simulation study suggest that the adjustment is effective in general at correcting for the effects of contamination, losing efficiency only under the most severe levels of contamination and the smaller sample size ('N' = 250). A second solution to the problem of contamination was the proposal of ' MB-DIF', a new statistic that is theoretically robust to the effects of contamination. The results of a simulation study indicate that the performance of 'MB-DIF' exceeds that of the MH adjustment, particularly when sample sizes were large ('N' = 1000). Under the condition of large sample sizes, 'MB-DIF' was completely robust to the effects of contamination, maintaining power and Type I error rates identical to control conditions in which no contamination existed. The performance of ' MB-DIF' suffered slightly when sample sizes were small ('N ' = 250), largely due to inflated Type I error rates under large levels of contamination. The findings have two implications. First, bias detection analyses should consider the possible magnitude of bias in other items in the test when investigating the magnitude of bias in any given item. Second, adjustment procedures can control for the majority of the underestimation in DIF statistics when the matching criterion is contaminated. It appears that 'MB-DIF' offers a more effective solution to the problem of matching criterion contamination than adjusting the MH value.