A Comparison Study Of The Reliability Coefficients From Five Approaches To Reliability Estimation

A Comparison Study of the Reliability Coefficients from Five Approaches to Reliability Estimation

In order to estimate reliability by a single administration of one test form, various approaches and corresponding reliability coefficients have been proposed so far. Currently, the five most influential approaches are: internal consistency, lower bound, principal components analysis (PCA), exploratory factor analysis (EFA), and structural equation modeling (SEM). Facing various approaches and thus dozens of reliability coefficients derived for estimating reliability, practicing researchers are curious to know which reliability coefficient(s) performs best, and under what circumstances. However, a comprehensive comparison of the reliability coefficients from the aforementioned five approaches has not been conducted yet. Therefore, a Monte Carlo study was conducted to evaluate the performances of the reliability coefficients from the five approaches under the conditions that are known to have effect on reliability estimation. Monte Carlo design factors included twelve specific measurement models, two levels of item number, three levels of sample size, three levels of error correlation, and two levels of factor correlation. In total, 72 simulation conditions were created by the combination of all design factors, and each condition was replicated 1,000 times in R environment. The results were collected in two stages. In the first stage, the percentage relative bias, standard error and root mean square error of each reliability coefficient were calculated for each condition. The rounded percentages of estimation failure numbers for each SEM reliability coefficient under all the manipulated conditions were also obtained to identify the conditions with serious estimation issues for the second stage analysis. In the second stage of this study, the percentage relative bias, standard error and root mean square error of Bayesian SEM estimates of reliability for the selected conditions were calculated. Results showed that correctly specified SEM estimates of reliability were least biased and comparatively stable under most of the conditions across the twelve measurement models in this study. However, under the conditions of small item numbers and complicated models, correctly specified SEM estimates of reliability were least accurate and exceptionally unstable due to estimation problems. In addition, over-specified SEM estimates of reliability were examined under the conditions in Model 1 (the tau-equivalent model with independent errors), Model 4 (the congeneric model with independent errors), Model 7 (the correlated factor model with factor correlation at 0.2 and independent errors) and Model 10 (the correlated factor model with factor correlation at 0.6 and independent errors). Results indicated that over-specified SEM estimates of reliability were as accurate and stable as correctly specified SEM estimates of reliability unless estimation problems occurred. Results in the second stage showed that the Bayesian estimation method with non-informative priors could effectively solve estimation problems but fail to eradicate the biases in SEM estimates of reliability. In order to solve estimation problems as well as maintaining the accuracy of SEM estimates of reliability, more types of priors need be tested and compared when using Bayesian estimation methods in a future study.

Aviation Psychology Program Research Reports

Quantitative and Qualitative Methods in Psychotherapy Research

In this collection, international contributors come together to discuss how qualitative and quantitative methods can be used in psychotherapy research. The book considers the advantages and disadvantages of each approach, and recognises how each method can enhance our understanding of psychotherapy. Divided into two parts, the book begins with an examination of quantitative research and discusses how we can transfer observations into numbers and statistical findings. Chapters on quantitative methods cover the development of new findings and the improvement of existing findings, identifying and analysing change, and using meta-analysis. The second half of the book comprises chapters considering how qualitative and mixed methods can be used in psychotherapy research. Chapters on qualitative and mixed methods identify various ways to strengthen the trustworthiness of qualitative findings via rigorous data collection and analysis techniques. Adapted from a special issue of Psychotherapy Research, this volume will be key reading for researchers, academics, and professionals who want a greater understanding of how a particular area of research methods can be used in psychotherapy.