Supplemental Material for Interrater Reliability for Multilevel Data: A Generalizability Theory Approach

2021 ◽  
Keyword(s):  
Interrater Reliability ◽  
Generalizability Theory ◽  
Theory Approach ◽  
Multilevel Data

When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching

2021 ◽  
pp. 109821402093194
Author(s):  
Timothy J. Weston ◽  
Charles N. Hayward ◽  
Sandra L. Laursen
Keyword(s):  
Teaching Practices ◽  
Interrater Reliability ◽  
Teaching And Learning ◽  
Generalizability Theory ◽  
Learning Activities ◽  
Observation Data ◽  
Reliable Measure ◽  
Single Class ◽  
Research On Teaching ◽  
The One

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.

scholarly journals Reliability of observers' subjective impressions of families: A generalizability theory approach

2013 ◽  
Vol 23 (4) ◽  
pp. 448-463 ◽  
Author(s):  
Bent Stora ◽  
Knut A. Hagtvet ◽  
Sonja Heyerdahl
Keyword(s):  
Generalizability Theory ◽  
Theory Approach

Inter-Observer, Intra-Observer and Intra-Individual Reliability of Uroflowmetry Tests in Aged Men: A Generalizability Theory Approach

2013 ◽  
Vol 6 (2) ◽  
pp. 76-80 ◽  
Author(s):  
Ying-Buh LIU ◽  
Stephen S. YANG ◽  
Cheng-Hsing HSIEH ◽  
Chia-Da LIN ◽  
Shang-Jen CHANG
Keyword(s):  
Generalizability Theory ◽  
Theory Approach

scholarly journals Estimasi kesalahan pengukuran standard setting dalam penilaian kompetensi matematika tingkat SMP di Kabupaten Sumbawa

2018 ◽  
Vol 22 (1) ◽  
pp. 24-34
Author(s):  
Weni Wendari ◽  
Samsul Hadi
Keyword(s):  
Measurement Error ◽  
Focus Group ◽  
Error Estimation ◽  
Focus Group Discussion ◽  
Generalizability Theory ◽  
Group Discussion ◽  
Standard Setting ◽  
Group Method ◽  
Theory Approach ◽  
Cut Score

Penelitian ini bertujuan untuk mengetahui metode yang lebih akurat dalam mengestimasi kesalahan pengukuran standard setting pada metode Ebel, Bookmark, dan Contrasting group. Data penelitian ini merupakan dokumen Dinas Pendidikan dan Kebudayaan Kabupaten Sumbawa berupa respon peserta Ujian Nasional Matematika Paket P0C5520 tahun ajaran 2015/2016 yang berjumlah 352 siswa. Guru juga dilibatkan dalam penelitian sebagai panelis dalam Focus Group Discussion  (FGD). Data yang terkumpul kemudian dianalisis melaui tiga tahap.. Tahap pertama yaitu persiapan, kegiatan pada tahap ini mencakup penyiapan data, penggolongan SMP, dan penentuan karakteristik butir. Tahap kedua yaitu FGD dilakukan dalam dua putaran. Tahap ketiga yaitu mengestimasi kesalahan pengukuran dengan menggunakan pendekatan Generalizability Theory dengan bantuan program eduG. Hasil penelitian menunjukkan bahwa metode Contrasting group memiliki estimasi kesalahan pengukuran paling kecil dibandingkan metode Ebel dan Bookmark, oleh karena itu, metode Contrasting grouplebih akurat dibandingkan dengan dua metode lainnya.Kata kunci: cut score, standard setting, generalizability theory MEASUREMENT ERROR ESTIMATION OF STANDARD SETTING IN MATHEMATICS COMPETENCY ASSESSMENT FOR JUNIOR HIGH SCHOOL IN SUMBAWA REGENCYAbstractThis research aims to find the most accurate methods in estimating measurement error of standard setting among Ebel, Bookmark, and Contrasting group methods. The data used in this study were 352 students’ responses on Mathematics National Exam Package P0C5520 in the academic year of 2015/2016. The document was collected from the Department of Education and Culture in Sumbawa Regency. Teachers were also involved in this research as panelists in the Focus Group Discussion (FGD). The data collected were then analyzed through three stages. The first stage was preparation stage, including the activities of data preparation, school classification, and item characteristics analysis. The second stage was two-round FGD. The third stage was estimating the measurement error using Generalizability Theory approach assisted by eduG program. The research result shows that Contrasting Group method produces the smallest measurement error estimation compared to Ebel and Bookmark methods, therefore, Contrasting group method is considered as the most accurate method.Keywords: cut score, standard setting, generalizability theory

Sign in / Sign up

Export Citation Format

Share Document