Validation of scales for measuring factors of teaching quality from the dynamic model of educational effectiveness

Large-scale educational effectiveness research requires valid student questionnaires to assess teaching practices. This research validated eight scales for measuring teaching factors from the Dynamic Model of Educational Effectiveness (DMEE). Parallel versions of scales for measuring teaching factors in mathematics and biology were constructed and validated in two studies. In the first study, an exploratory factor analysis was conducted on data from 683 students. In the second study, the structure was cross-validated via a confirmatory factor analysis (CFA) on a sample of 5,476 students. The multi-group CFA resulted in an acceptable metric invarience for all scales, indicating that the scales have comparable factor loadings. However, unsatisfactory scalar invariance suggested that the scales could not be used to compare teachers of different subjects. Testing alternative structural relations between the teaching factors did not confirm that the data fit the DMEE model adequately, although the fit parameters were better than for the alternative theoretical models. For mathematics, the external validation of the scales showed that the scales correlated with job satisfaction, external control, and teacher self-efficacy reported by the teachers. The scales are reliable and valid and could be applied to different school subjects.

The Design of Clustered Observational Studies in Education

Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would have been conducted were it feasible. We emphasize the key role of understanding the assignment mechanism to study design. We review methods for statistical adjustment and highlight a recently developed form of matching designed specifically for COSs. We review how regression models can be profitably combined with matching and note best practices for estimates of statistical uncertainty. Finally, we review how sensitivity analyses can determine whether conclusions are sensitive to bias from potential unobserved confounders. We demonstrate concepts with an evaluation of a summer school reading intervention in a large U.S. school district.