Testing Inter-Rater Reliability in Rubrics for Large Scale Undergraduate Independent Projects

This work outlines the process of testinginter-rater reliability in rubrics for large scaleundergraduate independent projects; more specifically,the thesis program within the Division of EngineeringScience at the University of Toronto, in which 200students work with over 100 supervisors on anindependent research project. Over the last few years,rubrics have been developed to both guide the students inthe creation of their thesis deliverables, and to improvethe consistency of supervisor assessment. To examineinter-rater reliability, 12 final thesis reports wereassessed using the course rubric by the two generalistexperts, who have worked extensively with the thesiscourse and designed the rubrics, alongside the projectsupervisor. We found substantial agreement between thetwo generalist experts, but only fair agreement betweenthe generalist experts and the supervisors, suggesting thatwhile the rubric does help towards developing a commonset of expectations, there may be other aspects of thesupervisor’s assessment practice that need to beconsidered.

Projected Federal Government Computer Personnel Needs: Results Of An Oklahoma Survey

The purpose of this paper is to report the results of a study made to determine the projected performance requirements and skills (training needed) by personnel entering governmental information processing jobs during the next several years as viewed by supervisors in governmental agencies. The specific problems addressed are as follows:1. The identification of the performance requirements for computer programmers and systems analysts which government employers expect to be needed for employment in the field of information processing during the last have of the 1980s;2. The effectiveness of currently employed computer programmers and systems analysis as determined by supervisor assessment;3. Identification of the areas in which colleges and universities will need to make curricular changes in order to provide the government as well as the business community with properly trained professionals in information processing.