scholarly journals International Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews

2012 ◽  
Vol 2012 ◽  
pp. 1-11
Author(s):  
Inga Arffman
Keyword(s):  
International Education ◽  
Cognitive Processes ◽  
International Student ◽  
Student Assessment ◽  
Reading Test ◽  
Language Test ◽  
Written Responses ◽  
International Student Assessment ◽  
Item Functioning ◽  
Control Procedures

In international education studies, the different-language test versions need to be equally difficult to read and answer for the test to be valid. To ensure comparability, several quality control procedures have been developed. Among these, surprisingly little attention has been paid to judgmental reviews and their ability to identify language-related sources of bias. Also, the reviews have often failed in identifying biases. This paper explored whether it is possible to improve the ability of judgmental reviews to identify language-related sources of bias. A new review was made of two Finnish items which in the PISA (Programme for International Student Assessment) 2000 reading test showed differential item functioning but for which no clear language-related explanations were found in the review in 2000. The items were compared systematically, at all linguistic levels, to the corresponding items in the English and French source versions, at the same time taking into account the cognitive processes required to answer them and students’ written responses to them. Language-related explanations were found for both items which may have led to differences in performance, suggesting that it is possible to make judgmental reviews better able to identify language-related bias. Suggestions are given on how to do this.

scholarly journals Avaliações e pesquisas educacionais internacionais: entrevista com Andreas Schleicher

2017 ◽  
Vol 28 (68) ◽  
pp. 630
Author(s):  
Gabriela Moriconi ◽  
Adriana Bauer
Keyword(s):  
International Education ◽  
Comparative Research ◽  
International Student ◽  
Student Assessment ◽  
Educational Programs ◽  
Secretary General ◽  
International Student Assessment ◽  
Palabras Clave ◽  
International Assessment

<p>Nesta entrevista, Andreas Schleicher, Diretor de Educação e Assessor Especial em Política Educacional da Secretaria Geral da Organização para a Cooperação e Desenvolvimento Econômico (OCDE), comenta sobre o papel da OCDE e, em especial, do Programme for International Student Assessment (PISA), no contexto educacional internacional, sua relação com atores da comunidade educacional, bem como os impactos de sua atuação. A entrevista aborda, ainda, aspectos relativos à participação do Brasil em programas e projetos educacionais da OCDE.</p><p><strong>Palavras-chave:</strong> Avaliação Internacional; Pesquisa Comparada; Pisa; OCDE.</p><p>  </p><p><strong>International education assessments and surveys: interview with Andreas Schleicher</strong></p><p>In this interview, Andreas Schleicher, Director for Education and Sills, and Special Advisor on Education Policy to the Secretary-General at the Organisation for Economic Co-operation and Development (OECD), comments on the role of the OECD and, in particular, of the Program for International Student Assessment (PISA), in the international education context, its relationship with actors in the educational community, as well as the impacts of its performance. The interview also addresses aspects related to Brazil’s participation in OECD educational programs and projects.</p><p><strong>Keywords:</strong> International Assessment; Comparative Research; Pisa; OECD.</p><p>  </p><p><strong>Evaluaciones e investigaciones educativas internacionales: entrevista con Andreas Schleicher</strong></p><p>En esta entrevista, Andreas Schleicher, Director de Educación y Asesor Especial en Política Educacional de la Secretaría General de la Organización para la Cooperación y Desarrollo Económico (OCDE), comenta sobre el papel de la OCDE y, en especial, del Programa Internacional de Evaluación de Estudiantes (PISA), en el contexto educativo internacional; su relación con actores de la comunidad educacional; así como los impactos de su actuación. La entrevista aborda asimismo aspectos relativos a la participación de Brasil en programas y proyectos educativos de la OCDE.</p><strong>Palabras-clave:</strong> Evaluación Internacional; Investigación Comparada; PISA; OCDE

Examining Sources of DIF in Psychological and Educational Assessment Using Multilevel Logistic Regression

Methodology ◽  
2014 ◽  
Vol 10 (2) ◽  
pp. 71-79 ◽  
Author(s):  
Nekane Balluerka ◽  
Ian Plewis ◽  
Arantxa Gorostiaga ◽  
José-Luis Padilla
Keyword(s):  
Logistic Regression ◽  
International Student ◽  
Student Assessment ◽  
Educational Assessment ◽  
Multilevel Logistic Regression ◽  
Multilevel Regression ◽  
International Student Assessment ◽  
Item Functioning ◽  
Item Characteristics ◽  
Item Bundles

In the last three decades, important progress has been made toward more efficient statistical techniques for detecting Differential Item Functioning (DIF). However, the findings are scant when it comes to explaining DIF. Multilevel regression models can expand the knowledge of DIF causes, specifying a DIF parameter that varies randomly over items and testing hypotheses on sources of DIF shared by item bundles. The present study uses multilevel logistic regression to identify the item characteristics that could explain the presence of DIF in short tests or questionnaires, which are usually used in psychological and educational assessment. The usefulness of the approach is tested on measurements of the attitudes toward science of Spanish and English pupils obtained from the OECD Programme for International Student Assessment database.

The Reliability of Trends over Time in International Education Test Scores: Is the Performance of England's Secondary School Pupils Really in Relative Decline?

2012 ◽  
Vol 42 (2) ◽  
pp. 259-279 ◽  
Author(s):  
JOHN JERRIM
Keyword(s):  
Secondary School ◽  
International Education ◽  
Test Performance ◽  
International Student ◽  
Student Assessment ◽  
Educational Systems ◽  
International Student Assessment ◽  
Data Limitations ◽  
Over Time ◽  
Secondary School Pupils

AbstractThe Programme for International Student Assessment (PISA) and Trends in International Mathematics and Science Study (TIMSS) are respected cross-national studies of pupil achievement. They have been specifically designed to study how countries’ educational systems are performing against one another, and how this is changing over time. These are, however, politically sensitive issues, where different surveys can produce markedly different results. This is shown via a case study for England, where apparent decline in PISA test performance has caused policymakers much concern. Results suggest that England's drop in the PISA ranking is not replicated in TIMSS, and that this contrast may be due to data limitations in both surveys. Consequently, I argue that the current coalition government should not base educational policies on the assumption that the performance of England's secondary school pupils has declined over the past decade.

scholarly journals Teacher Praise and Encouragement: Towards an Education for Democracy

2008 ◽  
Vol 2 (1) ◽  
Author(s):  
Herner Saeverot
Keyword(s):  
International Education ◽  
International Student ◽  
Student Assessment ◽  
Science Study ◽  
Teacher Praise ◽  
International Student Assessment ◽  
Education For Democracy ◽  
Goals And Objectives ◽  
Mathematics And Science

International education tests, such as the Organization for Economic Co-operation and Development (OECD) International Student Assessment (PISA) and Trends in International Mathematics and Science Study (TIMSS), steer the debate about education, form the image of a school, and specify the goals and objectives of the curriculum. Thus praise, prizes and other positive reinforcements become common methods in school. Through Franz Kafka's Letter to his Father I examine encouragement and teacher praise, which are, when we look into it, held in the grip of behaviourism. I take issue with this dominant thinking and make an attempt to go beyond praise and encouragement as positive reinforcements of desired behaviours that are postulated in advance. I make an educational argument out of Heidegger's phenomenological term, Ereignis, and suggest that teachers ought to praise that which the student is suited for and not only when they have fulfilled the demands of the curriculum. I also make an educational argument out of two Derridaean terms, hospitality and forgiveness, and claim that teachers ought to give back -- through praise and encouragement -- the students' self-respect, whatever their faults. Instead of being manipulative and serving narcissistic needs, praise ought to serve an education for democracy.

scholarly journals Funcionamento diferencial dos itens de ciências do PISA: Brasil e Japão

2017 ◽  
Vol 28 (68) ◽  
pp. 538
Author(s):  
Andriele Ferreira Muri ◽  
Tufi Machado Soares ◽  
Alicia Bonamino
Keyword(s):  
Differential Item Functioning ◽  
Scientific Literacy ◽  
International Student ◽  
Student Assessment ◽  
Differential Functioning ◽  
Japanese Students ◽  
International Student Assessment ◽  
Item Functioning ◽  
Brazilian Students ◽  
International Comparative

<p>O Programa Internacional de Avaliação dos Estudantes (PISA) é uma avaliação comparada, aplicada a uma amostra de estudantes de 15 anos de idade. Juntamente com vários outros países, Brasil e Japão participam desde a primeira edição, em 2000. Com o objetivo de identificar fatores capazes de explicar as diferenças de resultados encontradas no Letramento em Ciências, entre alunos brasileiros e japoneses, na edição de 2006, foi utilizada a análise de DIF (Differential Item Functioning), que possibilitou extrair, dos resultados dos testes, padrões de efeitos diferenciados. Para identificar os itens que apresentaram funcionamento diferencial entre Brasil e Japão, empregou-se o modelo bayesiano integrado que, além de confirmar a ocorrência, também pode explicar o DIF. Encontramos DIF em todas as covariáveis elegidas, embora nem sempre esse comportamento diferencial tenha privilegiado um dos dois países. Há competências que discriminam mais os alunos brasileiros e áreas de aplicação dos itens ora mais fáceis para o Brasil, ora para o Japão.</p><p><strong>Palavras-chave:</strong> Differential Item Functioning – DIFF; PISA; Letramento Científico; Ciências.</p><p> </p><p><strong>Funcionamento diferencial de los ítems de ciencias del PISA: Brasil y Japón</strong></p><p>El Programa Internacional de Evaluación de Estudiantes (PISA) es una evaluación comparada, aplicada a una muestra de estudiantes de 15 años de edad. Junto con varios otros países, Brasil y Japón participan desde la primera edición, en el 2000. Con el objetivo de identificar factores capaces de explicar las diferencias de resultados encontradas en el Letramento en Ciencias entre alumnos brasileños y japoneses en la edición de 2006, se utilizó el análisis de DIF (Differential Item Functioning), que hizo posible que se extrajera de los resultados de las pruebas patrones de efectos diferenciados. Para identificar los ítems que presentaron funcionamiento diferencial entre Brasil y Japón se utilizó el modelo bayesiano integrado que, además de confirmar la ocurrencia, también puede explicar el DIF. Encontramos DIF en todas las covariables elegidas, aunque no siempre este comportamiento diferencial privilegie uno de los dos países. Hay competencias que discriminan más a los alumnos brasileños y a las áreas de aplicación de los ítems, que a veces son más fáciles para Brasil y otras para Japón.</p><p><strong>Palabras-clave:</strong> Differential Item Functioning – DIFF; PISA; Letramento Científico; Ciencias.</p><p> </p><p><strong>Differential functioning of PISA science items in Brazil and Japan</strong></p><p>The Programme for International Student Assessment (PISA) is an international comparative assessment program applied to samples of 15-year-old students. Together with other countries, Brazil and Japan have participated in this program since its first edition in 2000. DIF (Differential Item Functioning) analysis was used to identify factors that could explain performance differences in scientific literacy between Brazilian and Japanese students, in the 2006 edition. Based on the test results, this analysis showed patterns of differentiated effects. To identify the items that showed differential functioning between Brazil and Japan, we used the Bayesian integrated model. In addition to confirming this occurrence, this model may also explain the DIF. DIF was found in all covariates selected. However, differential functioning did not always favor either of the two countries. There are competencies that discriminate more against the Brazilian students and areas of application of the items that sometimes were easier for Brazilian and sometimes for Japanese students.</p><strong>Keywords:</strong> Differential Item Functioning – DIFF; PISA; Scientific Literacy; Sciences.

Three Paradoxical Effects on Academic Self-Concept Across Countries, Schools, and Students

2019 ◽  
Vol 24 (3) ◽  
pp. 231-242 ◽  
Author(s):  
Herbert W. Marsh ◽  
Philip D. Parker ◽  
Reinhard Pekrun
Keyword(s):  
International Student ◽  
Student Assessment ◽  
Cross Cultural ◽  
Frame Of Reference ◽  
School Level ◽  
Self Concept ◽  
Positive Effects ◽  
International Student Assessment ◽  
Country Level ◽  
Starting School

Abstract. We simultaneously resolve three paradoxes in academic self-concept research with a single unifying meta-theoretical model based on frame-of-reference effects across 68 countries, 18,292 schools, and 485,490 15-year-old students. Paradoxically, but consistent with predictions, effects on math self-concepts were negative for: • being from countries where country-average achievement was high; explaining the paradoxical cross-cultural self-concept effect; • attending schools where school-average achievement was high; demonstrating big-fish-little-pond-effects (BFLPE) that generalized over 68 countries, Organisation for Economic Co-operation and Development (OECD)/non-OECD countries, high/low achieving schools, and high/low achieving students; • year-in-school relative to age; unifying different research literatures for associated negative effects for starting school at a younger age and acceleration/skipping grades, and positive effects for starting school at an older age (“academic red shirting”) and, paradoxically, even for repeating a grade. Contextual effects matter, resulting in significant and meaningful effects on self-beliefs, not only at the student (year in school) and local school level (BFLPE), but remarkably even at the macro-contextual country-level. Finally, we juxtapose cross-cultural generalizability based on Programme for International Student Assessment (PISA) data used here with generalizability based on meta-analyses, arguing that although the two approaches are similar in many ways, the generalizability shown here is stronger in terms of support for the universality of the frame-of-reference effects.

Are There Test Administrator Effects in Large-Scale Educational Assessments?

Methodology ◽  
2007 ◽  
Vol 3 (4) ◽  
pp. 149-159 ◽  
Author(s):  
Oliver Lüdtke ◽  
Alexander Robitzsch ◽  
Ulrich Trautwein ◽  
Frauke Kreuter ◽  
Jan Marten Ihme
Keyword(s):  
Large Scale ◽  
International Student ◽  
Student Assessment ◽  
Student Groups ◽  
Mathematics Scores ◽  
International Student Assessment ◽  
Sample Attrition ◽  
Educational Assessments ◽  
Test Administrator ◽  
Practical Implications

Abstract. In large-scale educational assessments such as the Third International Mathematics and Sciences Study (TIMSS) or the Program for International Student Assessment (PISA), sizeable numbers of test administrators (TAs) are needed to conduct the assessment sessions in the participating schools. TA training sessions are run and administration manuals are compiled with the aim of ensuring standardized, comparable, assessment situations in all student groups. To date, however, there has been no empirical investigation of the effectiveness of these standardizing efforts. In the present article, we probe for systematic TA effects on mathematics achievement and sample attrition in a student achievement study. Multilevel analyses for cross-classified data using Markov Chain Monte Carlo (MCMC) procedures were performed to separate the variance that can be attributed to differences between schools from the variance associated with TAs. After controlling for school effects, only a very small, nonsignificant proportion of the variance in mathematics scores and response behavior was attributable to the TAs (< 1%). We discuss practical implications of these findings for the deployment of TAs in educational assessments.

Sign in / Sign up

Export Citation Format

Share Document