The OECD responds to the experts who found an error in PISA: "The conclusions remain valid"

BarcelonaThe infinite amount of data analyzed by the PISA report and the many ways of analyzing it continue to cause a stir. After researchers from EsadecPol –Esade's center for economic policies– and Innovamat Research Lab warned in a study on Wednesday that there was an error in the PISA report which meant that the assertion that wealthy students had lost more level than vulnerable ones was false, the OECD has responded. In a written response provided to the media, the OECD –the organization that organizes PISA tests around the world– downplays the matter: it points out that the difference between the conclusion of its report and that of the Spanish researchers "reflects a methodological choice and not a methodological error". Although admitting that they have not yet "had the opportunity to review the research in detail", they do defend that "the conclusions of the PISA volume remain valid and consistent".

According to the researchers from EsadecPol and Innovamat, the drop in level of students with a high socioeconomic status, which the OECD itself described as a "new phenomenon" and a "surprise", was due to a change in the way students are classified and not to a drop in points in languages and mathematics. In this sense, the OECD in its statement acknowledges that "different approaches" to defining the socioeconomic situation "can produce different results and that, therefore, one must act with caution when interpreting changes in performance gaps associated with the socioeconomic context".

The organization defends that the team of Spanish researchers conducted the analysis by comparing the vulnerability items that have been repeated throughout the latest editions of PISA, whereas the OECD researchers have used aspects that have been updated based on expert advice to decide year by year which are most relevant for defining the students' socioeconomic situation. In this way, they insist that according to "this basis" the PISA conclusions are "valid and consistent with the methodology used".

The drop is not limited to vulnerable students

While not invalidating the analysis carried out by the Spanish researchers, OECD sources make it clear that "a conclusion that seems to hold across different approaches is that the declines in performance are not limited to students from more socioeconomically disadvantaged families, but are also substantial among students from more affluent families with higher levels of education".

Far from confronting the EsadecPol and Innovamat researchers, the OECD explains that their "findings" may constitute "a useful starting point to delve into this issue". And they add: "The OECD encourages researchers around the world to explore the relevant questions raised by the initial publication of the PISA results and to use these data to strengthen the empirical base that allows for improving the educational outcomes of young people all over the world".