EsadecPol and Innovamat find an error in PISA: wealthy students would not have lost more level than vulnerable ones

BarcelonaA surprising plot twist in the PISA test results. Researchers from EsadecPol –Esade's center for economic policies– and Innovamat Research Lab have found an error in the analysis of the OECD's educational tests. The error implies that the claim that wealthier students had lost more level than vulnerable ones would be false. According to these researchers, the drop in level of students with a high socioeconomic status, which the OECD itself qualified as a "new phenomenon" and a "surprise", is due to a change in the way students are classified and not a drop in points in languages and mathematics. For the moment, the OECD has not commented on this study.

According to the researchers, if the students' socioeconomic situation is measured in the same way as in other editions of the PISA report, the result is that all groups of students lose level in a similar way and, contrary to what had been concluded, the gap between the wealthiest and the most disadvantaged students is the highest of the decade in Spain and in the OECD as a whole. To give an example, according to this new research, in the OECD as a whole, the wealthiest students would have lost six points in mathematics and not 14 as the PISA report concludes. "The global decline in performance is real in OECD countries, but this drop is not due to a greater drop in the results of students with a higher socioeconomic level, but rather it occurs in a similar way in all groups. This forces a reinterpretation of the results and the establishment of new hypotheses for the causes of the drop that are affecting all students equally," insist the authors of the study, made public this Tuesday.

From asking about books to knowing if there is a swimming pool

Let's take it step by step. The OECD classifies students into four groups based on the ESCS (the economic, social, and cultural status index), which is calculated by averaging the parents' education level, their occupational status, and an index of material possessions in the student's home. However, in the 2022 edition – the one made public just a few weeks ago – eleven of the questions usually asked in the household possessions section were removed, including questions about the types of books in the students' homes. These questions were replaced by other inquiries, such as whether there is a guest room, a swimming pool, running water, or electricity. Elements that, according to researchers at EsadecPol and Innovamat Research Lab, are "almost universal in the OECD and have less relation to academic performance".

The change in the way vulnerability is calculated would have meant that a "considerable" proportion of students with good PISA results and high cultural capital, but lower living conditions, do not appear in the group of more affluent students, whereas until now they did appear there. Similarly, students with worse PISA results and lower cultural capital, but with higher living conditions, entered the group of more affluent students, whereas until now they were part of another group. "As a consequence of these changes, the decline observed in 2022 with the socioeconomic index published in the report was greater than in the rest of the groups," the research concludes.

According to the authors of the study, the change in the formula for calculating the ESCS means that the level of students, according to this index, cannot be compared between the 2022 and 2025 editions. In this sense, the research points out that while the technical document of the PISA report did warn that this change invalidated the comparison, later in the general report – Volume I of PISA, to which everyone had access – the OECD did compare the statistics and drew conclusions from them.