An error will make the results of Catalonia in the PISA tests not comparable with other territories or previous years

BarcelonaSerious setback in the Catalan education system. The results of the PISA tests in Catalonia, which will be made public in early September, will not be comparable with other territories or with previous editions. In other words, the PISA report will not allow us to see if the level of Catalan students has improved or worsened. The reason is what the Department of Education has described as a "technical, isolated, and isolated incident". In broad terms, Catalonia has not correctly applied the criteria that the OECD — the initiators of the PISA tests — sets to justify a student not passing the tests, which has resulted in the sample of students who did take the exams not being representative of the Catalan student population aged 15. This fact was alerted to the Government in March and communicated this Friday, July 24. The Government has already opened an investigation to clarify the causes of the error.

As explained by the Director General of Curriculum and Professional Development, Sónia Fernández Cuenca, there has been an error in the exemption index, i.e., in the percentage of students who have been considered, either due to physical, cognitive, and emotional disabilities or lack of language proficiency, not to have to take the PISA exams. Specifically, the OECD sets the percentage of exempted students at 5%, and in the 2025 tests in Catalonia, this percentage soared to 23.2%. In other words, the number of students who have not taken the PISA tests in Catalonia is too high for the exam results to represent the actual level of students born in 2009. A priori, the sample for the PISA tests in Catalonia was 2,545 students, and of these, up to 591 were decided not to take the exams: six for physical disability, 92 for insufficient language proficiency, and up to 493 for cognitive, emotional, or behavioral disability.

All of this will mean that the results will not be representative and that, according to the Department of Education and the Ministry of Education, they cannot be compared with the scores of other territories or with Catalonia's own results in previous years.

However, this does not mean that Catalonia will not receive the results of the PISA tests. In fact, the average obtained by Catalan students who have taken the test will be used to determine Spain's results in the OECD report because, according to the Government, the error in the sample does not affect the representativeness at the state level. "If the sample is not balanced, the results are not comparable, but they are valid," defended Fernández Cuenca. He also assured that errors like the one perpetrated in Catalonia are "a phenomenon that the OECD itself has had to manage on other occasions".

"We are facing a technical error, not a problem with the students' level or the quality of the educational system. There has been no manipulation or concealment of data," insisted the department representative on several occasions during the appearance she made out of "responsibility and transparency".

What happened?

Although the department has assured that the reasons for the error are still being investigated, they have put forward several facts that warn that they are "not conclusions", but which provide contextual information around the error. One of these "contextual" facts is that, as detailed by Fernández Cuenca, the PISA tests coincided in a very short time with the end-of-stage tests for 4th year of ESO —the old basic competencies— and the Catalan tests have broader criteria than those of the OECD when deciding which students are exempt from taking the exam. This could have led educational centers themselves to have "confusion" when applying the correct criteria to each test.

Here, it should be taken into account that, as Education has described, although the OECD, two companies, Spain and the communities are involved in the circuit to determine which students pass the tests and which are exempt, the final step in applying the exemption criteria lies with the educational centers themselves.

Another of the facts admitted by the Director General of Curriculum is that the application of the PISA tests —from May 31 to May 16— coincided with a "transition process" between the Higher Evaluation Council (the body that had been responsible for applying the tests for years) and the Agency for Evaluation and Prospective of Education (the new body on which the evaluations of the education system depend). "At the time of the events, the agency's creation process was still underway," admitted Fernández Cuenca.

And now what?

According to details provided by the Government, in March the OECD alerted the ministry to an error in the Catalan sample, and they communicated this to the director of the Evaluation Agency. From then on, the department "immediately" opened "confidential preliminary information proceedings" and this process to gather information "is still open". "I can state that it is an isolated case," justified the Director General of Curriculum. Furthermore, she also assured that the Government "has evidence" that it is an isolated case because this year state, international (PIRLS), and Catalan tests have also been conducted, and in all of them the exemption rates "have been within the margin of the threshold set by each test".

Education has also insisted that its priority now is to "ensure that [an error like this] is not repeated". In this regard, Fernández Cuenca explained that "this very day" a body "non-existent until now" has been created to order and apply each test in the system, establishing "protocols, action circuits, and alert thresholds for each of the tests". The objective of this body, named the Commission for the coordination of evaluation tests and studies, will be to ensure that "the validation of exemptions—when a student is exempt from taking the exams—is done "always and for all tests". "Until now this depended on informal circuits, and now it will correspond to a stable order that will have a specific protocol to manage any incident with sufficient advance notice," assured the representative of the department.