Jesús Moral: "I don't know if Education has neglected the support to the schools that participate in PISA"

BarcelonaJesús Moral (1959, Córdoba) is a mathematics professor and has worked for almost two decades in the Department of Education. He has been head of the Center Organization service, secretary of the School Council of Catalonia, secretary of the Higher Council for Evaluation of the Educational System, and deputy director general of Innovation and Training. In 2018, he was part of the department's team responsible for organizing the PISA tests. In a conversation with ARA, he analyzes the error that has invalidated the inclusion of Catalonia's results in the international tests and which, therefore, makes it impossible to compare the level of Catalan students with that of other territories and with previous years.

Did you imagine that a situation like this could happen?

— No, I didn't expect it. I followed the minister's appearance very carefully, but I lack data, I lack a lot of information to understand what happened. The PISA tests are very important, they cannot be underestimated, and their procedure is very regulated and has a lot of credibility. There are different supervision and control mechanisms so that strange things like what happened do not happen, there are many organizations involved and, therefore, they are very rigorous and very strict in the procedure.

How does it work exactly?

— The Organisation for Economic Co-operation and Development (OECD), which drives the PISA tests, samples and selects the centers that will participate. This selection is communicated to the National Institute for Educational Evaluation and, together with the Department of Education, work is done to verify that the sample is truly representative. In case a center cannot participate, there are substitute centers also chosen by the OECD with the same characteristics so as not to alter the sample. We convened the centers, explained the conceptual framework of the test so that they would know the application criteria and also the exemption criteria set by the OECD.

One of the hypotheses is a possible confusion.

— Yes, due to the coincidence in time between this international assessment (which took place between March 31 and May 16) and the end-of-stage ESO exams (on April 10 and 11). It is true that the OECD exemption criteria are different from local criteria, which include other aspects such as dyslexia or emotional vulnerability, among others, while PISA only considers physical disability, insufficient language proficiency, or cognitive, emotional, or behavioral disability. We took great care not to make them coincide. In the years 2015, 2018, and 2022, the 4th ESO exams were in February and PISA in April-May.

Do you understand that the Government did not know about the incident until March?

— It's strange. It could be, but the Agency for Evaluation and Foresight of Education (APE) should have been very attentive to avoid these incidents and ensure the tests are not biased. It used to be done by the Superior Council for Evaluation, which is now transitioning to the APE, which has assumed its competence, and I don't know if this has meant that the support for participating centers has been neglected. A thorough study is carried out of the centers in the sample, their census, their social composition, the students' language proficiency, and their learning disorders, and then it is validated by the National Institute for Educational Evaluation. One must be very rigorous and apply the criteria appropriately. It's strange, I eagerly await the conclusions of the investigations opened by the department.

Were there ways to detect the error before?

— PISA has a weak point which is the lack of real-time validation before starting the test. But it is the department's responsibility to adequately transmit the information to the schools. The exemption criteria for the test are different from those of our assessments, but the department must know this, must take responsibility, and must anticipate possible incidents. It is their obligation. They must mobilize inspectors and inform directors and teachers. In 2018 we even created a Moodle with the students so they would be familiar with the test before taking it.

What consequences do you think this error will have?

— Not having reliable data has a very serious impact on the educational system because there will be no indicators to guide public policies. The latest results indicated that we were doing badly, and whether there was a recovery or not, we will no longer know. Furthermore, there is the problem of discredit and distrust. Educational centers have gone through a very complicated year and now this serious incident is added to it. In addition, Catalonia has lost the census tests for basic competencies, because they are now sample-based, which is why we are in a complex moment and there is no reliable information to make decisions. We must take advantage of this to open a deep debate on the evaluation model we want in Catalonia because we cannot afford more mismatches in an educational emergency situation like the one we have.