PISA: technical error or human error?

These days we have been surprised by the announcement from the Department of Education according to which the latest PISA results will be published, but they will not be comparable. A technical error is accepted as a reason, stating that the percentage of students excluded from the sample was finally 23%, a figure that far exceeds the 5% considered tolerable.

Indeed, it is not entirely credible that it is a technical error, after more than 25 years of participation (PISA began in 2000). The department's experience in the matter was very high and this type of error is frankly implausible. The facts set off alarms thinking about other motivations.

The PISA test has very interesting contributions for educational systems, but it also generates undesirable attitudes. The first major innovation they developed was to base them on performance approaches not focused on curricular knowledge, but on its application in complex real-world contexts. This meant a change in focus in teaching-learning procedures and caused profound methodological changes in most educational systems. Many of the difficulties students have lie precisely in this change.

It was also sought to make systems comparable and in this way share experiences and try to promote improvements for all. The reality has been that the OECD itself has distorted this function and has focused more on raw results and rankings. The tests give more visibility to entering the culture of competition than to sharing projects and processes of transformation and change.

This latter fact generates attitudes on an international scale in which the aim is not so much to enjoy the best system as the one that scores highest in PISA. We believe that PISA basically tests three competencies (reading comprehension, mathematics, and science and technology) and an educational system is something more. For many, it is fundamentally a system of learning opportunities and of personal and social transformation for all.

Nowadays, with the profound paradigm shift that education is undergoing, simplifying its quality to the three mentioned competencies is nonsense; they should be taken as necessary basic indicators to guarantee its correct functioning, but they do not justify the entirety of educational action.

What happens? Well, since PISA moves out of the educational framework and into the political one, the most important thing is no longer so much having the best educational system as looking good in the photo. And to achieve this, anything goes. There have been systems that have reduced their curricula, focusing them fundamentally on the three mentioned competencies and impoverishing the rest; others have tried to improve results by focusing on the quality of the sample. The OECD has information on many of these practices.

Normally, the procedure involves requesting a list of all schools with 15-year-old students and randomly selecting a sample of about fifty schools, then informing the department, which checks if the selected schools are representative and, if so, validates the sample. It can change 5% of the students if it really believes there is a bias in the sample. Once accepted, the list of students who will participate is requested, and some well-founded exclusions are allowed.

Excluding practically a quarter of the students was an absurd mistake. The OECD and the ministry had the initial list and the exclusion was very evident. What was bound to happen has happened.

The key is that from the department, as soon as the names of the participating centers are known, strategies for support and supervision must be developed. Normally, what had been done was to summon them to a meeting, provide them with information (dates, previous items, documents about PISA, etc.), they had an assigned consultant, and on the days of the test, someone from the department (inspector, technician from the High Council for Evaluation, etc.) was there to support them.

A decision like excluding students had to necessarily go through departmental supervision. It's hard to consider the possibility that some kind of purely technical error occurred. We think that in the previous legislature, poor results were attributed to the presence in the sample of a percentage similar to what they have now excluded, of immigrant students. It is easy to derive the idea that it was necessary to control the presence of vulnerable students, which, in the opinion of some, contaminated the results.

The great contribution of PISA lies in offering standardized and therefore comparable results with our previous ones and with those of other communities and countries. Achieving a real snapshot of the system – and these excluded students, let's not fool ourselves, are part of the system – allows for dissecting the data, studying it dynamically, analyzing it in a disaggregated and joint manner, and ultimately deriving decisions that help, in line with the original intentions of PISA, in decision-making for the transformation and improvement of education.

Now it is not a question of seeking blame, simply of learning from mistakes (technical or human) and trying to improve, in the future, the activity around all evaluation processes. We must manage to overcome the purely procedural vision to fully enter what is called the culture of evaluation.