Understanding the PISA Tables

The PISA math results show Canada trending downwards internationally. Here are the rankings of the top 20 regions (of approximately 65) that participated in the 2003, 2006, 2009, and 2012 rounds:

Note:  Shanghai and Singapore have been greyed out for 2003 and 2006 because neither took part in those two rounds. Since it is hard to assess a trend when the population base changes, I think that either Shanghai and Singapore should be included for all rounds, or both should be excluded for all rounds. I have chosen to include them, and, based on their results in 2009 and 2012, I have assumed that they would have finished atop the PISA table had they competed in the earlier rounds.

With Shanghai and Singapore included, the perspective on the Canadian trend changes somewhat. However, some slippage is definitely happening, and for many people the trend is alarming.

But I wonder if we are concentrating too closely on the rankings. Here is the same table when grades are included:

Note: OK. This doesn't look like the PISA scores that you have seen. The results shown above are percentage grades, and the PISA consortium does not report the scores this way. Instead, they use a numerical scale that ranges from about 300 to over 600.

I have a fondness for percentage grades. They provide me with a more serviceable scale than grades out of 600. For example, to me, a class average of 67% indicates an ordinary but acceptable performance, while I am not sure what an average of 527 on the PISA scale conveys.

The conversion from the PISA scale to percentage grades in the table above was done linearly:

Percent_grade = 60 + (Pisa_Score − 482) × (10 / 62.3).

As is evident, Canada has remained remarkably stable: 68%, 67%, 67%, 66%. 

Proficiency Levels

The PISA results are intended to be a measure of mathematical proficiency, and PISA describes six different levels. The grade ranges for each level are as follows:

The descriptions above are extremely abbreviated. Detailed descriptions can be found in the 2012 PISA technical report (link [5] below). Also, PISA does not use verbal descriptions like Highly proficient or Basic proficiency.

Some features of the PISA proficiency levels

• For each round, the PISA consortium adjusts the grades so that the proficiency levels retain the same meaning over the years. A percentage score of 65% in 2003 has exactly the same meaning as a percentage score of 65% in 2012. In other words, PISA takes care to avoid grade inflation.

• On the percentage scale, each Level band is 10 points wide. On the PISA scale, the bands are 62.3 points wide.

• The PISA scales are designed so that the OECD mean is 500 (or very close to 500). On the percentage scale, the OECD mean is 63%.

• Over the four rounds from 2003 to 2012, Level 6 has never been achieved by any of the participating regions. Level 5 has been achieved by one participant. Level 4 has been achieved by five participants.  In the four rounds from 2003 to 2012, Canada has resided in the Level 3 band.

Is the slippage significant?

The difference between Canada’s score in 2003 and 2012 is statistically significant. The words mean that, with high probability, the difference in the scores is real and is not a result of the chance that arises by testing only a sample of the population. Statistical significance tells us that the statistics are reliable, but it doesn't tell us anything about whether or not the statistics are important.

(The word "significant" has considerable emotional impact. I remember a Statistics professor telling us that the word should be outlawed when we are discussing statistical results.)

Do the differences in the PISA tables show that Canada must do something to improve how our students learn mathematics? Some people believe that the statistically significant difference between 2003 and 2012 is also of great practical significance, and they have strong opinions about what should be done.

As well, the PISA group itself has investigated what will and won't work to improve students mathematical proficiency, and has published its findings in a 96 page document [6]. Some of the conclusions appear to run counter to what the back-to-basics people advocate — one conclusion in the document is that dependence on intensive memorization and rote practice of the basic algorithms is a poor strategy for learning anything but the most trivial mathematics.

That particular PISA document may leave some of my friends in a slightly awkward place: Those who have used the PISA tables as a rallying cry for a return to traditional methods now find themselves having to rebut some of PISA's own conclusions about what constitutes good teaching and good curriculum.

In any event, when I look at the percentages for Canada from 2003 to 2012, I see changes, but I don't think that they are earth-shattering. Would you be concerned if one section of your Calculus course averaged 68% and another section averaged 66%? I’ve run into that situation teaching back-to-back sections of the same course, and I was not convinced that the section with the higher score was the superior one.

* * *

In a few weeks, the OECD will release the results of the 2015 PISA round. I am very interested in seeing if Canada's results are significantly different in a practical sense. My expectation (based on no evidence whatsoever) is that the results will be much the same as before, and will fall within the 65% to 67% range. ( Hey! A Halifax weatherman had a very successful forecasting run by predicting tomorrow's weather to be the same as today's.)  I'll do a Trumpian gloat if my prediction is correct, and a Clintonian sob if it's wrong.

References

[1] 2003 PISA report (Math scores are on page 92)

[2] 2006 PISA report (Math scores are on page 53)

[3] 2009 PISA report (Math scores are on page 134)

[4] 2012 PISA report (Math scores are on page 5)

[5] PISA 2012 Technical report (Proficiency levels are described on pages 297 - 301)

[6] Ten Questions for Mathematics Teachers… and How PISA Can Help Answer Them, OECD Publishing, Paris (2016).