The Meaning of the 2015 PISA Canadian Math Results

For the past few years various news media have repeatedly conveyed the belief that:

1. The mathematical abilities of Albertan and Canadian students have declined significantly.
2. This decline was caused by the introduction of a new curriculum.
3. Evidence is provided by the results of the PISA tests.

That sentiment is what this post is about, but let me start with this question.

At a certain junior high school the averages of the math grades for the past three years were 73%, 75%, and 73%. How would you rate this school?

1. Definitely a top notch school.
2. That’s a pretty good school.
3. That’s what an average school should be like.
4. Mediocre — room for improvement.
5. Poor — I would think about sending my child to a different school.

Myself, I would choose (2).  I would expect grades of 65% - 70% to be about average so I feel the school is a pretty good one. Not absolutely fantastic, but pretty good.

The real story:

The averages cited above are not for a particular school, but for an entire country. The figures are the average math scores for top notch Singapore in the 2009, 2012, and 2015 rounds of the PISA tests. (If you are familiar with the PISA tests, you know that the scores are not recorded as percentages. The numbers 73%, 75%, and 73% are what you get when you convert the published PISA scores in a reasonable way.¹ )

For comparison, here are the Alberta and Canadian results for the years from 2003 through 2015, again converted to percentages. (Singapore did not compete prior to 2009).

The variation across the PISA rounds for all three regions seems pretty unexceptional. I would expect changes of ±3% at the very least — the year-to-year variability in my university math classes were at least as large as that.

[As an aside, I predicted in a previous post that Canada would get a grade in the range from 65% to 67% on the math part of the 2015 PISA test. I promised a Trumpian gloat if the prediction was correct, and a Clintonian sob if it was wrong. No guru here, I’m afraid — it was pure luck — but "Gloat. Gloat" anyway.]

In spite of the stable relationship between the results for Alberta and Canada, there is alarm that Alberta's performance in the 2015 round has pushed them below the rest of Canada. That may be numerically accurate when you look at the raw scores published by PISA, but the results are so close that when converted to percentages the difference vanishes. And, more importantly, the PISA consortium itself reports that the difference is so minuscule that it is not statistically significant. Which means there is no reliable evidence that there is any difference whatsoever.  

I don’t think that our ranking among the PISA participants indicates that our math curriculum is either superb or deficient. But using the PISA rankings to critique the curriculum is what people do. To confirm this, all you have to do is read the opening foray in the petition that stoked the current math wars in Alberta. Here is what it says:

"The Organization for Economic Co-operation and Development (OECD) recently released a report confirming that math scores, measured in 15yr olds, have declined significantly in the ten years since the new math curriculum was introduced across Canada."

Additionally, there is this report in the Globe and Mail in January 2014:

Robert Craigen, a University of Manitoba mathematics professor who advocates drilling basic math skills and algorithms, said Canada’s downward progression in the international rankings – slipping from sixth to 13th among participating countries since 2000 – coincides with the adoption of discovery learning.

So, what about that new curriculum?

The petition was presented to the Alberta legislature on Tuesday, Jan 28, 2014. (A later article gives a date of March 2014.)

So apparently the new curriculum was introduced a decade earlier, in 2004. However, that's not what happened — no students were taught the new curriculum prior to 2008 - 2009. To see just how far off the 2004 date is, let’s look at how the curriculum was actually phased in. Here is the schedule for the years that each grade was first exposed to the curriculum. (I trashed around a long time trying to find this. Thanks David Martin and John Scammell for providing me with the information. Since Albert Ed does not have this information on its website, I cannot be sure that Alberta actually followed the implementation schedule, but I'm assuming that was the case.)

The PISA exams take place every three years, and Canadian students take the tests in Grade 10. Just like David and John did in their 2014 posts², I’ll save you some time and do the math so you can see when the students were first exposed to the new curriculum. 

• Those who took the 2003 test - never exposed
• Those who took the 2006 test - never exposed
• Those who took the 2009 test - never exposed
• Those who took the 2012 test - first exposed in grade 7
• Those who took the 2015 test - first exposed in grade 4

The curriculum changes could not be reflected in any way by the PISA results prior to 2012. And those who wrote the PISA test in 2012 were not exposed to the new curriculum in the elementary grades (which is where those who blame the new curriculum for our decline are pointing their fingers).

The biggest drop in the PISA scores seems to have occurred in 2006, and the PISA grades since then have remained quite steady. What’s going on here?  Did the introduction of the new curriculum in 2008 cause a drop in the PISA results in 2006? Is this is a brand new type of quantum entanglement? As you all know, with ordinary everyday quantum entanglement, changes in physical properties can be transmitted across great distances instantaneously.³  Yes, instantaneously, but never backwards through time. As someone might say, that would be really spooky! 

Sorry for the sarcasm.

If you accept the PISA results as evidence of a decline of our students' math abilities, then those same results show that the decline occurred well before the new curriculum could have had any influence. And, this also means that the PISA results provide no evidence that returning to the previous curriculum will repair things.

Do the PISA scores matter?

By the way, if you are concerned about our sinking rank in the math PISA league tables, note that for the 2015 round, Canada moved up to 10th from the 13th place.

David Staples (a columnist for the Edmonton Journal) has said that the people who want to return to pre-2008 curriculum and teaching methods are not really concerned with our PISA rank (despite the comments quoted above). Their concern is that the proportion of lower performing students is increasing while the proportion of higher achieving students is decreasing.

Although that's true, and it might be serious, the decline started prior to 2015, when none of the students who took the PISA test were exposed to the new curriculum in elementary school. So, again, the PISA results are not evidence that the new curriculum is at fault. We'll have to go elsewhere to get evidence that it was caused by the new curriculum.

And if we can't find any, please let us not blame the front line workers and have this discussion degenerate into a round of teacher bashing.

---

REFERENCES

¹ The conversion from the published PISA scores to percentages was done as follows:

Percent_grade = 60 + (Pisa_Score − 482) × (10 / 62.3). 

More details can be found here .


² David Martin's post about the 2012 PISA round is here, and John Scammell's is here.

³ Quantum entanglement — I'll let others explain it. Here are two you tube videos that do a pretty good job.

Quantum Entanglement & Spooky Action at a Distance

Yro5 - Entanglement

** post was edited Dec 14 to correct two typos

Understanding the PISA Tables

The PISA math results show Canada trending downwards internationally. Here are the rankings of the top 20 regions (of approximately 65) that participated in the 2003, 2006, 2009, and 2012 rounds:

Note:  Shanghai and Singapore have been greyed out for 2003 and 2006 because neither took part in those two rounds. Since it is hard to assess a trend when the population base changes, I think that either Shanghai and Singapore should be included for all rounds, or both should be excluded for all rounds. I have chosen to include them, and, based on their results in 2009 and 2012, I have assumed that they would have finished atop the PISA table had they competed in the earlier rounds.

With Shanghai and Singapore included, the perspective on the Canadian trend changes somewhat. However, some slippage is definitely happening, and for many people the trend is alarming.

But I wonder if we are concentrating too closely on the rankings. Here is the same table when grades are included:

Note: OK. This doesn't look like the PISA scores that you have seen. The results shown above are percentage grades, and the PISA consortium does not report the scores this way. Instead, they use a numerical scale that ranges from about 300 to over 600.

I have a fondness for percentage grades. They provide me with a more serviceable scale than grades out of 600. For example, to me, a class average of 67% indicates an ordinary but acceptable performance, while I am not sure what an average of 527 on the PISA scale conveys.

The conversion from the PISA scale to percentage grades in the table above was done linearly:

Percent_grade = 60 + (Pisa_Score − 482) × (10 / 62.3).

As is evident, Canada has remained remarkably stable: 68%, 67%, 67%, 66%. 

Proficiency Levels

The PISA results are intended to be a measure of mathematical proficiency, and PISA describes six different levels. The grade ranges for each level are as follows:

The descriptions above are extremely abbreviated. Detailed descriptions can be found in the 2012 PISA technical report (link [5] below). Also, PISA does not use verbal descriptions like Highly proficient or Basic proficiency.

Some features of the PISA proficiency levels

• For each round, the PISA consortium adjusts the grades so that the proficiency levels retain the same meaning over the years. A percentage score of 65% in 2003 has exactly the same meaning as a percentage score of 65% in 2012. In other words, PISA takes care to avoid grade inflation.

• On the percentage scale, each Level band is 10 points wide. On the PISA scale, the bands are 62.3 points wide.

• The PISA scales are designed so that the OECD mean is 500 (or very close to 500). On the percentage scale, the OECD mean is 63%.

• Over the four rounds from 2003 to 2012, Level 6 has never been achieved by any of the participating regions. Level 5 has been achieved by one participant. Level 4 has been achieved by five participants.  In the four rounds from 2003 to 2012, Canada has resided in the Level 3 band.

Is the slippage significant?

The difference between Canada’s score in 2003 and 2012 is statistically significant. The words mean that, with high probability, the difference in the scores is real and is not a result of the chance that arises by testing only a sample of the population. Statistical significance tells us that the statistics are reliable, but it doesn't tell us anything about whether or not the statistics are important.

(The word "significant" has considerable emotional impact. I remember a Statistics professor telling us that the word should be outlawed when we are discussing statistical results.)

Do the differences in the PISA tables show that Canada must do something to improve how our students learn mathematics? Some people believe that the statistically significant difference between 2003 and 2012 is also of great practical significance, and they have strong opinions about what should be done.

As well, the PISA group itself has investigated what will and won't work to improve students mathematical proficiency, and has published its findings in a 96 page document [6]. Some of the conclusions appear to run counter to what the back-to-basics people advocate — one conclusion in the document is that dependence on intensive memorization and rote practice of the basic algorithms is a poor strategy for learning anything but the most trivial mathematics.

That particular PISA document may leave some of my friends in a slightly awkward place: Those who have used the PISA tables as a rallying cry for a return to traditional methods now find themselves having to rebut some of PISA's own conclusions about what constitutes good teaching and good curriculum.

In any event, when I look at the percentages for Canada from 2003 to 2012, I see changes, but I don't think that they are earth-shattering. Would you be concerned if one section of your Calculus course averaged 68% and another section averaged 66%? I’ve run into that situation teaching back-to-back sections of the same course, and I was not convinced that the section with the higher score was the superior one.

* * *

In a few weeks, the OECD will release the results of the 2015 PISA round. I am very interested in seeing if Canada's results are significantly different in a practical sense. My expectation (based on no evidence whatsoever) is that the results will be much the same as before, and will fall within the 65% to 67% range. ( Hey! A Halifax weatherman had a very successful forecasting run by predicting tomorrow's weather to be the same as today's.)  I'll do a Trumpian gloat if my prediction is correct, and a Clintonian sob if it's wrong.

References

[1] 2003 PISA report (Math scores are on page 92)

[2] 2006 PISA report (Math scores are on page 53)

[3] 2009 PISA report (Math scores are on page 134)

[4] 2012 PISA report (Math scores are on page 5)

[5] PISA 2012 Technical report (Proficiency levels are described on pages 297 - 301)

[6] Ten Questions for Mathematics Teachers… and How PISA Can Help Answer Them, OECD Publishing, Paris (2016).

A letter to my cousin

Hi Barb,

Thanks for the link.

I have thought a lot about this controversy (the Canadian Math Wars). It’s a horrible state of affairs, and it should stop.

I'm afraid that I'm going to have to play "Blame the messenger," because articles like this one
pit the general public and some academics against the teachers and our education systems. I think the fault lies with the laziness of news columnists who reflexively go to the easiest sources. Whenever something is written about math education in Canada you can almost guarantee that the most prominent position in the article will be given to Robert Craigen or Anna Stokke deriding the current state of affairs. Controversy sells newspapers and TV programs.

In my opinion, the math wars stem from two things. First, new students entering university are ill-prepared for their math courses (upsetting the math profs because the students do not know the fundamentals). Second, teaching methods have changed since the students' parents went to school (upsetting the parents because they cannot help their children with their homework).

This gives rise to the sentiment that math education was better in the past, and that it is currently in decline. This is bolstered by the fact that Canada’s scores are slipping in international math tests such as PISA. [ PISA = Programme for International Student Assessment. ] The result is a desire to go "back-to-basics" and to confine teaching to the "traditional" ways.

Here is my take:

When I first started teaching at U of A forty years ago, I also found the students to be ill-prepared, so this is not a new perception.

It is true that changes have been made in the way math is being taught, but I don't think it's a bad thing. Just ask a few of your friends about how they did in math at school. I’ll wager almost every one of them will say "I was no good at it!" In other words, we were pretty bad at teaching math in the past, so changes in our teaching methods had to be made. And I cannot understand why someone who says they are bad at math would want things to revert to the way they were taught.

As far as PISA goes, I have doubts that the results are a reliable measure of math education. But even if they are, I don't think we have slipped that much. Canada is still among the better performing countries. Finland is perennially cited as one of the best, and in the 2012 PISA round of tests, Canada was merely one step below Finland. Incidentally, Pasi Sahlberg, a highly respected Finnish educator, recently tweeted "The world needs more Canada."

And by the way, newspaper columnists almost always use the term "discovery math" when talking about some of the more up-to-date teaching methods. It's a very loosey-goosey term that implies that current teaching methods forbid direct or explicit instruction thereby leaving students to flounder helplessly on their own.

I don't believe that this is really happening, especially the part about forbidding direct/explicit instruction. In the past three years I must have read hundreds of blog posts by teachers describing how they taught one or another topic in mathematics. Most of them presented thoughtful routes through the lesson. They all involved direct or explicit instruction at some point, 'though not the way you and I experienced it. 

I guess I can sum up my feelings this way:

I agree that there is room and need for improvement in our math education, and I have opinions about that, but "Forward To The Past" should not be an option.

Regards,

Ted

PS. Somewhat off-topic: A few years ago, the doll with your namesake caused quite a stir when it was programmed to say "Math is hard!" A school teacher told me Barbie should have said: "Math is hard, and teaching math is even harder."

PPS. You can read about my biases against PISA here and here.

How did you learn your times tables?

When I was a kid, I liked playing ice hockey. I was actually not very good at it — no Connor McDavid here!  But I did acquire the basic skills. For example, I figured out how to lift the puck. (For you non-hockey players, that means shooting the puck in such a way that it flies off the ice into the air. It’s a essential skill if you want to be able to score goals.)

I practiced that skill a lot. Whatever I did, it worked. I could lift the puck consistently without thinking about it. I haven’t shot a puck for many decades, but whenever I imagine doing so, I swear can feel the memory in my triceps.

Of course, I really did not "figure out" how to lift the puck. I did not know the theory behind the lifting action. And to the extent that the skill was necessary, I didn’t need to understand the theory.

The lesson is this:

When learning something new that you will need for later use, master the mechanics first. You can learn why it works later.

* * * Warning: possible straw man ahead * * * 

It’s a useful lesson.  It helps me understand the approach to mathematics teaching advocated by the back-to-basics people: You can be successful by learning the how without understanding the why. Just learn the essential basic facts and algorithms. Don’t worry about why the puck flies into the air — just practice shooting enough so that you can lift it consistently and effortlessly.

Reasonable advice? Maybe. But, no matter how hard I practiced, I could not always "lift" the multiplication tables. As far as the basic multiplication facts are concerned, I do not have what some people call rote recall — I do not have the ability to rapidly and effortlessly retrieve all of the basic learned facts from memory.

A great chunk of my own elementary math education was founded on the contrary belief, that rote recall is, in fact, achievable by everyone — that all it takes is practice.  Accordingly, my classmates and I were regularly drilled and tested on the multiplication tables. I did not do well, and I argued with my teachers. Ultimately, I was punished for my inability to memorize the 12 x tables.

I don’t believe that my recall difficulties are exceptional. The more blogs I read, the more I suspect that there are many people who, no matter how much they practice, will never possess rote recall of the basic arithmetic facts. In that sense, those people can never know the basic facts.

So, it was with interest that I read that Nikki Morgan, the secretary of state for education in the UK, has decreed that:

"we are introducing a new check to ensure all pupils know their times tables by age 11"

An interesting post by @thatboycanteach asks what it means to "know" the times tables. Like me, he suffers from what might be described as rote recall deficiency. And like me, he survived (and even thrived) by using various work-arounds to compensate.

The UK times-table test will be computerized and time-restricted. It looks like it will be based on pure rote recall. For the flunkies, there will undoubtedly be some sort of penalty. It’s unlikely that they will be physically punished like I was, but even non-corporeal punishment can inflict great stress and harm and, in the end, may prevent them from learning mathematics.

What is of concern to me in Alberta is that, however sincere the back-to-basics people may be, they seem to be basing their reform efforts on the very thing that caused me difficulties, namely, the belief that all students can and must achieve rote recall, that this is the only way to know the basic facts.

That is the conclusion that I draw from reading their petitions and press releases. If I’m wrong, if I am raising a straw man, it is difficult to understand why they also want to banish the teaching of alternate approaches to the basic facts and algorithms that are needed so that people like me can compensate for our deficiencies.