Changing percent of students at each PISA math proficiency level

This is a graphical summary of how Canadian students have trended in the triennial math tests offered by the PISA/OECD consortium.

It is not so much about their international standing, but about how their achievement levels have varied over the five rounds from 2003 through 2015.

For each participating region or country, PISA publishes how the students’ grades are distributed across seven different proficiency levels. The levels are the same for all jurisdictions, and they have remained the same for the five rounds since 2003.

The grades defining the boundaries of the proficiency levels are shown in boldface. (The ones in parentheses are what you get when the PISA-assigned grades are converted in a reasonable way to grades out of 100.)  The yellow and blue levels identify low performing and and high performing students. The interval widths for each of Level 1 through Level 5 are the same. PISA does not define the widths for the lowest (less than Level 1) and highest (Level 6) proficiency levels.

The table below summarizes the distributions for Canadian students for each of the five PISA rounds.

In effect, each row of the table is an estimated probability distribution. For example, if you were to randomly choose a Canadian student in 2003, the probability that he or she would have achieved Level 5 is 14.8 percent.

Actually, the reason I wrote this post was to try to figure out how to use slides in blogger -- I think slides give a more vivid visual depiction of the changes than a table does.

The following slideshow displays the table as a sequence of histograms. There is some concern that Canadian students are trending in the wrong direction. Perhaps this graphical representation may help you decide if the trend is cause for alarm.

     

The Meaning of the 2015 PISA Canadian Math Results

For the past few years various news media have repeatedly conveyed the belief that:

1. The mathematical abilities of Albertan and Canadian students have declined significantly.
2. This decline was caused by the introduction of a new curriculum.
3. Evidence is provided by the results of the PISA tests.

That sentiment is what this post is about, but let me start with this question.

At a certain junior high school the averages of the math grades for the past three years were 73%, 75%, and 73%. How would you rate this school?

1. Definitely a top notch school.
2. That’s a pretty good school.
3. That’s what an average school should be like.
4. Mediocre — room for improvement.
5. Poor — I would think about sending my child to a different school.

Myself, I would choose (2).  I would expect grades of 65% - 70% to be about average so I feel the school is a pretty good one. Not absolutely fantastic, but pretty good.

The real story:

The averages cited above are not for a particular school, but for an entire country. The figures are the average math scores for top notch Singapore in the 2009, 2012, and 2015 rounds of the PISA tests. (If you are familiar with the PISA tests, you know that the scores are not recorded as percentages. The numbers 73%, 75%, and 73% are what you get when you convert the published PISA scores in a reasonable way.¹ )

For comparison, here are the Alberta and Canadian results for the years from 2003 through 2015, again converted to percentages. (Singapore did not compete prior to 2009).

The variation across the PISA rounds for all three regions seems pretty unexceptional. I would expect changes of ±3% at the very least — the year-to-year variability in my university math classes were at least as large as that.

[As an aside, I predicted in a previous post that Canada would get a grade in the range from 65% to 67% on the math part of the 2015 PISA test. I promised a Trumpian gloat if the prediction was correct, and a Clintonian sob if it was wrong. No guru here, I’m afraid — it was pure luck — but "Gloat. Gloat" anyway.]

In spite of the stable relationship between the results for Alberta and Canada, there is alarm that Alberta's performance in the 2015 round has pushed them below the rest of Canada. That may be numerically accurate when you look at the raw scores published by PISA, but the results are so close that when converted to percentages the difference vanishes. And, more importantly, the PISA consortium itself reports that the difference is so minuscule that it is not statistically significant. Which means there is no reliable evidence that there is any difference whatsoever.  

I don’t think that our ranking among the PISA participants indicates that our math curriculum is either superb or deficient. But using the PISA rankings to critique the curriculum is what people do. To confirm this, all you have to do is read the opening foray in the petition that stoked the current math wars in Alberta. Here is what it says:

"The Organization for Economic Co-operation and Development (OECD) recently released a report confirming that math scores, measured in 15yr olds, have declined significantly in the ten years since the new math curriculum was introduced across Canada."

Additionally, there is this report in the Globe and Mail in January 2014:

Robert Craigen, a University of Manitoba mathematics professor who advocates drilling basic math skills and algorithms, said Canada’s downward progression in the international rankings – slipping from sixth to 13th among participating countries since 2000 – coincides with the adoption of discovery learning.

So, what about that new curriculum?

The petition was presented to the Alberta legislature on Tuesday, Jan 28, 2014. (A later article gives a date of March 2014.)

So apparently the new curriculum was introduced a decade earlier, in 2004. However, that's not what happened — no students were taught the new curriculum prior to 2008 - 2009. To see just how far off the 2004 date is, let’s look at how the curriculum was actually phased in. Here is the schedule for the years that each grade was first exposed to the curriculum. (I trashed around a long time trying to find this. Thanks David Martin and John Scammell for providing me with the information. Since Albert Ed does not have this information on its website, I cannot be sure that Alberta actually followed the implementation schedule, but I'm assuming that was the case.)

The PISA exams take place every three years, and Canadian students take the tests in Grade 10. Just like David and John did in their 2014 posts², I’ll save you some time and do the math so you can see when the students were first exposed to the new curriculum. 

• Those who took the 2003 test - never exposed
• Those who took the 2006 test - never exposed
• Those who took the 2009 test - never exposed
• Those who took the 2012 test - first exposed in grade 7
• Those who took the 2015 test - first exposed in grade 4

The curriculum changes could not be reflected in any way by the PISA results prior to 2012. And those who wrote the PISA test in 2012 were not exposed to the new curriculum in the elementary grades (which is where those who blame the new curriculum for our decline are pointing their fingers).

The biggest drop in the PISA scores seems to have occurred in 2006, and the PISA grades since then have remained quite steady. What’s going on here?  Did the introduction of the new curriculum in 2008 cause a drop in the PISA results in 2006? Is this is a brand new type of quantum entanglement? As you all know, with ordinary everyday quantum entanglement, changes in physical properties can be transmitted across great distances instantaneously.³  Yes, instantaneously, but never backwards through time. As someone might say, that would be really spooky! 

Sorry for the sarcasm.

If you accept the PISA results as evidence of a decline of our students' math abilities, then those same results show that the decline occurred well before the new curriculum could have had any influence. And, this also means that the PISA results provide no evidence that returning to the previous curriculum will repair things.

Do the PISA scores matter?

By the way, if you are concerned about our sinking rank in the math PISA league tables, note that for the 2015 round, Canada moved up to 10th from the 13th place.

David Staples (a columnist for the Edmonton Journal) has said that the people who want to return to pre-2008 curriculum and teaching methods are not really concerned with our PISA rank (despite the comments quoted above). Their concern is that the proportion of lower performing students is increasing while the proportion of higher achieving students is decreasing.

Although that's true, and it might be serious, the decline started prior to 2015, when none of the students who took the PISA test were exposed to the new curriculum in elementary school. So, again, the PISA results are not evidence that the new curriculum is at fault. We'll have to go elsewhere to get evidence that it was caused by the new curriculum.

And if we can't find any, please let us not blame the front line workers and have this discussion degenerate into a round of teacher bashing.

---

REFERENCES

¹ The conversion from the published PISA scores to percentages was done as follows:

Percent_grade = 60 + (Pisa_Score − 482) × (10 / 62.3). 

More details can be found here .


² David Martin's post about the 2012 PISA round is here, and John Scammell's is here.

³ Quantum entanglement — I'll let others explain it. Here are two you tube videos that do a pretty good job.

Quantum Entanglement & Spooky Action at a Distance

Yro5 - Entanglement

** post was edited Dec 14 to correct two typos

A letter to my cousin

Hi Barb,

Thanks for the link.

I have thought a lot about this controversy (the Canadian Math Wars). It’s a horrible state of affairs, and it should stop.

I'm afraid that I'm going to have to play "Blame the messenger," because articles like this one
pit the general public and some academics against the teachers and our education systems. I think the fault lies with the laziness of news columnists who reflexively go to the easiest sources. Whenever something is written about math education in Canada you can almost guarantee that the most prominent position in the article will be given to Robert Craigen or Anna Stokke deriding the current state of affairs. Controversy sells newspapers and TV programs.

In my opinion, the math wars stem from two things. First, new students entering university are ill-prepared for their math courses (upsetting the math profs because the students do not know the fundamentals). Second, teaching methods have changed since the students' parents went to school (upsetting the parents because they cannot help their children with their homework).

This gives rise to the sentiment that math education was better in the past, and that it is currently in decline. This is bolstered by the fact that Canada’s scores are slipping in international math tests such as PISA. [ PISA = Programme for International Student Assessment. ] The result is a desire to go "back-to-basics" and to confine teaching to the "traditional" ways.

Here is my take:

When I first started teaching at U of A forty years ago, I also found the students to be ill-prepared, so this is not a new perception.

It is true that changes have been made in the way math is being taught, but I don't think it's a bad thing. Just ask a few of your friends about how they did in math at school. I’ll wager almost every one of them will say "I was no good at it!" In other words, we were pretty bad at teaching math in the past, so changes in our teaching methods had to be made. And I cannot understand why someone who says they are bad at math would want things to revert to the way they were taught.

As far as PISA goes, I have doubts that the results are a reliable measure of math education. But even if they are, I don't think we have slipped that much. Canada is still among the better performing countries. Finland is perennially cited as one of the best, and in the 2012 PISA round of tests, Canada was merely one step below Finland. Incidentally, Pasi Sahlberg, a highly respected Finnish educator, recently tweeted "The world needs more Canada."

And by the way, newspaper columnists almost always use the term "discovery math" when talking about some of the more up-to-date teaching methods. It's a very loosey-goosey term that implies that current teaching methods forbid direct or explicit instruction thereby leaving students to flounder helplessly on their own.

I don't believe that this is really happening, especially the part about forbidding direct/explicit instruction. In the past three years I must have read hundreds of blog posts by teachers describing how they taught one or another topic in mathematics. Most of them presented thoughtful routes through the lesson. They all involved direct or explicit instruction at some point, 'though not the way you and I experienced it. 

I guess I can sum up my feelings this way:

I agree that there is room and need for improvement in our math education, and I have opinions about that, but "Forward To The Past" should not be an option.

Regards,

Ted

PS. Somewhat off-topic: A few years ago, the doll with your namesake caused quite a stir when it was programmed to say "Math is hard!" A school teacher told me Barbie should have said: "Math is hard, and teaching math is even harder."

PPS. You can read about my biases against PISA here and here.

PISA: snapshots with a fogged up camera

When I was an undergrad I had a friend who lived by the motto “It’s easier to be infamous than it is to be famous.” He was a notorious “sh!+ disturber”. He caused havoc all over the campus, and he enjoyed it.

A certain Edmonton Journal columnist seems to have inherited my friend’s mantle. He enjoys tweaking the noses of the education community. I suspect his “sh!+ disturbing” also gives him much enjoyment. His weapon of mass disturbation is PISA.

Every three years the OECD tests a large sample of 15-year olds in Mathematics, Science, and Reading. This is PISA, the Programme for International Student Assessment. The OECD describes the programme as providing a snapshot of the state of education in the participating countries. Their objective is clearly stated: PISA is intended to guide the development of education policies around the world. 

Following each round of PISA exams, OECD releases a welter of results, presenting them as league tables, that is, as a list with the highest scoring countries at the top and the lowest scoring countries at the bottom. Newspaper columnists love the league tables.

Although I like to see how our education systems stack up internationally, I am not a supporter of PISA. There are lots of questions that I find unsettling. Here are two.

How reliable are the PISA results?  

Imagine that you are a teacher, and that you have decided to use a two-hour final exam to assess your students. You concoct a list of questions covering all aspects of the course, but you realize that it would take a student up to four hours to complete all of these questions. 

Here’s how you get around the problem: You have a lot of additional information about each student (assignments, quizzes, midterms, classroom observations). So you set a final exam by using a subset of the questions and you fill in the missing data by using all that extra information. 

Now imagine that you are the PISA examiner. To fully evaluate the students you also need a four hour exam. As before, each exam paper is only permitted to be two hours in length. 

But now there is a real problem: you do not have access to all that wonderful additional information. If you set a two-hour exam, there will be an awful lot of missing data, and you will have no way to compensate for it. However, in order to get reliable results, you absolutely need to use all the questions that you have concocted.

Here’s how PISA gets around the problem. The examiners are really not interested in individual results. They only want the aggregate picture. So they distribute the questions among different exam papers. The exams are not the same—the questions on Mary’s paper will be different than the ones on Peter’s paper. In aggregate, your class answers all test questions, but for each student there is a large chunk of missing data. PISA generates the missing data by using a psychometric model called the Rasch model. 

The situation may be described like this. Let's suppose we are testing mathematics. Everybody named Mary gets a question about area, but nobody named Peter does. Nevertheless, everyone named Peter still gets a grade for the area question, and the grade is determined by how the Marys answered the question.

In some circumstances, the Rasch model works, but Svend Kreiner, a Danish biomedical statistician, says that PISA uses it incorrectly. The use of the Rasch model has also been criticized by David Spiegelhalter, professor of the public understanding of risk at Cambridge University.

Some people say that Kreiner is wrong and that it is acceptable to use the Rasch model. I am not a statistician, and  it is not clear to me who is correct. (However, I find it relevant that Kreiner was one of Rasch’s students and that he has used the Rasch model for 40 years.)

How should we interpret the PISA league tables?

Let’s dismiss any misgivings about the reliability of the results and take the PISA league tables as valid. But keep in mind that some data is generated statistically, and the students taking the exams are only a sample of the 15-year old cohort. Consequently, there is some uncertainty in the published figures. 

The league tables show a plausible single score for each country, but the uncertainty means that there is actually an interval range of plausible scores. If the intervals for two countries overlap then you cannot really conclude that the scores are different. 

For example, in the 2012 PISA mathematics table, Finland finished 12th with an average score of 519. Canada placed 13th with an average score of 518. The intervals for Finland and Canada overlapped considerably. Do the results really allow us to say with any degree of certainty that Finland finished ahead of Canada? 

The uncertainty allows some bending of the league tables to suit your argument.

If you want to argue that Canada is slipping badly in math education, you use the league tables based on the single scores: In 2006 Canada was sixth from the top, in 2009 Canada was tenth from the top, and in 2012 Canada was thirteenth from the top. From sixth to thirteenth is a significant drop.

If you want to argue that we are actually not dropping very much, you use the statistical intervals. Based on these, in 2006 Canada was tied for fifth, and in both 2009 and 2012 Canada was tied for tenth. So, no drop from 2009 to 2012.

This still looks like a drop in performance since 2006. But wait: in 2006 neither Shanghai nor Singapore participated in PISA. So you can argue that if we want to compare the 2009 and 2012 results to 2006, we should exclude those “countries”. When we exclude them, we have Canada finishing fifth in 2006 and eighth in 2009 and 2012. And now the drop somehow doesn’t seem as drastic.

- - -

I don't think I've answered the two questions. 

Nevertheless, as OECD intended, PISA provides people with data that they can use to press their governments about their education system. It’s unfortunate that league tables may be flawed and that they are presented in a way that allows the data to be bent to fit one’s agenda. 

Thanks for reading.

- - -

Sources

The arguments against the use of the Rasch model:

An academic paper by Svend Kreiner .

A paper by Svend Kreiner and Karl Bang Christensen .

(To be honest, I didn’t read the above papers. They are highly technical and beyond my expertise.)

Two articles by David Speigelhalter here and here (these are readable).

Rebuttals of the above:

Ray Adams (member of PISA) has a longish article defending PISA’s methodology.

Jan-Eric Gustafsson (university of Oslo) discounts Kreiner and Christensen’s critique. (Gustafsson, however, has other reasons to doubt the valdity of the league tables.)

The PISA data:

The OECD-PISA league tables can be found at the OECD PISA site. CMEC also has them here:

PISA 2006,  PISA 2009, PISA 2012

The original arguments claiming that PISA is flawed were published some time ago. Here are some more recent ones: 

Matthew Smith in Education Week in review (April 2014).

William Stewart, a long time education reporter for TES. (Last Updated: 27 September, 2014).

Benjamin Reilly, Founder of Deans for Impact (Feb 2014).

Catherine Wolff, an education writer in New Zealand (Dec 2013).

How to raise our PISA scores

A rather facetious list of recommendations based on what some of the participants have done or are doing.

PISA is the Programme for International Student Assessment. Every three years, the OECD (Organisation for Economic Co-operation and Development) tests 15 year old students around the world in math, reading, and science. A lot of Canadians are upset because Canada ranked 13th out of 65 in the 2012 math test, a slight decline from previous results. The top five ranking countries in the PISA math test were Shanghai, Singapore, Hong Kong, Taiwan, and South Korea.

The PISA math test is a two hour test. There are some multiple choice questions and some long answer ones. I don't know how many there are. A selection of the questions can be found here and here. A quick look reveals that they are predominantly of the "fake world" type that Dan Meyer so dislikes. 

Based on the PISA results, some countries have concluded that their curricula and teaching are deficient. Some, like the U.K and New Zealand, are intending to use the tests as a benchmark for their education systems or at least they are tilting in that direction, and it looks like they are getting ready to retool. (see this BBC report, and this Radio New Zealand report)

To me, using the PISA tests to draw conclusions about either our curriculum or our teachers seems iffy. Judging our entire system by how well a group of teenagers did in a two hour test is like training our athletes to compete in a triathlon and then measuring our success by how well some of them did in a 100-metre sprint. 

However, in case you are really determined to improve our PISA ranking, here are some recommendations that may do the trick without spending millions to revamp everything. But be aware that sprinters do not always turn out to be good triathletes. 

1. Increase the amount of homework that the students have to do. 

Look at the top two finishers: Shanghai students average 13.8 hours of homework per week and Singapore just under 10, while Canadian students only average between 5 and 6 hours (source : Shanghai PISA team). 

2. Set up after-school training clubs. Encourage entrepreneurial teachers to run the clubs and to publish and sell PISA-type practice questions. Encourage parents to hire tutors to help their children.

Parents in the top five jurisdictions spend large sums for extra tutoring and after-school training.  In particular, the top five are know for their notorious private "cram schools". Students attend these schools in the hope of passing exams and achieving either high school or university entrance. 

3. Import instructors from higher ranking jurisdictions to teach our teachers how to teach. 

The UK is  bringing 50 teachers from Shanghai to do just this. Actually the imported teachers will help the Brits reform their math education into a system that is centred around 32 hubs, similar to Shanghai . The cost will be 11 million pounds (CAD 17.75 million), so if we are not careful, importing teachers may lead to expensive restructuring. 

4. Don't let the bottom 20 percent of our students take the PISA tests. We can do this by barring English language learners and students with low socio-economic backgrounds.

It is pretty well established that disadvantaged children do not perform well on tests, so barring them should raise our scores. Shanghai excludes migrant children from even participating in its education system, as was confirmed to me by a teacher who taught in Shanghai. The proportion of excluded students is difficult to determine, but seems to be somewhere between 20 and 50 percent. See the damning report by Tom Loveless.

5. Put pressure on the OECD folks to release the PISA scores for individual students and schools. To further increase competitiveness, institute a set of monetary rewards for schools and provinces whose students perform the best in the PISA tests, and deny those rewards to schools whose students don't do so well. 

I'm a bit late with this recommendation. The OECD has already developed PISA-based tests for schools in the United States, England, and Spain. (Condolences to our American neighbours - more tests, just what you need!) 

Let's be honest about the PISA tests. For most of us, the only thing that counts is where we rank. And this leads to my final recommendation:

6. Let's call the PISA test what it is: a competition, not an assessment. 

I'm not sure that education should be based on competition.