Making the grade

International testing that is used to predict the grim future of US science and technology is being vastly misinterpreted, say Hal Salzman and Lindsay Lowell.

it’s a familiar story. Children around the world have been tested, and the UnitedStates is in trouble. A US Department of Education report1 from March concludes that “without substantial and sustained changes to the educational system, the United States will relinquish its leadership in the twentyfirst century”. The panic plays out in countless newspaper articles and policy reports, recently leading to legislative responses such as the America COMPETES Act, which contains a list of measures to boost average mathematics and science test scores. A country’s place in the new global economy is, according to these reports, determined by its rank in the maths- and science-score hierarchy. Following this reasoning, one would conclude that the US economy is threatened not only by Japan and South Korea, but also by Finland, Singapore, New Zealand and the Czech Republic.
The rankings that engender these fears are primarily based on two tests administered to middle-and high-school students since 1995: the Programme for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS). Improving education should be a priority for the nation, but erroneous interpretations of international test scores may drive economic and competitiveness policy in the wrong direction. When we consider that education testing shows formidable US strength as the largest producer of top-scoring students alongside a significant problem at the bottom, the threat to future competitiveness seems to be something quite different from the headlines2. Caution is needed so we neither create
policies that overstock the science and technology workforce nor unthinkingly implement the education and social practices in other high-scoring countries. A full grasp of the meaning of testscore differences should lead the next president to address education and competitiveness problems more effectively than the recent America COMPETES legislation, which is now languishing for a lack of funding. Focusing the great consternation about education on real rather than imagined problems requires a careful assessment of the evidence. Lagging behind? It is misleading to gauge the relative position of the United States in the world based on a simplistic ranking of its students’ test scores. This is much like measuring shoe size to predict runners’ future race times while ignoring their past performance. There are substantial methodological limitations in using these tests to compare nations, including reporting ‘rankings’ that are based on minute differences that are not statistically significant3. For example, when considering statistically significant differences, national test scores can be clustered into three meaningful levels and the United States consistently ranks in a middle group on maths and science while being top ranked in civics4 — the study
of citizenship and government. Overall, about one-fifth of other nations rank better and twofifths rank underneath the United States. Still, average performance tells us nothing about the distribution of students with the very best test scores. In maths and science, when looking at average scores, the United States is outranked by countries such as Finland and South Korea. But the rankings change when we examine the percentage of students who perform at the top, those most likely to be tomorrow’s innovators. The South Korean average places it in the top-ranked group of nations, yet its relative proportion of top performing
students is 30% lower than that of the United States. In fact, the United States has a higher percentage of top-performing students than 5 of the 14 others in the top-ranked group of countries with high average scores. Moreover, it would seem inappropriate to consider the United States, a country with a population of more than 300 million, in competition with Singapore, a country of 4.5 million, or with even smaller New Zealand. The economies in these countries range from a gross domestic product (GDP) of $124 billion in New Zealand to $236 billion in Finland, compared with the $14-trillion GDP of the United States. Perhaps a more apt comparison would be Massachusetts with a population of 6.4 million and a gross state product of $338 billion, or Colorado with 4.8 million residents and a $230-billion state product. Although the top group also includes
economic powerhouses South Korea and Japan, which come in at under a fourteenth and less than a third, respectively, of the size of the US economy, for the most part it makes more sense to compare US state economies with other countries because it is
Massachusetts or California that is ‘competing’, for example, with Singapore in developing their biotech industries.
If, as we argue, average test scores are mostly irrelevant as a measure of economic potential, other indicators do matter. To produce leadingedge technology, one could argue that it is the numbers of high-performing students that is most important in the global economy. These are students who can enter the science and engineering workforce or are likely to innovate whatever their field of study. Remarkable, but little noted, is the fact that the United States produces the lion’s share of the world’s best students (see graph opposite). At the same time, low-performing students can hamper productivity and here, unfortunately,
the United States also stands out. The United States produces more than one million low-performing maths and science students
each year, more than any other country in the Organisation for Economic Co-operation and Development except for Mexico (see graph overleaf). Although programmes to improve education for low-performing students and schools are included in the various policy reports, they are shunted to the background when the headlines focus on increasing the numbers of those at the top, and overall seem to carry little weight when they are diluted as part of a long laundry list of recommendations. Market maths
Without a doubt, science, maths and technology education is needed in today’s society, whether for its citizens to understand enough to participate in public debate or just to operate the technology of everyday life. However, some argue for more advanced courses as if they want to prepare all students to be scientists or engineers. We believe that there is something fundamentally wrong with such an approach. History suggests that policies designed to stockpile scientists and engineers are counter-productive. The space race is typically cited as a success story of American technological prowess, but less often discussed is the impact of the workforce buildup on US engineering and science in the years that followed. Following a spike in the numbers of science and engineering college graduates in the late 1950s and early 1960s, a spectacular bust followed that led to high unemployment in these fields. For many years afterwards, fields such as physics were thought of as poor career choices5. Similar boom-and–bust cycles have continued for the past four decades, in engineering, in information technology (IT) and in science. When demand is translated into increased salaries and job openings, students respond. When the IT industry was growing, the number of graduates in computer science kept pace, doubling over six years. Following the collapse of the IT industry bubble, the number of graduates fell by 17% between 2003 and 2005. Employment in this field is just now reaching the levels of the boom years but, with little prospect of rapid growth, students seem to be wise in choosing other fields. Or, consider
petroleum engineering. This is an industry that has had slow growth for two decades and, correspondingly, undergraduate enrolments declined 85% during that period, and master’s programmes instead attracted students from areas of the world with fast-growing oil exploration. Today, 75% of US master’s graduates in petroleum engineering are foreign students on temporary visas. Now, the US industry has a real need for more engineers because of increased demand for oil and new exploration coupled with 20 years of minimal hiring and an ageing workforce. The oil industry has responded by increasing entry-level salaries 30-60% over the past four to five years, far greater than in other fields. As a result, petroleum-engineering graduates have doubled
in the past five years and freshmen enrolments, at Texas Tech University in Lubbock for example, have increased more than sixfold. When supply far exceeds demand, the bust that follows reverberates for many years and discourages students even when demand does increase later. As Michael Teitelbaum of the Alfred P. Sloan Foundation in New York notes, not only is there no evidence of any widespread shortages but “substantially more scientists and engineers graduate from US universities than can find attractive career openings in the US workforce”. Teitelbaum adds that this overproduction, which leads to increasingly longer postdocs in many science fields, makes our universities look more like a system to produce “a pool of low-cost research lab workers with limited career prospects than a high-quality training program for soon-to-be academic researchers”. Social choices The beauty of brandishing a simple number or a few facts is that they fit in a single headline and focus the reader’s attention. However, before we send teams of educators to discover the educational secrets of Finland, Singapore, New Zealand, South Korea or Japan, we should do more study into the nature and context of their education systems. As the PISA report6
notes, the tests do not evaluate schooling, per se, but the “cumulative impact of learning experiences ... starting in early childhood and up to the age of 15 and embracing experiences both in school and at home”. That is, much is made of a few select
schooling practices in each country and recommendations are made to emulate them without considering what the effect is both on the lives of these children and on the economy. Sending children to classes six days a week, extra preparation courses nights and weekends, and having a single examinaton that decides their fate, as is done in Japan, is not a choice most US parents would make. Nor is the social discipline in Singapore that seems to keep students on the straight and narrow path: death for drug pushers, prohibitions on spitting and, for offences in between such as robbery after 7 p.m., at least 12 strokes of the cane. Although South Korea’s spectacular economic rise is held in awe, and its tenfold-per-capita GDP increase over the past 20 years is widely praised, rarely noted is the close to 250% rise in the incidence of suicide over the same period, with suicide becoming a leading cause of death among young people. With South Korea are Finland and New Zealand at the higher end of the global rankings of test scores and suicide rates. No single factor is responsible for either high scores or suicides, but mental-health experts cite the pressure leading to one outcome as a factor leading to the other in many high-scoring nations7.

The future educational path for the United States should come from looking within the country rather than lionizing faraway test-score champions. Our analysis3 of the data suggests two fundamental problems that require different approaches. First, pedagogies must address science literacy for the large numbers of low-performing students. Second, education policy for our highest-performing students needs to meet actual labour-market demand. In the United States, a decade’s worth of
international test rankings based on slender measures of academic achievement in science and maths have been stretched far beyond their usefulness. Perhaps policy-makers feel it is better to motivate policy by pointing to highscoring Czechs with fear, instead of noting our high-scoring Minnesotans as examples to emulate. But looking within the United States may be the best way to learn about effective education. As the PISA authors emphasize in their report, 90% of the variance in the scores is within countries rather than between countries. Therefore, most of what one can learn about high performance is due to the variation in factors within the nation’s borders. It would seem far more effective to transfer best practices across city and state lines than over oceans. Chasing tails In America, little about the nation’s condition can be gleaned from averages, whether by assessments of income or education. Our great opportunities as well as our great limitations seem to be accompanied by great disparities. It is these extremes, the tails at either end of the distribution, that require much more discussion than the averages. So we need to look beyond the mean to consider size and quality of the workforce and the content of our education that will be driving innovation. In a country that has a long history of innovation and high productivity, we should start by looking at how our best schools educate top performers. It is unlikely that they do so by the types of education heralded in other countries.
Paying attention to the problems at the bottom is as important, if not more so, than focusing on the top. The most innovative technology has limited use if the more than 70 million workers without college degrees do not have the skills to use it effectively. The nation’s low performers and schools should be a headline concern and the remedies are often to be found in schools only
a neighbourhood or town away. It will be far more effective to take the best that America has to offer before seeking elusive and poorly understood practices found in a diverse collection of small countries around the globe. As advocates of evidence-based policy, we argue that competitiveness and education policy should use the best available evidence as a guide and not be driven by impressions and rhetoric. Our analysis suggests that a better understanding of the education data will lead to better and, in many cases, different policy directions from those now being advocated.

■ Hal Salzman is at the Urban
Institute, 2100 M Street NW, Washington, DC
20037, USA. Lindsay Lowell is at the Institute
for the Study of International Migration at
Georgetown University, 3300 Whitehaven Street,
Washington, DC 20007, USA.

1. www.ed.gov/about/bdscomm/list/mathpanel/report/
   final-report.pdf
2. Lynn, L. & Salzman, H. Issues in Science and Technology
   Winter, 74–82 (2006).
3. Lowell, B. L. & Salzman, H. In the Eye of the Storm: Assessing
   the Evidence on Science and Engineering Education, Quality, and
   Workforce Demand (Urban Inst., Washington DC, 2007).
4. Boe, E. E. & Shin, S. Phi Delta Kappan 86, 688–695 (2005).
5. Kaiser, D. Social Res. 73, 1225–1252 (2006).
6. Learning for Tomorrow’s World: First Results from PISA 2003
   (OECD, Paris, 2004).
7. Lim, M. Asia’s Ongoing Struggle with Suicide International
   Affairs Journal at UC Davis (29 June 2007).
8. PISA 2006 Science Competencies for Tomorrow’s World
   (OECD, Paris, 2008)