What Do Standardized Tests Really Tell Us About Learning?
by Grant Wiggins
Ed note: On May 26, 2015, Grant Wiggins passed away. Grant was tremendously influential on TeachThought’s approach to education, and we were lucky enough for him to contribute his content to our site. Thankfully his company, Authentic Education, is carrying on and extending the work that Grant developed.
We interrupt this general look at test validity to comment on very important educational research that was just made public (though the news of the findings was made known a few months ago on the APA website).
In an exhaustive study that used MCAS test scores from Massachusetts and various tests of cognition (related to working memory, processing speed and fluent reasoning) researchers from Harvard, Brown & MIT examined the relationship between achievement in school as measured on standardized tests and student cognition.
See also 25 Things Tests Can’t Measure In Students
We already knew that these cognitive skills are fundamental in advancing or inhibiting intellectual achievement generally and school achievement specifically:
These maturing mental abilities are thought to broadly underpin learning and cognitive skills. Variation in these measures predicts performance on a wide range of tasks among adults, including comprehension (Daneman & Carpenter, 1980), following directions, vocabulary learning, problem solving, and note-taking (Engle, Kane, & Tuholski, 1999). Critically, these cognitive abilities are associated with academic performance. Executive function measured in preschool predicts performance on math and literacy in kindergarten (Blair & Razza, 2007), and parental reports of attention span persistence in 4-year-olds predicts college completion at age 25 (McClelland, Acock, Piccinin, Rhea, & Stallings, 2013). Likewise, [working memory] skill correlates with math and reading ability among 5- and 6-year olds (Alloway & Alloway, 2010) and among 11- and 12-year olds (St Clair-Thompson & Gathercole, 2006), and predicts mathematics and science achievement among adolescents (Gathercole et al., 2004). Thus, cognitive skills appear to promote or constrain learning in school.
Given that results on tests of cognition predict achievement, might it work in the other direction? In other words, do results on achievement tests predict cognitive abilities?
What is unknown, and crucial for informing educational policy, is whether general educational practices that increase academic performance also have a positive impact on basic cognitive skills. Schools traditionally focus on teaching knowledge and skills in content areas, such as mathematics and language arts. Use of such knowledge can be referred to as crystallized intelligence (Cattell, 1967). In contrast, fluid intelligence refers to the ability to solve novel problems independent of acquired knowledge; the cognitive measures in the present study are indices of fluid intelligence. Do schools where students are experiencing high levels of academic success in crystallized intelligence achieve this success by promoting the growth of fluid cognitive abilities? The strong relation between cognitive ability and academic performance suggests that schools that are particularly effective in improving academic performance may also improve domain-independent cognitive skills.
And so: what did the researchers find? Oops. Better achievement on state standardized tests yields little or no gain on these cognitive skills:
Which school students attended explained substantial variance in students’ achievement scores, but not in measures of their cognitive skills…. These findings suggest that school practices that influence standardized achievement tests have limited effects on the development of cognitive skills associated with processing speed, working memory, or fluid reasoning…. These findings raise the question of what kinds of abilities are indexed by high-stakes statewide standardized tests that are widely used as a measure of educational effectiveness.
The finding that variation in schooling influences crystallized but not fluid intelligence is consistent with a population study of over 100,000 males in Sweden (Carlsson, Dahl, & Rooth, 2012)
As the researchers point out, such skills are improvable by teaching that targets them deliberately:
Although school-level educational practices that enhance standardized test scores may not increase broader, fluid cognitive abilities, there is evidence that targeted interventions—both in and out of school— may increase cognitive ability. Preschoolers enrolled in a year-long executive function training curriculum improved performance on untrained executive function tests (Diamond, Barnett, Thomas, & Munro, 2007). Children receiving an intervention emphasizing the development of cognitive and conceptual skills…from birth to either 5 or 8 years of age, performed better on both standardized intelligence (IQ) and academic tests (Campbell, Ramey, Pungello, Sparling, & Miller-Johnson, 2002). Teaching inductive reasoning to third and fourth grade students improved performance on untrained reasoning tests and a fluid reasoning measure if the intervention lasted for two years (de Koning, Hamers, Sijtsma, &Vermeer, 2002). … Eight-week training in after-school programs focused on either reasoning or speed training selectively enhanced performance in 7-9 year-olds (Mackey, Hill, Stone, & Bunge, 2011).
So, we are left with a vital question, once we realize the importance of traditional school tests (both standardized and locally designed): if schooling is supposedly key to adult success, what happens when schooling separates content knowledge from thinking skills and measures (and thus teaches) only the former? And might our stubborn problem of the achievement gap be based on measuring and teaching the wrong things?
It is unknown, however, how a selective enhancement of crystallized intelligence, without the enhancement of typically correlated fluid intelligence, translates into long-term benefits for students, and whether additional enhancement of fluid intelligence would further bolster long-term educational and SES outcomes.
No one familiar with my work will be surprised by these findings. They buttress the work of the last 15 years of Understanding by Design as well as warning what happens if the curriculum is reduced to teaching and testing of discrete content.
Yes, of course: it’s one study. And as a researcher will always say: “More research is needed.” And there are caveats in the research, noted by the authors (and, thus, ironically, a validity question, given the specific evidence gathered in terms of a broader goals of the research, just as was discussed in the previous post).
Students Can Do Well On Tests & Still Not Understand
However, this research underscores many other findings, summarized in the National Academy of Sciences seminal text How People Learn. This latest finding helps explain why transfer is far rarer than we want it to be and expect it to be given all the teaching. It helps explain the science and mathematics misconception literature which highlights the non-fluent rigidity of naïve concepts and knowledge.
And, as the authors note, it raises troubling questions about the validity of all typical tests of achievement used to evaluate student achievement and school effectiveness. Because if the tests reward content knowledge but not powerful thinking–yet, all Standards highlight important thinking–then the tests may be yielding invalid inferences and thus very harmful consequences.
Which won’t surprise anyone who has been paying attention to the reform agenda of the last 50 years. But it should make a whole lot of traditionalists–in psychometrics as well as in classrooms–do some re-thinking.
[Thanks to Rob Ackerman for alerting me to this research.]
This article first appeared on Grant’s personal blog; Students Can Do Well On Tests & Still Not Understand; What Tests Actually Measure