Schools for Tomorrow Blog

How much testing do we need?

Now before you accountability absolutists start hyperventilating, rest assured that I’m not one of those bleeding hearts who wants to abolish CSAPs because they traumatize sensitive 4th graders with iffy math skills, cost millions or tempt jumpy principals to flirt with bribery.

What I wonder about, though, is whether we’re learning anything new from standardized tests.

August is a time to look forward, with school starting, but it’s also the month to look back, because it’s test results season.

The 2008 CSAP results were released a couple of weeks ago, the national ACT results came out Wednesday and the annual SAT report is coming soon.

Because we Americans are addicted to stats and scores (just listen to those announcers at the Olympics), every release of test results is parsed exhaustively. Gains of a few tenths of a percent at the local elementary school are hailed as encouraging progress, and comparisons are eagerly made between districts, and between a district’s scores and state averages.

None of that may mean much at all. (Sorry DPS.)

Over time, if you look at CSAP percentages of advanced or proficient or at ACT average composites, it’s striking how little change there is. The numbers fluctuate within single digits over several years.

When you dig deeper into the scores, you see the same patterns you saw the year before and the year before that. Among them:

Minority kids don’t do as well as white kids.
Poverty is a factor.
There are differences between boys and girls.
Overall performance declines as kids get older.
There’s an astonishingly low level of math and science proficiency.
High schoolers who take a more rigorous class schedule do better on the ACT.
Most kids aren’t ready for college work.

No, we shouldn’t do away with standardized tests. The CDE’s growth model offers some potentially useful new ways to look at the annual avalanche of CSAP stats. (Although, growth stats also may just tell us what we already know. As CDE assessment whiz Rich Wenning said at the CSAP news conference on July 29, “Our lowest achieving students are not making enough growth to catch up.” He repeated that comment to the State Board of Education on Aug. 13.)

The real questions are what can be done to help them catch up, and how many of them can realistically be expected to catch up?

I’m grateful to Alan for the chance to contribute to this blog, even if the timing means my initial post is a self-referential link to this Op-Ed in today’s Denver Post.

The Post’s editors contributed the title and made a few edits for space. The deleted text was this: To draw even with the state on student proficiency within a decade, DPS realistically needs an increase in CSAP of at least 3 points every year. However as DPS has never made 3 overall points of progress in any single year, much less for 10 years straight, the chances are roughly equal to the Broncos winning the Beijing Olympics.

So whether one sees the recent DPS scores as a glass half-empty or half-full, Denver’s considerable talent, effort and money have produced, at best, incremental improvement. And it should also be clear by now that incremental improvement is not enough to provide another generation of Denver’s children with the educational opportunities they need.

My personal belief - not in the Op-Ed - is that reform undertaken at a slow pace is likely doomed to failure, and that the recent admirable gains from DPS can either serve as an accelerator to greater reform, or if taken as evidence that the current efforts are adequate, will soon be a historical blip.

For example, see this DPS press release about recent substantial improvements on the CSAPs. Just don’t miss that it is from August of 2005. History is a strong opponent.

My question is this: If one believes DPS is indeed making adequate yearly progress (sorry), is it reasonable to wait another generation or more just to equal to the Colorado average? If not, what should be done? I hope some of you will have ideas that eclipse mine.

Before we can move on to more earth-shaking matters, here’s another interesting side note from Tuesday’s release of CSAP results: The lack of a math gap between boys and girls at nearly every grade level.

This comports with a recent in-depth study from the National Science Foundation (NSF) that found no difference in mathematical ability between the sexes. Of course, stereotypes die hard, but we need to move beyond this notion when crafting education policy here in Colorado and abroad.

One noteworthy thing NSF researchers stumbled upon concerns the quality of math assessments:

Again, the effort uncovered little difference, as did a comparison of how well boys and girls did on questions requiring complex problem solving. What the researchers did find, though, was a disturbing lack of questions that tested this ability. In fact, they found none whatsoever on the state assessments for NCLB, requiring them to turn to another data source for that part of the study.

As usual, the best take on the subject comes from The Onion (H/T Joanne Jacobs), with a “man-in-the-street” survey to gauge opinion of the NSF’s findings. Pick your favorite answer.

One final note: The female advantage on CSAP reading assessments far outstrips the tiniest male edge in the subject of math. Maybe the NSF or another group could take a look at comparative skills in this area.

Back in May, I asked blog readers to give their over/under guestimates for the 2008 CSAP results.  Save for Alex Ooms, no one took up that challenge.  I continue to be amazed that we talk/gripe a lot about CSAPs, but when they actually come out, we provide no context to what the results might really mean, only some post hoc “spin.” 

In that spirit, then, I’ll offer some of my own spin. 

It is particularly hard to assess test score results, at least statewide, when they look like this year’s – a mildly positive trend, or flat.  The half full perspective is “oh, we are continuing to improve, slower than we would want, but at least improving.”  The half empty take is “wow, all this reform effort and almost nothing to show for it, and at this rate, it will take until 2050 for all of Colorado’s students to be proficient.” 

More and more, I don’t think the year over year CSAP results tell us much of anything.  The growth model results in August will be more helpful, but a longer trend is more instructive, too, as in this chart from the Rocky, for 2004-8.

Here, out of 27 tests (math, reading, writing in grades 3-10 and science in few grades), aggregate state scores are up for 19, down for 6 and even for 2, over the 5 year period from 2004-8. Overall, that is positive, showing improvement. However, none of the grade level tests in any subject show more than a 6 percentage point increase in the percentage proficient over five years. 

Thus, with a median across grade levels of about 65% proficient in reading, 55% in math,  50% in math, the state would need anywhere from 30-45 more years of the very best improvements that have ever been made in any single CSAP test in any grade level to get everyone to proficiency.  Not gonna make that 2014 NCLB target of 100% proficiency.

Actually, the most striking thing about the Rocky chart is how much math proficiency levels decline with grade level.  While Colorado 3^rd graders are in the 70% proficient range, by 10^th grade only about 30% demonstrate proficiency.  While problems with 10^th grade math CSAPs often have been noted, and this decline does partially mirror national trends, it is really disturbing and incredibly problematic for STEM pipeline hopes.

Of course, while we would all like it to not be the case, in the aggregate, test scores and family SES are highly correlated.  Breaking that correlation, in a setting beyond a single exceptional school or two, is really hard and really rare.  And, aggregate results are difficult to move.  In fact, the CSAP results over the past five years are probably a bit better than they appear, since Colorado’s percentage of free/reduced lunch students, a reasonable measure of students in poverty, has grown from 31% to 35% over that period, suggesting that Colorado has been teaching and testing more at risk kids each year.

On the less aggregated level, it appears that DPS did much better this year, making really strong gains in middle school tests and smaller but solid gains in nearly all grade levels.  One can hope that the determined and focused reform efforts of Michael Bennet and company are beginning to pay off, or perhaps, could it be ProComp incentives . . .?

Seems like Denver Public Schools is overdue for some good news: Even as the rest of the state treads water, today’s release of CSAP results underscores a positive trend in Colorado’s 2^nd largest school district:

Denver Public Schools made substantial 3.6 percentage point gains in reading scores, according to an internal memo obtained by Education News Colorado.

DPS also outpaced the state in terms of reading gains at all eight grade levels tested, the memo said. However, the urban district still lags far behind Colorado’s overall reading scores.

The Denver district made more modest gains in writing, mathematics and science, though even those gains – in the vicinity of 1 percent –  bested the state, which had a flat year.

“Though we always aspire for better outcomes, we are pleased with this year’s CSAP results,” DPS’ departing Chief Academic Officer Jaime Aquino wrote in an email to district staff.  “The scores are a validation of the work we are doing.  It shows that we are heading in the right direction and that our trends are promising since we continue to outpace the state’s change.”

Time for the crack research teams to dig deeper. What factor(s) might be making a difference here? Or is Denver’s performance just a fluke? Inquiring minds demand answers, while experienced souls anticipate a long time to learn any answers, if any are to be learned.

In the meantime, news like this just might give Superintendent Michael Bennet the momentum he needs to continue pushing his ambitious reform agenda.

But it doesn’t put the teachers union in a good place. Already having struggled to shed the well-earned mantel of obstructionists in the Bruce Randolph autonomy struggle, the DCTA’s latest position against a proposed upgrade in the innovative and nationally-observed ProComp teacher pay program now becomes more precarious.

The district still has a long way to go in raising performance and closing the achievement gap, but today brings more solid proof that it’s moving in the right direction. So does DCTA tout the improvement and concede that Superintendent Bennet has earned the benefit of the doubt in moving forward? Not likely. Or does DCTA downplay the news and forfeit what remains of the good faith earned as original partners in ProComp?

My guess is they stay silent as the mediation process continues and they ponder the potentially suicidal strategy of a strike during the Democratic National Convention. At least the intrigue keeps us paying attention during the slow summer months.

Today’s press conference unveiling 2008 CSAP scores was, when you get right down to it, much ado about, well, nothing much.

Windy speeches abounded, the crowd in the packed lobby of the Colorado Department of Education’s ornate HQ applauded dutifully at all the right moments. The word of the day was ‘mixed,” as in CSAP results were “mixed,” which is a polite synonym for not so great, or maybe even lousy.

As usual, some schools and districts did better than expected while others did worse. This year, Denver Public Schools and Aurora Public Schools exceeded expectations, giving those beleaguered urban districts, and their break-the-mold superintendents a boost, and raising expectations for next year.

But the whole event had an anticlimactic air about it, because the big shoe has yet to drop. That would be detailed data from the gaudy new Colorado Growth Model, also known as Information That Actually Means Something. We got a taste of the growth model today, with the release of statewide data.

Even that general data gives a fascinating peek behind the scenes of, say, the achievement gap. One tidbit: only 13 percent students rated unsatisfactory or partially proficient on the math CSAP are growing fast enough to get them to proficiency in three years. (Look for stories on this and other CSAP subjects on the Education News Colorado web site).

Late next month, when CDE releases growth model data for districts and schools, we’ll really be able to tell what’s going on, education-wise in Colorado – at least compared to the woefully incomplete and shallow stuff we’ve been fed until now. I’m sure the growth model data won’t be perfect, but having it at our disposal will be akin to moving from black and white to color, from two dimensional to three dimensional, from mono to stereo.  

The prolific research tandem of Jay Greene and Marcus Winters – this time teaming up Arkansas Tech’s Julie Trivett – released a Manhattan Institute report today with some interesting findings. Taking a look at Florida’s strong school accountability program, the team measured the effects of high-stakes testing sanctions on low-stakes subject performance:

The primary findings of the study are:

• The F-grade sanction produced after one year a gain in student science proficiency of about a 0.08 standard deviation. These gains are similar to those in reading and appear smaller than the gains in math that were due to the F sanction.
• There is some evidence to suggest that student science proficiency increased primarily because student learning in math and reading enabled that increase. That is, learning in math and reading appear to contribute to learning in science.

Boiling it down into simpler terms, the first-of-its-kind report strongly suggests that high-stakes testing doesn’t “crowd out” how well students do in lower-stakes subjects. As much as many in Colorado’s education establishment – and leading House Education Committee figures – rail against the CSAP concept and the evils of “teaching to the test,” such findings may throw a wrench into the conventional wisdom.

More studies are needed to unwrap the story hinted at in Greene’s and Winters’ research, but let’s just entertain the possibility. Could it be that CSAP stakes aren’t too high, but that they are in fact too low? Lawmakers would be wise to emulate Florida and toughen school accountability, and maybe reap the same results as the Sunshine State.

The reality of yet another year of basically flat third grade CSAPs scores, after more attempts at reforms, generated a lot of heat on this blog, with everyone trying to figure out what it means for future reforms.  

But, I continue to be frustrated by the ad hoc nature of the ex poste “spin” around test score results.  How about a discussion of what we would realistically hope to see from CSAPs, or other standardized tests.

Before the full results come out this summer, what would a positive result look like for Colorado?  We know that a decline in test scores would be considered quite bad, though for districts or schools undergoing major reform it is usually optimistically called an “implementation dip.”  Flat is not very impressive.

But, how positive should positive be?  NCLB wants no child left behind and all students up to proficiency by 2014.  For third grade reading statewide, with 71% proficient, that would require increases in the proficiency numbers of about 5% points each year (e.g., 29 points over 6 years).  That is our current national goal.  Anyone expect that to happen in Colorado?

Statewide improvement of more than 5 points would be looked upon with skepticism, about cheating, extreme teaching to test, or actual errors.  Statistically, with such large numbers, such huge improvement seems highly unlikely.  

Obviously, any positive movement is better than flat.  But a point or two, would that be worth celebrating, with all of the efforts underway, plus the very real likelihood that schools are spending a lot more time teaching at least towards the test, if lots of anecdotes and stories are to be believed?

I think that an appropriate but ambitious goal is to aim for annual increases in the 3-5 percent range.  Over five years, that would lead to cumulative 15-25% growth, which would be huge if the gains could be sustained.

Of course, “value-added” testing makes much more sense, actually measuring what the same set of students have learned from one point in the year to a later time – because the actual kids are held “constant.”  Without value added tests, we don’t know much.  It is at least possible that Colorado schools really did do a better job with third grade reading in 2007-8 than last year, but this year might have featured a 3rd grade school population with more at-risk students than last year, leading to flat aggregate scores.  

What is your over/under for the full CSAP results to be released this summer?