Schools for Tomorrow Blog

How much testing do we need?

Now before you accountability absolutists start hyperventilating, rest assured that I’m not one of those bleeding hearts who wants to abolish CSAPs because they traumatize sensitive 4th graders with iffy math skills, cost millions or tempt jumpy principals to flirt with bribery.

What I wonder about, though, is whether we’re learning anything new from standardized tests.

August is a time to look forward, with school starting, but it’s also the month to look back, because it’s test results season.

The 2008 CSAP results were released a couple of weeks ago, the national ACT results came out Wednesday and the annual SAT report is coming soon.

Because we Americans are addicted to stats and scores (just listen to those announcers at the Olympics), every release of test results is parsed exhaustively. Gains of a few tenths of a percent at the local elementary school are hailed as encouraging progress, and comparisons are eagerly made between districts, and between a district’s scores and state averages.

None of that may mean much at all. (Sorry DPS.)

Over time, if you look at CSAP percentages of advanced or proficient or at ACT average composites, it’s striking how little change there is. The numbers fluctuate within single digits over several years.

When you dig deeper into the scores, you see the same patterns you saw the year before and the year before that. Among them:

Minority kids don’t do as well as white kids.
Poverty is a factor.
There are differences between boys and girls.
Overall performance declines as kids get older.
There’s an astonishingly low level of math and science proficiency.
High schoolers who take a more rigorous class schedule do better on the ACT.
Most kids aren’t ready for college work.

No, we shouldn’t do away with standardized tests. The CDE’s growth model offers some potentially useful new ways to look at the annual avalanche of CSAP stats. (Although, growth stats also may just tell us what we already know. As CDE assessment whiz Rich Wenning said at the CSAP news conference on July 29, “Our lowest achieving students are not making enough growth to catch up.” He repeated that comment to the State Board of Education on Aug. 13.)

The real questions are what can be done to help them catch up, and how many of them can realistically be expected to catch up?

While it is probably human nature to look to solve a particular problem with a “silver bullet” (magic pill, pixie dust, panacea, Batman, whatever) – that single, usually simple solution rarely seems to exist in the complex world in which we actually live.

Certainly education policy research, as well as 30+ years of pretty well-analyzed experience on the ground, suggests that no silver bullet will solve yawning achievement gaps and lower than desirable average performance, whether in Colorado or any other state.  But, we tend to continue looking, instead of focusing our efforts on implementing a handful of policies that show at least some promise of gains.

I am reminded of this most recently by the CSAP results in Denver and Aurora.  Hopefully, those positive results are more indicative of a real turnaround taking hold in those urban districts, and not a one-year blip or statistical “regression to the mean” type result.

Assuming the more positive outlook, what is interesting is the different approaches in these neighboring urban districts.  While there are some overlaps, Aurora’s former military leader John Barry has pursued a series of reforms related to the Broad Foundation training he received to become a superintendent.  I know there was some initial skepticism in the Colorado reform community about these approaches, but they may now be paying off.

Michael Bennet in Denver focused upon The Denver Plan, better literacy, reading, and ESL training, middle school focus, and a variety of leadership and quasi-autonomy reforms.  They too seem to be working.

On the other hand, Mapleton has been the district that many reformers have highlighted as doing lots of great things.  But, their CSAP scores have been moving in the opposite direction, and not just this year – perhaps a long implementation dip, but you have to worry a bit.

All of this says to me that we know a few things that work, but a lot more effort needs to be expended in combining them and really implementing them thoughtfully, not just chasing the next magic fad.  And, that takes a little time and patience, not as an excuse, but as a reality.

 

Before we can move on to more earth-shaking matters, here’s another interesting side note from Tuesday’s release of CSAP results: The lack of a math gap between boys and girls at nearly every grade level.

This comports with a recent in-depth study from the National Science Foundation (NSF) that found no difference in mathematical ability between the sexes. Of course, stereotypes die hard, but we need to move beyond this notion when crafting education policy here in Colorado and abroad.

One noteworthy thing NSF researchers stumbled upon concerns the quality of math assessments:

Again, the effort uncovered little difference, as did a comparison of how well boys and girls did on questions requiring complex problem solving. What the researchers did find, though, was a disturbing lack of questions that tested this ability. In fact, they found none whatsoever on the state assessments for NCLB, requiring them to turn to another data source for that part of the study.

As usual, the best take on the subject comes from The Onion (H/T Joanne Jacobs), with a “man-in-the-street” survey to gauge opinion of the NSF’s findings. Pick your favorite answer.

One final note: The female advantage on CSAP reading assessments far outstrips the tiniest male edge in the subject of math. Maybe the NSF or another group could take a look at comparative skills in this area.

 

Those of you who have been following this or other ed blogs, or whose lives are sufficiently empty that they read Ed Week, have probably noticed a certain pattern: the arguments about what to do to fix the system are frustratingly circular.

Take ProComp.  The teachers union argues that the bulk of additional public money should go to more experienced teachers.  DPS says the money should focus on attracting new blood to the system.  Folks on the outside wonder whether more money for teachers, young or old, will lead to better results.  But here’s something none of the advocates have said:  "If you spend the money my way, I can prove that it will result in 24 more Hispanic kids graduating fully prepared for college.  That’s 15 more students than my opponent’s strategy." 

In fact, no one will say what the results of ProComp, or almost any other expenditure, will be in terms of a correlation between the investment and the result.

Let me illustrate the point a different way.  What does it cost a school district to produce one low-income Hispanic student who graduates high school ready to attend a four-year college?  What district in Colorado is most efficient in achieving that result?  How large a dollar difference is there between the best district’s performance in this category and the worst?  What does it cost the best district in the nation? Is there anything the best performing district does differently that might account for the better performance, and save taxpayers money? 

I can’t answer that question, and neither can you.  In fact, no one can.

Why?  Because no one measures the cost of outcomes.  No one measures what it costs to achieve a desired educational result. 

Even more odd, few districts even bother to define what a desired educational result might be.  Remember, it wasn’t until this year that there was even a national standard for defining the drop-out rate.

Remember the Denver Plan?  Here’s its first objective: 

All students will engage at every grade level in a rigorous course of study in the Denver Public Schools and, upon graduation, will exceed state performance standards in four core subject areas (literacy, math, science and social studies); be prepared to succeed in college/other post-secondary opportunities; and be critical thinkers.

The problem with this objective is that it is non-quantifiable.  It says, for example, that upon graduation, all students will exceed the state performance standards.  But it doesn’t say:  "All students will graduate".  Nor does it say they whether its a two or four year college, or whether they will "be prepared to succeed in college without remediation".  No numbers, no where, no how.

Now, according to the Colorado Department of Education (CDE) "report cards", DPS spends upwards of $700 million annually, or roughly $10K per student per year.  Is that money being well spent?  The only way to answer that is to have some measurable outputs, and see how they are doing. 

Let me digress by pointing out that this essay is not about DPS, and that, as I will explain over time (since I intend to write about this a lot), DPS actually understands the nature of this problem better than almost any district in the nation.  I could use Aurora, or Mapleton, or for that matter Cherry Creek or Scarsdale, and the point would be the same.  At least DPS has a plan.  Most districts nationwide don’t even have a mission statement saying that graduating students able to perform at a certain level is what the district is supposed to do.  There are no agreed-upon output standards, and because they don’t exist, it is impossible to measure performance against dollars invested.  So instead we waste a lot of time arguing about dollar inputs.

What might an output standard be?  Well, the Greenberg Standard for being considered "college ready" is scoring proficient or higher on the 10th grade CSAPS, and achieving a 20 or higher on the ACT composite.  Somewhat arbitrary, but it more or less aligns with the CCHE college admission index, and the data is more or less accessible from the CDE homepage.  It basically means a student could attend UNC or Western State without the need for remediation.

So how much does it cost DPS to generate one low-income Hispanic student who graduates, by the Greenberg Standard, "college ready"?

The primitive calculus goes something like this:  Take the cohort of all the free/reduced fund eligible Hispanic students who enrolled in DPS over the past 13 years as potential members of the Class of 2008.  Multiply that by $8000, which roughly represents, in constant dollars, the amount of per pupil revenue and federal money that was allocated annually to educate these kids.  Now, find the total number of kids in the cohort who actually graduated in 2008 having met the Greenberg Standard.  Use that as the denominator…for those a bit slow in math, that means take all the money DPS spent on all the kids, and divide it by the number of kids who met the standard.  Throw in a "fudge factor" to account for kids who left the DPS system but didn’t drop out and met the standard.

What you will find, in general terms, is that for every 100 shiny happy Hispanic boys and girls who walked into kindergarten with their heads held high, perhaps 3 walked out in 2008 being college ready, at least by the Greenberg Standard.  That’s a 97 percent failure rate, although if you take into account the "fudge factor", maybe its only a 90 percent failure rate.  Either way it means DPS invests somewhere north of $4 million to produce one "success"…and believe me, that’s a low estimate. 

By contrast, I can send a kid to Graland and Colorado Academy for roughly $250,000 (constant dollars) with a high level of confidence that the child, regardless of race or family income, will come out college ready.

I’ve deliberately picked a hard case; you could argue that the Greenberg Standard is unreasonably high.  If we asked how DPS performs with white, middle-class girls, my guess is that the output costs would be pretty reasonable, compared to other districts and private schools.  But no one knows, because no one keeps the data or asks the question.

No doubt there are dozens of flaws in my methodology, but it is "directionally correct" and it does illustrate the problem.  We don’t know, and we don’t talk about, what the outcome costs are.  And if we don’t know, then what is the point of debating who gets the crumbs in ProComp, or whether charter schools are a good or bad investment for a district, and a million other questions?

So next time you advocate a specific reform, tell us not just what it costs but how it will change the cost of a given outcome…that should save a lot of trees, and, in the case of blogs, make life easier on those poor, innocent electrons.

 

Blogfather’s note: Celeste Archer, a veteran teacher, is leaving Denver this summer and will resume her teaching career in her native Arkansas. This is one of several parting blog posts.

 This blog entry was meant to explain why educators and a lot of other good thinkers have such a problem with No Child Left Behind – or as I like to call it, Every Child Left Behind.  But, as I was cleaning out my e-mail in-box, I found the article below.  It had been sent by one of the good thinkers.  This analogy says it so well that there’s not much to add.  Have no doubt, I’m sure some will take exception and over-think, but, please, take just a moment and try to hear what is being said.  It IS valid.

No Dentist Left Behind

My dentist is great! He sends me reminders so I don’t forget checkups. He uses the latest techniques based on research. He never hurts me, and I’ve got all my teeth.  When I ran into him the other day, I was eager to see if he’d heard about the new state program. I knew he’d think it was great.

"Did you hear about the new state program to measure effectiveness of dentists with their young patients?"

"No," he said. He didn’t seem too thrilled. "How will they do that?" "It’s quite simple," I said. "They will just count the number of cavities each patient has at age 10, 14, and 18 and average that to determine a dentist’s rating. Dentists will be rated as excellent, good, average, below average, or unsatisfactory. That way parents will know which are the best dentists. The plan will also encourage the less effective dentists to get better," I said. "Poor dentists who don’t improve could lose their

licenses to practice."

"That’s terrible," he said.

"What? That’s not a good attitude," I said. "Don’t you think we should try to improve children’s dental health in this state?"

"Sure I do," he said, "but that’s not a fair way to determine who is practicing good dentistry."

"Why not?" I said. "It makes perfect sense to me."

"Well, it’s so obvious," he said. "Don’t you see that dentists don’t all work with the same clientele, and that much depends on things we can’t control?  For example, I work in a rural area with a high percentage of patients from deprived homes, while some of my colleagues work in upper middle-class neighborhoods. Many of the parents I work with don’t bring their children to see me until there is some kind of problem and I don’t get to do much preventive work. Also, many of the parents I serve let their kids eat way too much candy from an early age, unlike more educated parents who understand the relationship between sugar and decay. To top it all off, so many of my clients have well water which is untreated and has no fluoride in it. Do you have any idea how much difference early use of fluoride can make?"

"It sounds like you’re making excuses," I said. "I can’t believe that you, my dentist, would be so defensive. After all, you do a great job, and you needn’t fear a little accountability."

"I am not being defensive!" he said. "My best patients are as good as anyone’s; my work is as good as anyone’s, but my average cavity count is going to be higher than a lot of other dentists because I chose to work where I am needed most."

"Don’t get touchy," I said.

"Touchy?" he said. His face had turned red, and from the way he was clenching and unclenching his jaws, I was afraid he was going to damage his teeth. "Try furious! In a system like this, I will end up being rated average, below average, or worse. The few educated patients I have who see these ratings may believe this so-called rating is an actual measure of my ability and proficiency as a dentist. They may leave me, and I’ll be left with only the neediest patients. And my cavity average score will get even worse. On top of that, how will I attract good dental hygienists and other excellent dentists to my practice if it is labeled below average?"

"I think you are overreacting," I said. "’Complaining, excuse-making and stonewalling won’t improve dental health’… I am quoting from a leading member of the DOC," I noted.

"What’s the DOC?" he asked.

"It’s the Dental Oversight Committee," I said, "a group made up of mostly lay persons to make sure dentistry in this state gets improved."

"Spare me," he said, "I can’t believe this. Reasonable people won’t buy it," he said hopefully. 

The program sounded reasonable to me, so I asked, "How else would you measure good dentistry?"

"Come watch me work," he said. "Observe my processes."

"That’s too complicated, expensive and time-consuming," I said. "Cavities are the bottom line, and you can’t argue with the bottom line. It’s an absolute measure."

"That’s what I’m afraid my parents and prospective patients will think. This can’t be happening," he said despairingly.

"Now, now," I said, "don’t despair. The state will help you some."

"How?" he asked.

"If you receive a poor rating, they’ll send a dentist who is rated excellent to help straighten you out," I said brightly.

"You mean," He said, "they’ll send a dentist with a wealthy clientele to show me how to work on severe juvenile dental problems with which I have probably had much more experience? BIG HELP!"

"There you go again," I said. "You aren’t acting professionally at all."

"You don’t get it," he said. "Doing this would be like grading schools and teachers on an average score made on a test of children’s progress with no regard to influences outside the school, the home, the community served and stuff like that. Why would they do something so unfair to dentists? No one would ever think of doing that to schools."

I just shook my head sadly, but he had brightened. “I’m going to write my representatives and senators," he said. "I’ll use the school analogy. Surely they will see the point."

He walked off with that look of hope mixed with fear and suppressed anger that I,a teacher, see in the mirror so often lately.

If you don’t understand why educators resent the federal NO CHILD LEFT BEHIND ACT, this may help. If you do understand, you’ll enjoy this analogy.  It was forwarded by John S. Taylor, Superintendent of Schools for the Lancaster County, PA, School District.

It isn’t that we don’t agree that there should be accountability.  Being educators, though, we want smart and meaningful measures for that accountability.  We want funding for the mandated programs.

It’s hard when legislators and politicians, very few of whom have had much experience in education, especially public education, try to create law around something they know little about.  Having worked in Houston under the leadership of Rod Paige, it would be a difficult argument to make with me that there was good thinking behind the origins of No Child Left Behind. 

Then, when the legislators seek advice, it seems to come from wonks, academics and philanthropists (all of whom we educators need and want for their guidance and enrichment), excluding the very people involved in the day to day gritty business of the field. 

One of these former legislators in Colorado is a known opponent to public education. In this case it is ironic since he comes from a family devoted to the education and enrichment of children – his family is instrumental in the founding and development of a camp in the southern part of the state and they are absolute pillars of the college from which I received my undergraduate degree. 

My college also has an ECE through high school sister campus.  It was shocking to me when I moved here and discovered his attitude – it is elitist and exclusive, with little to back his conclusions.  I understand he went through that private school system his family helped create and has no knowledge of a public school experience.  Still naïve about how different he was than the rest of his family, I invited him into my classroom. 

As we say in the south, he looked like a doe caught in the headlights.  But this man is or was, although he still appears on public television, considered somewhat of an educational spokesperson for this state, particularly the more conservative thinkers. 

I’ve no doubt it’s a story that could be oft-repeated.  And, it is a story that, if not given a different ending, will eventually bring about this guy’s ultimate goal – the demise of public schools and a divisive privatization of our children’s education. 

There’d no better way to assure an achievement gap than that – and it would be the mother of all gaps.

 

It’s the middle of the summer, and I’m writing a post about No Child Left Behind. Someone please check my mental health (of course, most readers here have long been skeptical). Frankly, I couldn’t help it, after reading this enlightening post by David Hoff at Education Week (H/T Eduwonk):

Today, the Public Education Network released a poll that sheds some light on the reason why. Although the poll focuses on where education stands in the current political debate, the response to one of its questions shows that the minority community likes NCLB.

Forty-one percent of blacks and 39 percent of Hispanics believe that NCLB has helped improve their schools. Only 21 percent of African-Americans and 23 percent of Hispanics say the law is hurting their schools. (The rest says there’s no difference.) By contrast, 27 percent of whites say the law is helping schools, 31 percent say it is hurting, and 27 percent say it hasn’t had an impact.

Combine the three groups and here’s what you get: 31 percent say “helping;” 31 percent say “hurting;” and 27 percent say “no difference.”

Seldom do you see public opinion results so evenly divided three ways on a major question. Anyone who uses the term consensus to describe public opinion in a debate about NCLB should be ridden out of town on the rails.

Think about the different factors out there that have worked to push public opinion against NCLB:

- The National Education Association has spent lots of good money on a front group designed to rail ceaselessly against the requirements of NCLB – Its membership, largely unhappy with NCLB, makes a vast activist network throughout the nation’s public schools

- Barack Obama, the so-called great agent of hope and change, has parroted the NEA’s line on NCLB being too rigid and underfunded

- Many hard-core conservatives have never liked the growth of federal government intrusion into education, and have grumbled ever more loudly about it in recent years – many Congressional Republicans proposed scrapping the NCLB model for the more flexible A-Plus Act

- The program is unmistakably attached to the agenda of an unpopular President, who has used the bully pulpit throughout his term in office to trumpet NCLB

Yet in the end, none of this has seemed to deliver a crushing blow to the way Americans feel about the federal government’s reigning education program. Notably, racial and ethnic minorities tend to believe NCLB has improved their schools, an observation which must give us pause. All its flaws aside, the program is doing something to move toward closing the achievement gap.

Reform is needed, and a debate must ensue. But we can’t take the “scrap NCLB and start over” approach. As one recent president famously said, “Mend it, don’t end it.”

 

There are rumblings emanating from Boulder Valley School District, and they don’t have anything to do with exploding tofu factories or yoga studios.  Rather, the hubbub is due to a parent petition that would split the district right down the middle. 

The move is opposed by district bigwigs, including the superintendent and the president of the school board, and for good reason.  Splitting the district would not only be logistically onerous, but it would compromise the quality of one of the best school districts in the state.

BVSD is surprisingly diverse.  It’s 55 schools are spread over 500 square miles, and encompass affluent Boulder enclaves, newer suburbs, working class neighborhoods, and mountain towns.  The petition highlights an ongoing conflict in Boulder and other districts throughout the country.  A natural tension exists between the parents, usually white and affluent, who are deeply involved in their children’s schools, and the district and community at large, who are charged with providing a high quality education to all kids in the district, not just those who happen to live in wealthier areas. 

While it is laudable for parents to be involved in their kids’ schooling, too many parents are getting caught up in the “me first” thinking that creates systemic inequities in the first place.  Splitting BVSD would magnify the educational inequities that already exist in the district.  For example, while the district’s CSAP scores typically exceed the state average, its achievement gap between white and Latino students also ranks near the widest in Colorado.

Rather than Balkanize a high-performing district, Boulder Valley’s parents should work to make sure that every child in the district has access to a high quality education, whether they live in a two million dollar mansion on two room apartment.

The Colorado Children’s Campaign’s released its annual Kids Count report last week, and the news was shocking enough to merit a front-page spread in the Post and even an article in the New York Times.  It seems that the number of children living in poverty in Colorado jumped faster than in any other state over the past few years.   In fact, Colorado’s increase of 73 percent dwarfed the next highest state, New Hampshire, which had 50 percent.

Not only are more kids living in poverty now than in 2000, but more live at extreme levels of poverty (50 percent of the federal poverty level or lower), and the trends hold across ethnic lines. 

So, once  we have prepared our fingers for pointing, where shall we aim them?   Colorado’s dismal performance seems to be a result of a combination of increased child care expenses, a tough economy, and a school system, on average, that is struggling to rise out of mediocrity.  Our state has also seen fairly dramatic increase in the number of immigrant families, who are likely to be poorer, although the ratio of immigrant to non-immigrant families has stayed roughly constant.

Our high child poverty rates should be cause for alarm, as they foretell economic and social woes in years to come.  As these children grow into adults, they will be statistically more likely to commit crimes, less likely to attend college, more likely to require more expensive health care, and less likely to hold high-paying jobs.

And what to do about it all?  While some of our state’s economic woes are our own fault (see my frequent punching-bag TABOR, along with other financial missteps), I suspect much of the problem is tied to bigger trends, such as increases in the cost of living for working people. 

I say we start with those things we know work to help bring kids out of poverty – education and child care.  Both should be accessible to all children, no matter what job their parents happen to hold.