Schools for Tomorrow Blog

The U.S. Department of Education’s What Works Clearinghouse has launched a new section called Quick Reviews.  According to DOE’s website:

What Works Clearinghouse (WWC) quick reviews are designed to provide education practitioners and policymakers with timely and objective assessments of the quality of the research evidence from recently released research papers and reports. These reviews focus on studies of the effectiveness of education or school-based interventions serving students in the pre-kindergarten through twelfth grade age range, as well as those in a post-secondary setting.

The What Works Clearinghouse is a product of the Department of Ed’s effort to summarize the current expert thinking on topics of current interest in educational research.  Unfortunately, as with so many Bush-administration attempts to marry science and politics, the Clearinghouse has been a frequent target of criticism from the research community.

It seems that the Clearinghouse has narrowed the definition of what constitutes quality research to such an extreme degree that much good research is excluded.  (See this recent EdWeek article for more on the controversy).  In general, What Works places a premium on quantitative, true-experimental designs.  While these designs do often produce compelling findings, they are frequently either impossible or inappropriate to achieve in ed research.  Many researchers whose work was rejected by What Works also claim a political motive; I’ll let research experts be the judge of that.

Most of us can agree that more high-quality educational research is needed, and that more needs to be done to separate the research wheat from the chaff.  Too bad the What Works Clearinghouse isn’t yet up to the task.

Andrew Rotherham’s EduWonk blog reports this week on a pair of studies new studies about charter schools. 

The first study, a RAND report on Chicago’s charter school program, examined whether charter schools were “skimming” better students, whether they produced achievement gains relative to district-run Chicago schools, and whether their students had better long-term outcomes.  The authors conclude that charters seem to be successful on all counts:  they do not seem to be “skimming” better students, they seem to produce higher academic outcomes, and their students are more likely to graduate from high school.

The second study, awkwardly titled “The Muzzled Dog That Didn’t Bark”, examined the response to charter schools in Washington, D.C.  This study, conducted by researchers at George Washington University, found that public schools have not responded as expected to charter schools because of “a lack of commitment to a truly competitive model that incorporates non-trivial consequences for failure.”  According to one of the study’s authors, “although it has been suggested that school choice will spur competition and improvements in public education, choice alone does not necessarily equate to competition.”

These two studies are interesting additions a recent spate of research on school choice, including reports by Greene and Winters, West and Peterson, Hosby, Belfield and Levin, and others.  (For another take, see Uncle Charley’s summary of a couple of weeks ago).

Together, this body of research seems to suggest that charter schools can, in fact, be better alternatives to typical district schools, but only when done in the right policy context and with buy-in from schools and teachers.  To my eye, the RAND article is particularly compelling, in part because RAND has a better reputation for non-ideological work than others in the field (see, for example, Peterson and Greene, whose work on school choice, to my knowledge, have literally never produced a negative result.  Makes ‘ya wonder …)

Ideology notwithstanding, in short there seems to be enough evidence of the promise of charter schools to support their growth.  But the story is far from over.

A recent study finds that small class sizes help students.  USA Today reports:

Breaking up large classes into several smaller ones helps students, but the improvements in many cases come in spite of what teachers do, new research suggests.

New findings from four nations, including the USA, tell a curious story. Small classes work for children, but that’s less because of how teachers teach than because of what students feel they can do: Get more face time with their teacher, for instance, or work in small groups with classmates.

"Small classes are more engaging places for students because they’re able to have a more personal connection with teachers, simply by virtue of the fact that there are fewer kids in the classroom competing for that teacher’s attention," says Adam Gamoran of the University of Wisconsin-Madison, who analyzed the findings.

The study is reminiscent of the famous Tennessee class size study, which yielded similar findings:  smaller classes are better for kids.  What made that study remarkable was that it was one of largest “true experiments” ever conducted in the field of education.  7,000 students were randomly assigned to one of three types of “treatments”, based on the student-teacher ratio in the class.

The Tennessee study is unusual, indeed.  What makes education research so confounding is that it simply does not conform to the typical experimental methods available in other fields like medicine of physics.  For a variety of logistical, financial, political, or ethical reasons, we usually cannot create the “gold standard” scientific experiment, in which children are randomly assigned to treatments and their results are precisely tracked over time. 

In the absence of the type of experimental control available in other disciplines, we’re often forced to conjecture, hypothesize, or just plain guess more often than we’d like.  In the world of school choice research, for example, there have been few, if any, studies in which children have truly been randomly assigned to one type of school or another. 

Instead, we must draw conclusions from the limited information available, which is problematic because kids who choose different school choice options may be different:  their motivations may be different; their parents may be different; their skin color or religion or language may be different.

In the rare case in which a true experiment is possible – as in the Tennessee study – educators gobble up the results.  But until we’re all allowed to randomly assign kids into courses or schools or districts like so many laboratory mice, we’ll have to acknowledge that our passionately held policy positions are based on a little more conjecture and guesswork than we’d like.

Let me add one more salvo to the ongoing discussion about education research and bias.  The thoughtful responses to my original post, whether they swing left or swing right, suggest (to borrow from the X-Files) that the truth is out there. 

The argument — that with enough exhaustive, high-quality research, we can ultimately determine the validity of policy research — is accurate, but academic.  The problem is that research has a life beyond the peer-reviewed back-and-forth of academic journals and websites.   Most consumers of research findings — policymakers, parents, reporters, me — don’t have the time to conduct their own peer-reviewed studies of recent research.  Even within the field, few educators truly have the ability or inclination to thoroughly examine the claims of each study out there, and so must rely largely on the conclusions of others.

The result is that most of us trust the findings of those whom we trust.   If we’re teachers, we probably trust the findings of the AFT.  If we’re free-marketeers, our first stop is probably Jay Greene or the Cato Institute.  It’s one of those peculiarities of the field — unlike, say, that of medicine, in which research findings are less likely to be attached to ideology.  Education is simply too complex and interesting to yield so easily to the scientific method.

The bad news?  We actually know far less about the effectiveness of various ed reform approaches than we think we know.  The good news?  Lots of jobs for education wonks.

Hello HeadFirsters.  The post below introduces EPIC — the Education and the Public Interest Center.

Two of the primary goals we entrust to schools — public as well as private — are (1) to develop the potential of our individual children, and (2) to develop, across the student body, a set of skills, knowledge, and civic engagement that will help sustain our democracy.

While in no way minimizing the importance of the first goal, the primary focus of EPIC is expressly signaled by our name: our research and services are grounded in the belief that schools, particularly public schools, should serve the public interest.

EPIC is housed at the CU Boulder School of Education, and like the school we have a shared commitment to evidence-based policy and practice as well as to democracy, diversity and social justice. Each of these terms can have different meanings to different people. To us, they mean that we ask the following questions about school policy:

(1) Beyond what we hope to find, what does the empirical research actually tell us? and

(2) What in particular are the empirical effects of the policy on the public-interest role of schools?

If a policy, for example, stratifies opportunities by race, family wealth, or special education or English-language status, then we would probably raise those concerns — even if the policy has other positive attributes.

EPIC, working in partnership with a similar policy center at Arizona State University (called the “Education Policy Research Unit”), created the Think Tank Review Project, which has been discussed a couple times on this blog. The Project provides the public, policy makers, and the press with timely, academically sound reviews of selected think-tank publications. These reports are written by independent scholars across the nation. We have also annually bestowed the Bunkum Awards to think tank reports judged to have most egregiously undermined informed discussion and sound policy making.

In addition, we offer policy briefs and other publications, and we founded the Education Policy Alliance, a nationwide network of (currently 36) university-based research centers and organizations.

EPIC (again, jointly with the ASU center) also has one hundred “Fellows�” — scholars from across the U.S., plus Canada and the U.K. — a group that includes some of the most accomplished people in the field of education policy. These Fellows serve on our Think Tank Review Panel as well as our Editorial Board, which provides peer review of our publications.

More than anything else, the work done at EPIC is intended to contribute to a vigorous public dialogue. For that reason, I welcome comments and critiques. With a nod toward the “Firing Line�” approach taken by the late William F. Buckley, I enjoy reading this blog more when entries are substantive and issue-focused, striving toward a serious and respectful debate even when disagreements are strong. Each contributor arrives with a different set of priors — different beliefs and values that ground one’s inquiries, in much the same way that the EPIC’s own research is grounded in the ideas set forth above.

Please do visit us at http://epicpolicy.org. The site was recently revamped and now offers timely summaries of news and journal-articles, plus our many publications, think tank reviews, and daily raffle drawings to win a Caribbean cruise (I may have made that last part up).

Last week�s EdWeek features a guest commentary by CU School of Ed professor (and sometimes HeadFirst blog contributor) Kevin Welner.  Co-written with Alex Molnar, an Arizona State ed policy prof who, like Welner, maintains a dependably anti-market position, the piece reviews the policy work of education think tanks such as the Friedman Foundation and the Cato Institute.  Their review of 18 think tanks concludes that these organizations are heavily biased on the side of market solutions:

� [A] phalanx of �one-armed� policy analysts are plying their trade in free-market-oriented think tanks. For these analysts and their think-tank sponsors, privatization is the preordained solution for each new educational problem. Indeed, time spent reading their reports leaves the unmistakable impression that the public nature of public education is the root problem for all that ails schools. Everything else is just a symptom.

Welner and Molnar�s analysis sheds light on one of the eternal elephants in the education policy research room:  the conclusions of such research are highly correlated with the researchers� ideologies and previous research findings.  (Put less delicately, this means that much education policy research is  biased).  The trend is most obvious in politically charged areas like vouchers and charter schools.

Though Welner and Molnar identify a market-oriented bias to the think-tank research they survey, they would do well to extend their investigations to research organizations of all political persuasions.  I�m willing to bet the work of, say, the American Federation of Teachers might be just a wee bit �one-armed�, too.   

There are times when reading through the latest news on education reform feels like turning on CNN to hear Nancy Grace talk about her latest breastfeeding experience.

Not that there�s anything important to talk about given that the economy is diving headfirst into the crapper and we�re still quite happy to parrot phrases like �college isn�t for everyone,� and rave on about our American �communication skills.� The hell with actual skills that people might pay you for — i.e. 18 percent of high schoolers take advanced classes in physics, chemistry or biology and plenty of people don�t think they should have to.

I mention this only because I finally went through some of the newsletters in my inbox. (A few more of my wiseass blogs and I will probably be permanently blackballed from all mailing lists.)

Notable headlines: Teachers College Record (The voice of scholarship in education): �Psychological type and asynchronous written dialogue in adult learning.� I�m not kidding.

Edweek.org (1/28)

Explore Quality Counts: Tapping Into Teaching, � I tried to discover why quality counts in teaching, but was directed to a page called �digital directions: trends and advice for K-12 Technology Leaders�. The lead article �e-mail overload� was dead. Hmm. Says something I suppose.

And here in the state, apparently legislators are keen on spending taxpayer dollars concocting bills that allow students to opt out of dissecting frogs, chickens, or something like that.

My point? The news is out there and somewhere it�s being reported. There are cool things happening. But Nancy Grace�s breastfeeding is fogging it all up. 

Some of you might find this article by Andy Rotherham interesting. It�s in this month�s Kappan, and focuses on the relationship between journalism and educational research. 

The article does an excellent job of outlining some of the fundamental differences between the lenses of journalism and research.   It is great reminder of the need to work with journalists and researchers about realities of policymaking, so that policy becomes better informed.  

I was a bit surprised that Andy did not do more to raise legitimate concerns about the overall quality of education research, which really does make it difficult for journalists or anyone else to make informed decisions based on evidence. 

Education research is like 19^th century medicine —  better than 18^th century medicine, but still a long way from an MRI.  At this rate, I�m not sure we educators will ever discover the equivalent of a virus.  

I received the following tonight from Kevin Welner, an associate professor of education at the University of Colorado in Boulder, in response to Uncle Charley’s post earlier today: 

I would like to thank Uncle Charley for promoting (in his own special way) the Think Tank Review Project, which I co-direct. The main purpose of Charley�s blog entry seems to have been to praise a critique written by Dick Carpenter. Interestingly, back in May of 2006, we commissioned a review of a report (setting forth a �typology of charter schools�) that Mr. Carpenter wrote for the Fordham Institute. Small world.

Our review of Mr. Carpenter�s earlier report is available here and Mr. Carpenter�s report itself is available here

Our reviewer (Gary Miron) concluded as follows, �This review identifies and explains a number of limitations in the study in terms of design, methods, and sources of data. While a substantial and commendable effort was put into developing the typology and placing schools within it, less effort and rigor were used in analyzing the differences between the school types. Because no comparisons are made with traditional public schools, the study provides little relevant information that could inform policy in the area. The study is criticized for including false claims to the effect that it is the first charter-school typology and the first to compare different types of charter schools on standardized tests performance. Finally, the review identifies major weaknesses in the study�s reporting of statistical findings. Because the use of typologies can further our understanding of charter schools, it is recommended that the author as well as the sponsor of the study further develop and expand the analyses based on this and other possible typologies.�

I encourage any readers interested in such things to take a look at this and other reviews.  In fact, we issued a review today on the issue of public versus private school achievement. Please  read the review here; I think even Uncle Charley would find it to be informative and fair.

Btw, two implications in Uncle Charley�s note are incorrect.

1. Although most of our reviews do focus on reports issued by free-market think tanks (which issue the vast majority of education policy reports), we have also reviewed reports from places like RAND and the Center on Education Policy. And we have reviewed reports where I tend to agree with the reports� conclusions.

2. More insulting is Uncle Charley�s implication that the indirect source of most funding for this project (the NEA) has controlled or influenced the work of the scholars I�ve asked to write these reviews. I hope it should go without saying that our scholars have complete control over the content of their reviews. These are people at the top of their field; even if NEA wanted to interfere in their work (and they never have), the scholars would never put up with that (nor would I).

Over just two years, the Think Tank Review Project has gained a wonderful following. Policy makers, reporters and others have quickly come to recognize the value of having independent experts review think tank reports. As I�ve said repeatedly, these reviews are not offered as the last word. The reviewers are not put forward as a truth squad or final arbiters. Nor is our goal to stop think tanks� contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate.

Education reformers of a free market persuasion, with whom I share sympathies, sometimes could benefit from more scrupulous reviews of their research. Over at the NEA-funded Think Tank Review Project, selective critiques of free market-oriented education publications are made with varying degrees of validity and credibility. The purpose of this post is not to dwell on this particular issue.

Instead, I want to point reformers to a great example of self-policing by Dr. Dick Carpenter, director of strategic research for the Institute for Justice, a national �libertarian public interest law firm.� Carpenter, also a Colorado native, recently issued a comprehensive and well-documented review (titled Fatally Flawed) of a report published by the Wisconsin Policy Research Institute (WPRI), a free market think tank.

The impetus for the response came from a spate of articles and press releases that grossly distorted the WPRI report�s conclusions: �that public school choice and parental involvement have little effect on student performance in Milwaukee Public Schools.� From the Kansas City Star to the Utah Democratic Party, many hailed the findings as a condemnation of Milwaukee�s 16-year-old private school choice program.

But Carpenter further found serious problems in the report, including a sampling error with national data, unfounded cause-and-effect conclusions, improper creation of variables, a key subjective definition, and other sweeping assumptions. The author sums up:

The fatal flaws in the WPRI report not only undermine its conclusions about public school choice, they also make it impossible to draw any broader conclusions about school choice programs involving private schools, such as the Milwaukee Parental Choice Program. The poor quality of the WPRI report, and the unfortunate way it has been used in the media and by opponents, is a case study in the need for sound research to inform public policy debates.

It seems unlikely that a group like the Think Tank Review Project would have any interest in critiquing a report that already tends to support the predisposition of its union backers. Even so, reformers of a free market stripe (and others, as well) would benefit greatly from more internal peer review � whether the conclusions of their work confirms or undermines their reform philosophy and goals. (Another good example is provided by the careful standards used in the research produced by the University of Arkansas Department of Education Reform.)

Just a word to the wise: Carpenter�s piece is a needed reminder for all of us.