Rejecting "Evidence-Based Practice"

There are many words and phrases I'd love to toss out of our educational vocabulary. "Rigor" is, of course, a big one. "Flipped Classroom" is another - unless we're going to really try the old Fred Astaire-Royal Wedding thing. "Special Needs," "Special Education," and "SpEd" are others because it is way beyond time to embrace Universal Design for Learning, full inclusion, IEPs for everyone, and Toolbelt Theory for all. But here I am with one more... the dreaded "Evidence Based Practice."

"Evidence Based Practice" sounds so good, sounds so authoritative, sounds so "medical." It is, after all, medical. The term was "Evidence Based Medicine," and like the "Gold Standard" research promoted by the AERA and the US Department of Education, it sounds great to the uneducated, but remains not just impossible to transfer to the field of education, but undesirable.

I heard a medical researcher on NPR this week describing how a highly effective counter-treatment to the side effects of antibiotics couldn't really spread because it could not be evaluated the FDA "gold standard" way, and it reminded me of how evaluation systems stymie innovation everywhere.

But even if these research and practice paradigms worked fully in medicine... we still could not, and should not, import them. First, it is not possible to do a "double blind" trial in education, the practitioner surely knows what "treatment" is being used, and so do most students (we often might as well give out "Control Group" T-shirts) so the "Gold Standard" promoted by AERAis simply fraudulent. Then there's the undesirable - we are educators, it is malpractice for us to withhold interventions from students when we believe they will help.

And so if the methods of collecting our "evidence" are both fraudulent and irresponsible... where does that leave "Evidence Based Practice"?

It leaves it as the tool of those who want to preserve - to harden - the industrial-process nature of education, and as the tool of hucksters with products to sell, like researcher-salesmen Slavin and Marzano. As Peter Smagorinsky says, in terms of how the US Department of Education operates:

 "Consider the ways universities operate. Universities initially compete for students, but once they’re enrolled, that game is over. After enrollments are secured, universities compete for prestige, and the game is not zero-sum: The more prestigious universities, the better the nation is served. The U.S. university system is not centrally organized, so each campus competes in its own way. Public universities are bound by state regents systems, at times intrusively; but for the most part are entrusted to be the institution they want to be. Innovations are very public and available as models for others to consider. Indeed, the publishing imperative forces good work out into the open and contributes to the status of the institution as others seek to adopt it. This approach works well for universities, but public school teachers are not entrusted with the same opportunity.

"The competition and innovation that local control would promote would enable teachers to exercise judgment, an opportunity that is rarely available under the current testing regime. Just as importantly, it would help to shape the incentive system within which education takes place. Presently, young people looking for a field of endeavor read daily attacks on teachers and schools, and see a punitive, assembly-line approach to teaching that follows from standardized curriculum, instruction, and assessment imposed by Washington bureaucrats with no teaching experience. Replace that image with a workplace where teachers’ ideas matter and their innovations are appreciated as dynamic and worthy of admiration. Which environment would you decide to enter if you were a talented young person looking for a rewarding career? Most teachers I know say that they want to teach because they want to make a difference. It’s hard to make a difference when the system rewards those who make everything the same." - Peter Smagorinsky, Distinguished Research Professor of English Education at The University of Georgia and included in Valerie Strauss' Washington Post blog The Answer Sheet, March 28, 2012

I am not against "evidence," we all collect "evidence" to support our actions or to suggest change in our actions every day, but I am also for innovation and creation and experimentation every day, especially in schools. For example, I have never repeated the same "class" - not even when teaching the same thing three times in a row. I learn from each presentation and then adapt, that's my evidence based practice, you know, "well, that didn't work," but there is something else creating the change - the "audience," the interactions change every time - and I can not let "evidence" from a previous experience interfere with the need to innovate, improvise, based on the needs currently before me.

This is the problem with "Evidence Based Practice," with "Common Core," with any enforced "National Standards," all limit our possibilities by enforcing "what is" as Gary Stager noted on Twitter: "All capitulations to national standards reduce your ability to make decisions as a teacher." All capitulations to the "scientific evidence" requirement prevent innovation.

What we need instead in education is a nation, nations, of experimental researchers working constantly with Grounded Theory based Action Research. All teachers as innovative experimenters, all schools as laboratories of what might be possible. This is essential, because otherwise we are, to use the metaphor for this week, simply re-arranging the deck chairs on the Titanic because we saw them arranged differently on the Central America.

In other words, our current praxis, our commitment to "Evidence Based Research," is utilizing fraudulent research paradigms to investigate micro-changes in a system developed not through science or research at all, but as a Social Darwinistic effort to prevent social change which might threaten power structures. [See the Five Part series - Designed to Fail.]

Grounded Action

"If a grounded theory is one that, as Strauss (1990) contends, “…is inductively derived from the study of the phenomenon it represents," then grounded action could be described as “…one that is inductively derived from the study of the phenomenon it represents for the purpose of creating and applying practical solutions to social systems” (Simmons & Gregory, 2003)."

Putting together Grounded Theory [link is to a pdf], the observation-based data collection concept which accepts that the real world is more powerful than the laboratory for social structures,  with Action Research, the continually adaptive method of applying developing knowledge to the solution of problems in addition to the study of them, allows us to form a circle which moves us - and more importantly - our students forward. Forward into a world wholly unrelated to the Henry Barnard - Ellwood Cubberley structure of education which remains intact even when "flipped" or "flattened" or privatized.

"Operational theory is systematically generated from and grounded in explanatory theory. The operational theory provides a grounded theoretical foothold for action planning and implementation."

You learn to see your classroom anew. You observe all the possible options. That becomes your data floor. You make changes, observing the results, constantly observing, recording, and adapting to the inflow of information. You are educators. You are not going to trap a control group of children into a failed reading system for 5 months in order to prove that system works - you are going to keep searching for solutions for those kids - but you are going to do it like researchers... recording your observations, your changing efforts, and your successes and failures, and then sharing that with the world of education - wholly and freely.

And if we all work on invention, and we all share what we're doing, and we're all building on what we're doing, we can stop buying third rate adaptations and begin to grasp the possible future.

Which has to be our goal. To move rapidly into the future, where our students already reside. And we're not going to get there buying ten-year-old crap, that we collected "evidence" for five-years ago.

- Ira Socol

The art of seeing

part two    part three     afterthought

How do you see a classroom? What do you see when you look at a “learning space”? What questions do you ask when you watch someone teach? Or watch someone learn?

I sat in a conference session recently and listened to Matthew Kuhn explain how a “Marzano-Approved,” “Bill Gates-Loved,” “Arne Duncan-Endorsed,” iPad app from McRel could allow you to understand pedagogy in your school while “observing” for “three to five minutes.” Well, not actually observing, more like staring down at your high-gloss iPad screen while sitting in the back of a classroom. (for a look at Marzano's research regarding products he is paid to research, see Jon Becker)

Where's this on Marzano's checklist?

Mr. Kuhn promised “accuracy,” “reliability,” “validity,” and many other buzzwords of America’s corporate elite – from Jeb “You can buy my brothers school stuff” Bush to Joel “Give me your kids Money” Klein, from Salman “Learn via my home lectures!” Kahn to Bill “I copied all my major projects” Gates, from Michelle “Tape their mouths shut” Rhee to Arne “I hate schools, teachers, and students” Duncan. He also promised his that his company’s checklist for teacher evaluation was “non-evaluative.” And it is.

It is every bit as non-evaluative as the above paragraph.

There is a deep misconception which runs through the world of our educational policy “leaders.” It is a misconception based in enlightenment philosophy and the scientific management systems which grew up during the second industrial revolution. It is a very dangerous disbelief based in a sad underestimation of the human species and a specifically trained inability to actually see.

This misconception is the belief that human behavior and experience can be reduced to a numerical scale.

"We can't measure engagement" - Matthew Kuhn

It is the belief that you can take me and Michelle Rhee and add us together, then divide by two, and come up with an “average” educational change advocate.

It is the belief that you end up with an “objective truth” when you take a specifically chosen set of questions and determine that there will be a correct answer to those questions.

“You watch the video of this teacher,” Kuhn told us, “then discuss what you see at your tables, and then I’ll tell you the right answer.”

Oh my.

It will come as a shock to Mr. Gates, Mr. Duncan, Mr. Khan, The Wall Street Journal, The New York Times, the Washington Post, and others, but I’m here to tell you that quantitative analysis of your school will always tell you what the “researcher” wants to hear. And that analysis has no more validity than if you let anyone wander your building and report on your school. You will have a minimal snapshot, a story framed by the personal beliefs, experiences, and grudges of the person who chose the questions and the person who decided how, when, and where, to collect the “data.”

Imagine: I am at a gathering and I ask everyone, “how tall are you?” “how much do you weigh?” “can you explain Einstein’s theory of relativity?” And I ask these questions at the start of the event. Or, if I ask – after everyone has had three drinks, “who’s your favorite poet?” “what island would you most want to live on?” “can you explain how Liverpool beat Milan in the 2005 Champions League Final in Istanbul?”

Would I get the same view of the guests?

The comparison to education is not ridiculous. The questions asked, the checklist provided, the rubric you’ve created, will always limit your ability to observe. And the measurements you report will always limit your imagination and your ability to create and change.

“Where’s the checkbox for “kids drawing on the floor?”’ I asked the conference later. Where’s the checkbox for “student movement”? for “kids had seating choices”? for “students seemed really happy”? As Tomaz Lasic once asked, “Where’s the tickbox for “student smiled for the first time today”?

Those who only see the checklist will never be artists, or creators, or even valued observers

Because if that’s not on your checklist, you will not see it. Surely not in your five minute “power walkthrough.” And if you don’t see those things, and a million more things not on Marzano’s checklists, you will not see your school.

You have to learn the art of deep mapping. You have to learn the art of “ADHD vision.” You have to learn to see a holistic world made of many tiny elements. You have to learn again to see.

“This was hard,” one school administrator said after Pam Moran and I had sent a conference session out to try “ADHD vision” during a 15-minute walk, “I really don’t know how to observe people.”

She is not alone. The ability to observe humanity is taken away from us by a combination of our North American views of privacy and our belief in magical numbers. We think it impolite to look too closely at each other. We distrust our human vision and have been taught to rely on statistical models.

New York Mayor John Lindsay walks unplowed Queens streets

When I was young a huge snowstorm hit New York City. It took hours after the snow began for plows to begin rolling, and the result was a mess that took weeks to clean up. I still remember the city official on the news, “We can’t send the plows out without a report from the weather bureau,” he said. “Weather bureau?” the Sanitation Union leader laughed in response, “all you had to do was look out the window.”

And today, while Gates and Duncan count, they have taught us to not look out that window. No Child Left Behind tells us tons of things which everyone knows are not true, but our policies, our editorials, and our political campaigns keep claiming – in John Lindsay administration terms – that “no storm is coming.” Duncan says that NCLB will show next year that 82% of our schools are failing – which is obviously absurd – and that he will let you escape the failing label if you enrich his friends by buying their reform models. And “everyone” sits back and lets that untruth stand. Duncan says, high-stakes tests should determine teacher salaries while not forcing teachers to “teach to the test,” and gets away with this observable lie. From one end of America to another politicians claim that both “teachers make all the difference” and that “teacher training does not matter” (Teach for America) and fewer news media call anyone on that crazed contradiction than called Donald Rumsfeld on his duality of “Iraq is filled with weapons of mass destruction ready to be used in 15 minutes,” and, “Iraq will be very easy to conquer and control, and we need very few troops.”

On the other hand, none of the numbers we have show the brilliant, creative work being generated by kids in great classrooms. None of the numbers show the teachers struggling everyday to overcome America’s embarrassing cultures of poverty-acceptance and child disregard. None of the numbers show that system-changing adaptations – open classrooms, multiage education, studio-style schools, project-based learning, passion-based learning – lead to kids with more adaptable human skills.

Angels in America, the work of another victim of whole language instruction

Observation, however, shows that the much maligned progressive education of the Baby Boomers and early Gen Xers produced the world's most inventive economy. The group without rigorous testing, the "whole language victims," the "new math victims," produced everything from micro-computers to Google, from effective solar power to Pixar, from Foursquare to Burning Man.

Deep Mapping

So we must stop being blinded by our incredibly limited view of "science." Rather, we must learn to see again, to see widely and complexly. To build our own deep maps of the people, places, and experiences before us. You cannot describe the experience of a middle school English class without knowing what happened in the corridor before class began, or what happened the night before at home. You cannot describe the work coming out of a tenth grade math class without understanding the full experience of students and their parents with mathematics to that point. You cannot tell me anything about reading among seven-year-olds without deeply considering the brain function and homelife of each student. And you cannot tell me about the "performance" of any school if you have not deep-mapped it to include a million data points - most of which cannot be charted or averaged or statistically normed.

Human observation and deep mapping are hard, but hardly impossible. These are skills which we all had before school began, and which we must recapture. We'll start by putting down our checklists... and in the next post, we will start to practice...

- Ira Socol

Measurement and the Overpromise

Measurement has its problems.

Listening to the American "Main Stream Media" speak about education today, or reading most of what passes for educational research, is much like waking up too early on a weekend morning and watching infomercials.

@arneduncan?

There's the "prosperity theology," 'just give two years of your life to Teach for America and God will smile on your CV for all time.' There's "weight loss," 'lose hundreds of pounds of poverty with our KIPP program.' There's "get rich on government money," 'just become a charter school operator!' And then there are dozens of miracle cures for what ails us, the 'Marzano Magic Learning Machine will cure boredom and everybody gets smart.' The 'Slavin Super Reading Pill,' will solve illiteracy. Or the "Harvard Lecture" will make all your students upper class success stories. Or just ridding yourself of unsightly unions will cure all.

And just like those infomercials, anyone with half a functioning brain knows this is all nonsense, and that the peddlers, wearing their wealth (Bill Gates, Eli Broad) or their connections (Michelle Rhee, Wendy Kopp) or their PhDs (Marzano, Slavin, the entire Harvard Education faculty) like the white coats of hair restoration tonic hucksters, are self-serving phonies. And yet...

"Education Reform" has been very
good for Our Miss Broomstick

And yet "we," as a society, are fully suckered in. And the fault lies not just in the work of the evil rich - Gates, Broad, Koch, Bloomberg - but in our very belief in scientific management.

Since the beginnings of the Industrial Revolution "we" (the "western" human) have been obsessed with measurement. We measure "production" and "gross domestic product" and "per capita income" and the height, weight, and intelligence of our children, and we compare these against the measurements of others.

Actually, it goes back a bit further, to the Protestant Reformation, and the belief that humans and communities might "measure up" to the perfections described by Calvin and Luther, or not. And to the idea that wealth (responsibly handled) indicated God's love and approval.

The problem with measurement is that it does three very negative things: (1) It creates false comparisons against a fiction - "the average human" and "the average human experience." The child born in the village in rural Kenya is made to line up against Bill Gate's children, on a scale created by Bill Gates. Thus that child is "not white enough," "not Protestant enough," "does not read enough books," and simply lacks "computer time." (2) It ties us firmly to the past - we can only measure against a known, thus measurement itself binds us to prior function and blocks future dreams. The measurer wants a faster horse, he/she cannot conceive of another method of transport. And (3) measurement limits what a society thinks is important. A local bad businessman measures European tax rates against those in the US and insults me by damning European society (no, I won't come back). He focuses on what Americans consider valuable (not supporting their neighbors and children) rather than what Europeans might (supporting their neighbors and children) and thus comes to the set of decisions which put people like Paul Ryan, John Boehner, Scott Walker, Chris Christie, and Rick Snyder in power, which, in turn, probably terminates the American future.

And of course, "measurers" become "fixers." No one "measures up," an idea which has kept preachers employed for 500 years, or 5,000. So the "fix" is needed. "Oh my, I'm not as rich as my neighbor, I need a fix." "Oh my, my child is not as smart as my neighbor's kid, I need a fix." "Oh my, our children do not interpret alphabetical symbols as quickly as those in Scarsdale (on average), we need a fix."

The "fix" is, of course, joined to the overpromise. With so many "fixes" competing for money and attention (and "attention" leads to money), every "Hooked on Phonics" huckster, every aggressive PhD candidate, every professor with dollar signs in his or her eyes, every politician desperate for a CNN or FoxNews soundbite, claims that "their fix" is "the fix" - God has smiled on them and made them the conduit of the perfect truth.

any difference between these two videos? ("fixers" at work)

And each of these fixes limits, and ties us to the past, and preserves the status quo. Dr. Robert Slavin has never spent a minute considering (publicly) what "reading" means. He just wants kids to spend their entire childhood chasing his grandchildren, so his grandchildren stay on top (those other kids cannot possibly catch up unless Bob's grandchildren stop moving forward, since they've begun so far ahead). Bill Gates has never spent a minute considering (publicly) what "education" means. Rather, he sells a system which lists his kids as the 'best in the world' and measures all children by what's going on in that big house outside Seattle.

build your own Bill Gates house, and your kids can grow up to be like his...

Thus, we as a society don't consider how we might re-imagine reading so it is no longer the elite activity of the Gutenberg/Reformation era. We don't consider how we might re-imagine education so it is no longer the industrial process crafted by 19th century Social Darwinists. We don't consider how we might re-design our social structure to eliminate the poverty which destroys opportunity. Instead we run in circles chasing the fictional... that "average" human which does all the things important in the past pretty well.

Measurement has its problems. Measuring ensures that most will be "left behind." Measuring ensures that destructive competition will rule. Measuring blocks truly new ideas. Measuring rates the status quo as the best we can do.

So maybe, just maybe, we should stop doing it.

- Ira Socol