The Problem with Hypotheses

In what I would call "an informal study" I have ranked the "atmosphere" in various high school hallways around the area in which I live. Yes, it is a "convenience sample," consisting of schools I happened to be visiting. No, I had no particular method of recording observations. No, I will not provide a long bibliography of sources cited. Yes, I have developed my own way of coding the "data" and my own way of ranking the "school atmosphere."

And yet, my study has one thing I think is essential for validity. I began this study with no hypothesis. I had no idea, no theory, no assumptions in mind when I began visiting these schools. Thus, unlike almost every study of education in the United States I have read in the past five years, there is no "confirmation bias" in my study, because their was nothing to confirm.

In my first semester of graduate school, in my first research methods class, we read a study that Dr. Robert Slavin of Johns Hopkins University had done on the reading project he sells. In fact, almost everyone connected with the study had a very strong financial incentive to make the product, Success for All, look good.

My professors thought this was a great study. It followed "all the rules" in Scientific Research in Education1, the "bible" of "gold standard" educational research created by top AERA professors for the Bush Administration.

But I, and a number of other students, found the study to be, as research, worthless. First, as I noted, it was conducted entirely by people with a financial interest in proving the program's success. Second, it combined dozens of changes in school operations with the introduction of Slavin's product, including things like free food and massive tutoring efforts - how could anyone possibly suggest which part of those changes produced any effect? Third, it refused to look at any "side effects" - what happened to students in other areas when most of the school day became devoted to chanting sentences?

But, the more I thought about it, the biggest problem was the hypothesis. Slavin, of course, set out to "prove" that his product worked. That was his hypothesis. And, shock of shocks! he confirmed his belief.

We constantly teach hypothesis in our schools. It is as much the lifeblood of almost every Science Fair as the fact that parents do the project. But does hypothesis help or hurt our science?

Let's go way back to a very famous example. Galileo hypothesized, via Copernicus, that the planets circled the sun. And, yes, his "experiments" (observations) confirmed this. But when Jesuit researchers looked up they noted that what Galileo was saying could not possibly fit with their observation.

The problem is, of course, that the planets orbit the sun, but not in circles. At Galileo's trial both sides were "right" about celestial mechanics and both sides were quite wrong. Galileo had the idea but not the details, the Jesuits had the details but not the idea. Both had gotten to the wrong place through what we now call "confirmation bias." And confirmation bias is a direct result of our commitment to hypothesis.

Hypothesis should not be first. The first question when we study something needs to be "what is happening?" The problem is - a problem most social science research "leaders" gloss over - is that when someone like Slavin goes out to "prove" something "works," every question he asks, every bit of data he collects, every measure he uses, is, at least in part, designed to prove his hypothesis. Slavin measures short term reading test gains, not for example, student interest in literature. The tests he uses measure components of reading, not gains in subject interest. He does not compare "his" schools to others with extensive tutoring. He uses statistical models which presume that the human experience can be "averaged." It is not that he is lying. He is not. It is not that he falsely manipulates data. He does not. But his study (studies) are fatally flawed from the inception because the intent is not to observe but to confirm.

It would be no different if I ran a study to prove that Success for All was a ridiculous waste of school money. My questions, my measures, my statistical analysis would be selected to confirm that.

Is there another way? 

When people ask my hypothesis I tend to say, "I have no idea."  This didn't go over big on my "practicum" project with the faculty, but I tried hard to stick to it, even when measurement structures were imposed on my by the federal grant. On my dissertation, I began with the question, "how did this system develop?" and then, "why did this system develop?" If I had begun with "America's schools were designed to limit opportunity," I would - for example, have read both Horace Mann and William Shearer in very different ways. But by beginning with non-leading questions, I was free to hear these men as they wished to be heard, not through the results of 50 years or a century later.

So, could you, a researcher, walk into a school without a hypothesis? Could you clear your mind of your years of training and just "see"?

How might that change educational research?

- Ira Socol

Oh, the results of that "informal" study?

My top five "best," "healthiest" environments - corridors seemed under control, kids were polite, there was little or no bullying, kids and adults interacted well, kids seemed to arrive in class after passing times not unduly stressed...
1. Black River Public School (a small, urban, fairly diverse, non-profit charter)
2. Godfrey-Lee Middle School and Lee High School (an impoverished urban school)
3. Holland Christian High School (a small Christian Reformed Church school)
4. Hamilton High School (a large rural/suburban public school)
5. Reeths-Puffer High School (a large suburban/rural public school)
My bottom five (opposite of above)
5. Zeeland West High School (a large suburban/rural public school)
4. North Muskegon High/Middle School (a small wealthy suburban public school)
3. Holland High School (a large urban public school)
2. Zeeland East High School (a large suburban public school)
1. West Ottawa High School (a large suburban/rural public school)

What's important here is not the ratings, informal and, as with all research, highly subjective. What's important is that this reveals that, at least to me, the questions we ask about size, income level, management might be the wrong questions. If I had gone out to compare large to small, urban to suburban, non-public to public, I suspect what I would have "seen" would have changed, and my very hypothetical question would have altered my results.

1. I have called Scientific Research in Education "the most destructive book of the last decade," so, there is some bias there.

How to help students see differently

There is a reason that, when the American Film Institute surveyed people, Atticus Finch from To Kill a Mockingbirdwas chosen the greatest hero in American film history.

For me this heroism is not simply represented by the fact that Atticus is someone "born to do our unpleasant tasks," as Maudie says, or by his unstinting sense of justice, but in something even finer.

Atticus teaches all three children in the story to look at the world in ways unknown to the society which surrounds them. Because of this they are able to see Boo Radley and Tom Robinson and Mr. Cunningham in new ways, and - much more importantly - those children, at least two of them, grew up able to transform our ways of seeing. 

So when we read - or see film versions of - To Kill a Mockingbirdor Breakfast at Tiffany'sor In Cold Blood, we are enmeshed in unique world views enabled by Atticus Finch, or at least by Amasa Coleman Lee, father of Harper Lee, neighbor of the child Truman Capote.

On Sunday evening on Twitter @stardiverr and I were discussing the concept of hypothesis. I tend to think schools, and especially the dreaded Middle School Science Fair, should avoid the "hypothesis" idea entirely.

I think this because hypothesis in schools - as in the social sciences - too often means "guess" or - worse - "desired outcome." We "guess" that 'fertilizing' a bean plant with nicotine will damage it, we "want" our reading program to be effective. The result of both is bad observation - confirmation bias - and bad science.

I said that I thought the most important thing we could do with our students - in 'hard' sciences, in social sciences, in literature, in history, across the board - was to teach them to see differently, to look beyond the obvious. @stardiverr agreed, but said that is a hard thing to do. I agreed with that, absolutely.

Invention comes from new ways of seeing. Most of us can not really imagine pulsed signals moving across a wire, yet those who could "see" that, from Samuel Morse to Nikola Tesla to Bob Taylor and his ARPA colleagues, have fundamentally redefined our world. Most of us can only see gravity when we, or something else, falls, but Albert Einstein saw this radically differently, because he saw time and space differently - Einstein saw so differently that we are still confirming his ideas. Most of us can't see a lawn as Scott Fitzgerald did, "The lawn started at the beach and ran toward the front door for a quarter of a mile, jumping over sun-dials and brick walks and burning gardens--finally when it reached the house drifting up the side in bright vines as though from the momentum of its run. The front was broken by a line of French windows, glowing now with reflected gold, and wide open to the warm windy afternoon, and Tom Buchanan in riding clothes was standing with his legs apart on the front porch," but because he could merge the new art of the motion picture with the older art of writing in his mind Fitzgerald could create a timeless masterpiece about the American dream.

Is there a way to teach this skill? Or, perhaps more to the point, is there a way not to drive this skill out of children? I ask this the latter way because we all know that if we listen to young children, they see the world in amazing ways, they ask amazing questions, they haven't yet been taught the expectations which limit our vision. Young kids might ask if trees speak to each other, which, it turns out, they seem to do. But older kids, facing "science," stop asking those questions because they are beyond the ability of kids to effectively test. Young kids might wonder what it might be like to live under a purple sky or what would happen to a fish in bowl which somehow ended up in earth orbit. But older kids know these as "dumb questions."

Young kids might be marvellously inspired to rethink what they see as they come and go by a book like And To Think That I Saw It On Mulberry Street, but soon enough, their writing is reduced to essays "comparing" and "contrasting," or worse, re-writing Wikipedia articles so they can explain who John Adams was to a teacher who already knows.

Dr. Seuss's first book

Neil Postman and Charles Weingartner in their classic book Teaching As a Subversive Activitysuggested prohibiting teachers from "asking any questions they already know the answers to." (p. 117) That's a good start, it would, as those revolutionary authors suggest, "not only force teachers to perceive learning from the learner's perspective, it would help than to learn how to ask questions that produce knowledge." Yet that's only a start.

No science experiment should ever begin with anything but observation. "I see this and I wonder..." "I hear this and I wonder..." "I smell this, feel this, taste this..." Let us observe, and wonder, and then ask systematic questions which help us get to a hypothesis - an idea far closer to the "end" than the beginning of scientific learning.

No writing assignment should ever begin with anything but, "What questions do you have about this?" or, "What do you imagine about this?" We need to push our children to investigate, to imagine, to dream, to play, to never be satisfied with the canned explanation.

We need to help our students stand on their heads, or look down from a high tree, or to look up from under water, or whatever, as long as we help them to find new ways of seeing. Once they have this new view, then structure can help us climb down into the questions. But before they have this new view, all structure does is prevent people from climbing to new places.

So, we need to create environments in our schools, real, virtual, and academic environments, which allow dreams to evolve and collide, develop and connect.

What we don't need is structures which end up as prisons for the mind.

- Ira Socol