Flawed research: Rockport’s boys-only math classroom

Last November, the Bangor Daily News published an article about Rockport trying out a boys-only classroom for math.

Because this article did not point out the very basic research flaws in Rockport’s effort, I wrote a critique.

Once again, the Bangor Daily News has an article about this which doesn’t address the flawed research design.  I am really, really disappointed in both the school district and the newspaper.

Frankly, it is a disservice to the students in the school district to have carried out research done this way and it is a disservice to the readers of the newspaper to treat this research as credible.

Boys-only classrooms might be helpful for some students. But this research is so badly designed that it cannot tell you if it is.

This research is so flawed it is impossible to know the impact of the endeavor. Therefore the research cannot provide a guide to future educational efforts.

Why? As I explained in November:

“[I]t’s not just politicians making basic errors, nor political reporters overlooking research errors.

The case of the Rockport, Maine third-grade boys-only math classroom is yet another example. Here’s what the Bangor Daily News reported:

The major difference between the all-boys class and the two other third-grade classrooms isn’t the material taught, but the environment. The class can get very noisy. Boys are allowed to get up, move, do their math while lying on the floor if they need to. . . [The room is] full of stuff — computers, a turtle tank, a plant tank, bookshelves, sand trays, sinks and boxes of bones. . .

On this particular morning a group of four boys went into an open room outside the class to look at their “dig site.” The boys already had dug up artifacts from the school sandbox, re-created the dig site inside the school and mapped the artifacts on paper charts. On Friday the boys had to categorize artifacts and count them. The rubble was supposedly from an early Native American site.

This approach is predicated on the view that boys are rowdier than girls. The classroom will be tracked and learning results will be compared to other classroom arrangments to see if outcomes are any better.

But how could anyone know that any improvement is due to the class being boys-only?

Let’s think about this for a minute. There are two differences between this classroom and others:

1. It’s boys-only.

2. It uses a very different teaching approach, one which is very engaging and which requires students to apply their math to many subject areas.

Thus the answer to my question is — they can’t.  It is impossible to know if any improvements are due to the class being boys-only.

Well, why not?

One, you don’t know if different outcomes are due to having a boys-only classroom or having a different model for learning. It could be that the new model for learning would produce great outcomes in a classroom with both boys and girls or for girls only. Better outcomes might have nothing to do with putting boys together.

Two, it’s well established that teachers trying out new curricula tend to be very enthusiastic about trying out something new. That, by itself, typically produces better outcomes — until the new teaching approach becomes the standard.

Third, people who are involved in a new approach that is being studied by others typically perform better, at least for awhile. This very well-known effect is called the Hawthorne effect.

All of these add up to a classic problem when it comes to just trying new curricula or new programs. You have to try them awhile to make sure there aren’t enthusiasm and Hawthorne effects. And if you don’t have the proper research design, you can’t know the impact of the curricula or program.

What’s the right research design for determing the impact of new educational approaches?

One must use an experimental design. This means setting up proper control groups in which each different variable is held constant.

Thus you could have two classrooms with the new, highly engaged curricula; one is boys-only.

Or you could have the old curricula with one class that’s boys-only and the other is not.

But comparing a boys-only classroom with a new teaching/learning approach with a non-boys only classroom using the older teacher/learning style won’t tell you if boys learn better if they are sex-segregated.

To assess effectiveness, the two variables must be untangled or else you know — well, nothing.”

The educational administrators should have designed a better study. The reporter should know the design is badly flawed and should explain that to readers.

Amy Fried

About Amy Fried

Amy Fried loves Maine's sense of community and the wonderful mix of culture and outdoor recreation. She loves politics in three ways: as an analytical political scientist, a devoted political junkie and a citizen who believes politics matters for people's lives. Fried is Professor of Political Science at the University of Maine. Her views do not reflect those of her employer or any group to which she belongs.