Posted by Gary Hurd on December 2, 2005 04:12 PM
The mainstream media, and a growing number of academics have “discovered” the threat that the new creationism, AKA intelligent design, poses to science education in the United States.
That’s the good news.
The bad news is that some of these newly minted ‘experts’ will be proffering up their solutions, many of which will be shallow, and some even counterproductive. The recent proposal for high school debates on evo/creato by Michael Balter is an example. During the long and contentious discussion of Balter’s editorial and proposals, a research article by Prof. Steve Verhey was introduced by Balter who claimed it was a vindication of his proposal. A short while later Verhey also joined the discussion. That Verhey’s work did not support Balter is clear, as was stated explicitly by Verhey,
I don’t know what to say about high school evolution education. I don’t think my approach would work there. Perhaps it could work, but it would take too much time. Evolution can’t be avoided in HS biology classes, and creationism/ID can’t be presented as even vaguely valid alternatives, so we are where we are.
Since the paper in question had not been seen in print, we deferred further discussion of its contents. Dr. Verhey has now kindly made the PDF of his paper available to Panda’s Thumb readers. Note also that he has also presented key portions of his raw data as well.
I commend Dr. Verhey’s efforts and transparency which are in the best scientific tradition, and I will insist that any comments by PT readers will also. Dr. Verhey and I have exchanged a number of emails over the last two weeks concerning his paper, and the data which informs his conclusions. These emails (with only trivial edits) form the bulk of the following post. Quite obviously any cogent remarks regarding Dr. Verhey’s paper and the material below will require that one has read and understood the paper. Non-cogent remarks will be simply deleted.
If you’d like to open up a new thread to discuss my BioScience paper, I’d appreciate it if you’d include the following in the initial post.
Thanks and best wishes,
The topic of my paper in the November issue of BioScience, “The effect of engaging prior learning on student attitudes toward creationism and evolution,” BioScience 55 (11): 996-1003), has come up a couple of times here on PT, and there seems to be interest in discussing it. I’ve put a pdf of the paper on my web page (http://www.cwu.edu/~verheys).
The paper is based on a version of Biol 110, “Basic Biology” that I have taught several times. The paper describes the third time I taught this particular version, in the Fall of 2003. Each time I have taught the class I have collected data like those in the paper; the results have always been similar.
In the fall of 2002 I had my classes read “Icons of Evolution” and “The Blind Watchmaker for the first time. Since Ellensburg is just 1.5 hours east of Seattle, home of the Discovery Institute, that first time I also invited Jonathan Wells to speak to my class and to give a special university-wide seminar. He was accompanied by a handler from the PR department at DI, who passed out DVDs. Needless to say, I couldn’t use the data I collected from that first class in the paper, since it was an unusual instantiation. It seemed to me that the students, by the way, saw right through Dr. Wells. My colleagues, on the other hand, having taken to heart the dogma that creationists are not to be debated, were nonplused.
I chose the Wells book for a couple of reasons. At the time, it was new. It also went well with Dawkins, which I had used alone the year before. Dawkins uses big words in a book with small type, while Wells uses small words in a book with large type. I don’t mean this entirely as a slam against Dr. Wells, who I found charming and who I enjoyed meeting. But while Dawkins is unapologetic and appears not to care what his reader thinks of him, Wells’ tone at the beginning of Icons is exceedingly soothing and reasonable. By the end of the class, students completely reversed their opinions of Dawkins and Wells, which I think was very good for their critical thinking skills. I also introduced students to fallacious arguments, and pointed out toward the end of the term that Dawkins uses no fallacious arguments to make his points, while Wells uses them frequently. Finally, since Icons is a book about what’s wrong with biology textbooks, students could compare their textbook and their experience with Wells’ claims.
I haven’t seem much discussion of the BioScience paper on creationist blogs, except FaithFusion (http://www.faithfusion.net/index.php?itemid=108 ), which wrote
“The problem is, the point of these discussions is to slam creationism. No classroom should have an agenda that it is pushing. Creationists don’t want creationism to be taught solo; we want a balanced teaching. This isn’t balanced teaching.”
This is not true: at no time do I “slam creationism,” including now. It’s important for everyone to realize that, within obvious ethical and legal boundaries, I did the best I could to present a balanced approach, and for at least the first half of the term generally kept my views to myself. The data support the notion that my approach was balanced * a few students changed their views to less rationalist ones. In addition, I didn’t just present the Judeo-Christian point of view, although it was hard not to spend most of our non-rationalist time on it, and our non-rationalist time was necessarily limited.
As I explain in the paper, my approach is based on fairly standard pedagogical theory: it is hard to learn things without first connecting them with what we already know. This also helps to explain why it is especially hard to learn things correctly when we have first learned them incorrectly. My own experience with my evolving attitude toward creationism is that my progress was slowed by a hard-core biology undergraduate experience that ignored/dissed creationism. I don’t think I’ve ever been a creationist, but I did have some questions, and not getting them answered in college caused me to avoid dealing with the then-raging creation science controversy as a TA in graduate school. I still feel a little guilty about that. I’ve tried to make up for it by starting an annual Darwin Day tradition here at CWU, and by trying to be the best teacher I can be as often as possible.
The delay in posting your email to Panda’s Thumb has been the result of a discussion and partial reanalysis of the data you provided.
We have a number of concerns regarding the comparability of groups AB and CD, sample size, and the statistical treatment of the student reported assessments. For example, if we were to ignore who changed or how, the intervention group AB did differ greatly from the “control” group CD, but they apparently differed significantly from the onset. And we question the rather strongly stated major result which turns out to be driven statistically by the self reports of 7 undergraduates.
Pretest intervention group is high in creationists, AB v. CD Chi^2 = 3.5
Sig 0 .05
Pretest intervention group is low in evolutionists, AB v. CD Chi^2 = 1.64
So, looking internally at the intervention group self reports we find
AB Change (sign test)
23 17 36 2.12
11 17 36 2.12
4.24 Sig 0.05
However the only sub table that approaches significant is,
YEC to nonYEC* AB Change (sign test)
16 12.5 12.25 0.98
9 12.5 12.25 0.98
1.96 Sig 0.1P0.05
*Verhey’s CL and YE categories combined.
The total number of creationists of one stripe or another (we see problems with some of the categories you used) that were in the intervention group AB were not statistically different before and after.
AB Change Creationists before and after
Chi^2 0.47 not significant
In fact, as you know, the number of “Old Earth/Intelligent Design” creationists actually increased in group AB.
(The following analysis was by Douglas Theobald, University of Colorado, Dept. of Chemistry & Biochemistry)
If we move away from the null-hypothesis testing mentality, we can also do a quick likelihood/model selection analysis. Lets assume that there is a rate (or probability) for spontaneously converting from a YEC/CL type to an evolutionist (or ID-ist). Such a “rate” can be modeled statistically by the probability parameter of a binomial distribution. Then, the maximum likelihood estimate for the rate of the AB class is 7/16 = 0.438, and the rate for the CD class is 1/7 = 0.143 . As I see it, there are four obvious competing models to consider:
(1) AB and CD are different, and they have different rates (0.438 and 0.143, respectively),
(2) AB and CD are actually just random samples from the same group, and there is thus one rate that describes both classes (maximum likelihood combined rate = 8/23 = 0.348),
(3) AB and CD are the same group, and the students convert according to the rate in the AB class (0.438),
(4) AB and CD are the same group, and the students convert according to the rate in the CD class (0.143).
Here’s the likelihood and Akaike information criterion (AIC) table for these four competing models:
model K logL AIC dAIC Bayesian prob(%)
1 2 -2.55 -4.55 0.00 33.0
2 1 -3.57 -4.57 0.02 32.7
3 1 -3.95 -4.95 0.40 27.0
4 1 -6.59 -7.59 3.04 7.2
From a model selection perspective, the highest (most positive) AIC wins, so model 1 is the best. But in model selection theory, only an AIC difference of 2 to 3 or more is considered significant. So, all we can say here is that model 4 is lousy, and that the first three models all explain the data about equally well. There just isn’t enough data to distinguish between them with any confidence. If you are willing to make the Bayesian “leap of faith”, and you consider each of these four models equally likely *a priori*, then, based on this data, the posterior probability that a model is correct is about 30% for each of the first three models. IOW, from this perspective the “teach the controversy” style doesn’t appear to have any detectable effect on how likely a student is to abandon a YEC/CL mentality.
Thanks for what looks like a significant amount of time spent reanalyzing part of my data. I’ve spent a little time trying to understand your comments, and I have a few questions and responses.
1. As I’ve said, I can’t help the fact that AB and CD aren’t perfectly comparable. They are actually more comparable than I had any right to expect, since at least the students were placed in the sections at random and various characteristics of the groups were similar. As I said in my posting on the “Contrarian or just lame” thread, it would not be possible to do this experiment with the same level of randomization here now. There’s also nothing to be done about the small sample size. These issues – particularly the pseudoreplication – were raised by reviewers, and are part of why I clearly state in the last section that the results technically “are not generalizable beyond this case study.”
2. I agree that the high number of creationists in AB / low number of creationists in CD is curious. I address this in the left-hand column on p. 1002 when I suggest that my approach may have made some creationists more comfortable sharing their views. This is an essential point: it is not possible to change the mind of anyone who feels disrespected or defensive.
Of course, the number of creationists in section A is not too different from the number of self-reported creationists in the general US population, so an alternative question might be why there was such a low rate of creationist beliefs in the other sections that were the subject of this paper, or in the other papers cited in the “Alternative explanations” section of the paper. Read on for a possible answer to this question.
3. As I discuss in the “Section D” section on p. 1002, there are good reasons to think that creationists declined to participate in the survey, particularly in that section. This was really striking as I was collecting the data. The surveys had been placed in envelopes and held until after the term was over. As I went through the section D surveys for the first time, I was struck by the fact that at least six of the surveys had been handed in blank along with the completed ones. It was as if the students didn’t even want anyone to know that they weren’t participating, let alone to know what their beliefs were.
The apparent failure of creationists to participate in the surveys, especially in section D, may have led to the disproportionate numbers of creationists in AB vs. CD. I think I address this satisfactorily in the paper.
As I understand it, your reanalysis of my data assumes that there really were different numbers of creationists in AB and CD. I suggest that the missing students were creationists. If this is true, the two pairs of sections would have been (more) similar if all students had participated. I also think we can assume that, if these missing creationists had had their minds changed, they would have been more likely to participate. Let’s say the six students who declined to participate all began as, and remained, creationists. Then the 1/7 = 0.143 that you use in our likelihood/model selection analysis becomes 1/13 = 0.077. I think this might change the results of the analysis.
Regarding section D, my paper is as much about what doesn’t work as it is about what does work.
4. Honest, my goal wasn’t to “convert” creationists – the off-line PT discussion seems to assume that it was, and that that is the only worthy goal. My goal actually was to do as I describe in the paper: to engage the students’ very real prior learning about creationism, to give them information, to help (or to stay out of the way of) their cognitive development, and to let the chips fall where they might. I did expect that this approach would convert creationists, and it did do so, but what I claim in the paper is that my approach “produced more attitude change than the other approaches.” There doesn’t seem to be any disagreement about this among your discussants.
For the purposes of my paper, the direction of attitude change doesn’t matter. What is very clear is that the “traditional approach” typified by sections C and D produced very little change.
5. I’m not sure what you mean by my “rather strongly stated major result,” with which you disagree. Could you clarify this for me? Also, your use of the word “undergraduates” in this section of your comments has a pejorative feel to it.
6. You suggest I might want to revise my PT comment. I assume you mean the one I e-mailed directly to you, but I don’t see anything that might have prompted your suggestion. I don’t even restate any conclusions from the paper, I just give a little more background about the class and my approach. I assume you’ll want to post this exchange between us, which is fine with me.
7. Finally, my approach might resemble “teaching the controversy,” but labeling it as such misstates what I was really doing. I was applying basic educational theory to the issue of creationism/evolution by acknowledging that my students had heard of the issue before and had their own opinions which had value. I was recognizing that my students didn’t arrive in my class cognitively ready to think effectively about such complex issues, and so I helped them toward that state of readiness. And, as much as I could, I allowed students to find their own way to their conclusions, so that they could have a greater sense of ownership of their ideas.
When you get right down to it, my data simply support the notion that basic educational theory works, and that’s what Craig Nelson was responding to in his editorial. Come to think of it, his editorial really deserves discussion. I thought his statement that “Public rejection of sound science is not primarily the result of some facet of popular culture. Rather, it is the predictable result of ill-founded pedagogical choices.” was really insightful.
Thanks again for the careful consideration of my data, and sorry for the long response.