Let Them Roll Logs 

The brouhaha about opening academic peer review to the hoi polloi assumes that peer review is effective in the first place.

peerreview.jpg

Whoever it was that said academic politics are so vicious because the stakes are so small got it all wrong: Academic politics are a doll's tea party compared to the real thing, and they are that way precisely because stakes are so vertiginously high, at least from the perspective of a nontenured academic. I bring this up in connection to an August 24 piece in the New York Times about scholarly experiments in replacing or augmenting peer review with online crowdsourcing.

Old-school peer review works like this: A scholar submits a manuscript to an academic journal. An editor at the journal enlists established authorities in the field to vet it. Usually the process is double-blind; the authorities don't know who the author is and vice versa. The two possible outcomes: the article is accepted for publication, provided the author makes whatever changes the reviewers suggest, or it's rejected as unworthy for publication. Not all academic journals are peer-reviewed, but those that aren't carry far less weight with tenure committees.

The same system generally applies to the publication of academic books, although there are some academic presses that dispense with peer review. The publication of peer-reviewed books and articles is the main criterion that guides tenure committees in deciding whether a junior colleague is worthy of tenure—teaching evaluations are an astronomically distant second. Once published, books are also reviewed in the conventional, nonanonymous sense in academic journals. These reviews, too, exert a lot of influence on tenure committees.

In the eyes of its critics, anonymous peer review sucks because it's slow, undemocratic, and hostile to new and potentially game-changing ideas and arguments. Advocates of the new approach discussed in the Times propose "using the Internet to expose scholarly thinking to the swift collective judgment of a much broader interested audience." It's an interesting and attractive idea, and one rife with potential for unintended consequences.

There's no question that the existing review process is slow: an article can take years to reach publication. But it's not clear, at least not to me, that it really disfavors innovation by empowering unaccountable gatekeepers who select against new and threatening ideas.

That's because academics are fundamentally creatures of consensus.

According to an analytic philosopher friend of mine, none of what I'm about to say about academic life applies to his field. From what he tells me, when analytic philosophers get together at conferences, they show up with their gloves off, and the participant who succeeds in punching the biggest hole in the argument set forth by the participant presenting a paper is regarded as the hero of the occasion, with no hurt feelings incurred on the part of he or she who has been publicly corrected.

This couldn't be further from my experience of the average gathering of humanities and social science professors.

Last summer I attended what will likely be the last academic conference of my fizzled career as a historian of American radio broadcasting. The paper preceding mine was dedicated to the proposition that the popular public radio program This American Life, while superficially "gay-friendly," is actually an artful trap designed to "accommodate and contain gay voices," thus neutralizing their subversive potential and reinforcing "hetero-normative values." Looking around the conference table, I wondered how many present were secretly thinking, as I was, "Really? This American Life isn't gay enough for you?"

After the panel was over, I learned, by cautiously sounding out some other attendees, that I was not in fact alone in finding this to be a perfectly hilarious argument. And yet none of us managed one squeak of demurral in public.

(The first thing I did after leaving the conference was send an e-mail to arch homophobe Ira Glass, who happens to be a friend of mine, explaining to him that he had the blood of Matthew Shepard on his hands. Glass wrote back to me: "This is my favorite thing that's ever been written about the show. Very exciting for the staff, and has caused a lot of conversation. Thanks.")

Anyone who knew me in graduate school can vouch that there was a time when I would have made myself unpopular by asking questions like "Are messers David Rakoff, Dan Savage, and David Sedaris aware that they are lackeys of hetero-normative hegemony?" The fact that I didn't on this occasion demonstrates the extent to which I have internalized the prevailing values of academic culture, which stress politeness over virtually all other considerations. I have, in other words, become a great big wuss.

And of course I too profited from the stifling atmosphere of courtesy in the conference room when I got up to give my paper, which posited that commercialism triumphed on the early American airwaves because most listeners, especially working-class ones, either didn't mind it or thought it was swell. I'm pretty sure I was laying an egg, but the response was perfectly gracious, albeit in a noncommittal sort of way.

It's hard to pick a fight around an academic conference table because every step of the industry's socialization process after admission to graduate school stresses playing well with others. When newly minted PhDs totter off to their first job interviews, they'll be lucky if a single person in the department to which they are applying has read so much as the title page of their dissertations. "On-campus" interviews, during which the candidate is typically entertained by his prospective employers for 48 hours, are simply protracted exercises in mutual butt-sniffing, designed to determine whether the prospective hire will fit harmoniously into the polyamorous marriage that is an academic department.

The field of academic book reviewing is likewise a pillow fight. Few reviewers care to freely speak their minds when they know that an uncensored takedown might do real injury to the reviewed author's prospects for tenure or other advancement—and might piss off the author's friends and allies in the bargain. (Fun facts: Amid the thousands of academic book reviews accessible through Project Muse, a gargantuan database of 393 peer-reviewed journals, only nine books were deemed "poorly researched" by their reviewers. On the other hand, 374 were found to be "magisterial"—pretty much the warmest plaudit in the humanistic vocabulary.)

On the odd occasion when scholars do get out their flick knives, it tends to happen outside of class and out by the bike racks. Again, from personal experience: In spring 2001 I got a call from a New York Times reporter looking for help with an article on "radio and democracy." An acquaintance of mine from graduate school, the reporter didn't initially intend to interview me for the piece—I was not yet a full-fledged PhD. He just wanted to pick my brain as to who the big cheeses in the field were. Disinterested and impartial mooncalf that I was, I rattled off some prominent names, regardless of how I felt about their work. But since I was giving him background on broadcasting history that he wasn't hearing elsewhere, he ended up using me as a source.

When the story ran, it was a classically structured rule-of-three job, with Expert One (yours truly) saying A, Expert Two saying B, and Expert Three delivering C, a healing synthesis of A and B.

The morning that the story ran, I received a remarkable e-mail from my reporter pal, forwarding the e-mail he had just received from Expert Two. It read: "Fine piece. I was struck, though, by your inclusion of the comment by Clifford J. Doerksen. As far as I can tell he has written one 5 page article in an obscure journal based on one radio station. Hardly the stuff for breaking new paradigms. Did you see his book mss to see what sort of evidence he has to back up his rather enormous and extraordinary claims? Do you know how I can contact him to see what sort of evidence he has? If he does have hard evidence of lower class support for commercial radio over any alternatives it would be one of the breakthroughs of the decade in radio and television history. If he doesn't have evidence, you have been hoodwinked and your readers have been woefully misled."

No sooner had I read this e-mail when my phone rang, and I found myself on the receiving end of an onslaught of abuse from Expert Two, who had obtained my number from the reporter, and who repeatedly yelled, "WHERE'S YOUR EVIDENCE? WHERE'S YOUR EVIDENCE?" when he wasn't accusing me of shilling for the National Association of Broadcasters. Adjusting the receiver so the flying spittle hit my shoulder and not my ear, I initially did my best to steer the conversation onto a genteel plane where two educated people of goodwill could disagree, but when my attempts to tell Expert Two how very much I admired his scholarship went nowhere, I got into the spirit of the thing. Twenty minutes later he rang off, but not before warning me that I'd be "hearing from other people about this," which I never did.

Given the essentially medieval nature of academic hierarchy, I wasn't thrilled to have cultivated such a powerful enemy. Friends and family tried to cheer me up by arguing that it augured well for my book's reception that it was already creating controversy even though I'd barely begun writing it. (My wife pointed out that Expert Two's e-mail was actually a glowing blurb in disguise: "The stuff for breaking new paradigms . . . Enormous and extraordinary! . . . One of the breakthroughs of the decade in radio and television history!" To this day, she maintains I never would have finished my manuscript if my conversation with Expert Two hadn't left me with a clear sense of who my target reader was.)

The funny thing is this: When my turn came around to have my book warmly reviewed by the only five people I'm not related to who will ever read it, one of my reviewers—by happenstance the same scholar who finds This American Life insufficiently gay—took pains to refute my contention that it any way negated, threatened or undermined the work of Experts Two and Three. To the contrary, it seems, my shit is "more complementary to this scholarship than it is a refutation."

Getting back to the issue of double-blind peer review: Theoretically the anonymity of the process creates conditions in which an academic referee can vote his or her conscience. In practice I question whether double-blinding really makes much difference, any more than the acquisition of tenure reliably turns your average prudent career academic into a fearless speaker of truth to power after years of careful self-censorship. Doubtless if the obscure journal that published my one peer-reviewed article had consulted Expert Two (it didn't), he would have voted against it, but that still would have left the other two referees, both of whom awarded it two cheers apiece as I recall. By and large I have to agree with Professor Drummond Rennie, deputy editor of the Journal of the American Medical Association, when he asserts that "there seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print." And keep in mind Dr. Rennie is referring here to the scientific precincts of scholarship, where especially robust standards of proof and argumentation supposedly prevail.

So what then might we anticipate from a brave new protocol of scholarly review premised on the idea that anyone with access to the Web is entitled to an opinion? Probably something pretty interesting, and at least as dysfunctional as the mess we have now. One might look to the interesting case of disgraced historian Michael Bellesiles for clues.

In 2000 Knopf published Bellesiles's Arming America: The Origins of a National Gun Culture, a weighty tome dedicated to the thesis that Americans prior to the 20th century didn't own a whole lot of guns and weren't terribly interested in them either. A shot across the bow of the National Rifle Association, the book copped the 2001 Bancroft Prize, the highest honor the historical profession has to offer.

But by 2002, Bellesiles's critics—primarily nonacademics, at first—had debunked his documentation to the extent that his Bancroft Prize was rescinded and he was obliged to resign from his tenured position at Emory University.

Scholarly malfeasance of this kind is probably commonplace, but who really knows? There are practically no mechanisms in place to discourage or detect it. Peer review is not fact-checking. When I was in graduate school, one of my contemporaries wove her dissertation out of whole cloth. She was just about out of the gate when she finally got busted. She'd nailed the best job in her field when a rival student finally caught the ear of an unusually courageous nontenured faculty member, who began tugging at the threads of the fictional monograph.

It's no surprise that the Web played such a pivotal role in the campaign that led to Bellesiles's downfall. (Bellesiles, who has been working as a lowly adjunct teacher since then, is currently wangling for a comeback: his second book, 1877: America's Year of Living Violently, was recently published by the New Press.)

The new way of business espoused by the champions of open review is thus really just a formalization of something already well under way. But look how far Bellesiles had to stick his neck out in order to get his head chopped off. He picked one of the few topics that enough Americans care about to guarantee that somebody would bother to audit his footnotes. Bellesiles's Javert-like hero nemesis was one Clayton Cramer, a graduate student and Second Amendment enthusiast who's claimed that it took him 12 hours of hunting to find one citation that wasn't tainted to some degree.

Most academic writing simply isn't going to generate that kind of heat. Open-source reviewing is unlikely to spawn anything more productive than an orgy of orchestrated logrolling: I would rally my friends to the support of my crappy, inconsequential monograph, then scratch their backs when the time came around.

And academics who dream of reaching out to an engaged audience should probably be careful what they wish for. Populism is rarely pretty, and it's a big, ugly, crazy world out there on the Web, replete with opinionated Holocaust deniers, objectivists, truthers, believers in the Protocols of the Learned Elders of Zion, and dedicated rereaders of None Dare Call It Conspiracy.

Whatever. Let the games begin.   

Tags:

Comments (13)

Showing 1-13 of 13

Add a comment

 
Subscribe to this thread:
Showing 1-13 of 13

Add a comment

More by Cliff Doerksen

  • Listings: Spots for Seafood

    Listings: Spots for Seafood

    From shrimp shacks to supper clubs
    • Apr 14, 2011
  • Into the Friar

    Into the Friar

    An inquisitor poses some tough questions: Where do you get those robes? And can you exercise in them?
    • Dec 23, 2010
  • Stop The Presses!

    Stop The Presses!

    Viva Las Vegas
    • Dec 9, 2010
  • More »

Agenda Teaser

Music
Charles Curtis Univ. of Chicago Logan Center for the Arts
December 09

Tabbed Event Search

Popular Stories

Follow Us

Sign up for newsletters »

 Early Warnings
 Food & Drink
 Reader Recommends
 Reader Events and Offers