 |
| the FB conversation |
In some kind of invented ideality, we'd all be friends. If we could all be friends of Facebook, that'd be one weird realization of the semantic web utopia, or more accurately, heterotopia. About two seconds after that realization, it would begin to fail epically, for the obvious reasons.
Since we all can't be friends on Facebook, what happens? Professionally, for medievalists, and all sorts of other academics, all kinds of impromptu conversations that begin casually often cant in unexpectedly valuable directions. And there they usually stay, in Facebook's not-quite-private-not-really-public liminal state. In this blog and in his "Blogging the Middle Ages" chapter for the vastly entertaining Geoffrey Chaucer Hath a Blog, our own JJC wrestles with exactly what the relationship is between new digital frenemies Facebook and what can now, already, be called the "traditional" blog. Will so much of the new electronic orality once buzzing around in public fora cloister now in semi-private spaces? Certainly, I expect, just as other professional conversations will remain in selective email exchanges, Skype calls, conference sessions and good old face-to-face meetings.
The beauty of digital media, as Henry Jenkins points out, is its urging for convergence. The moment one event happens in ones and zeroes, it has the opportunity to happen in another. A few days ago, I FB’d a link to a New York Times article on new models of digital peer reviewing for scholarship, which discusses the recent Shakespeare Quarterly experiment (also discussed here at ITM), as well as the "Becoming Media" issue of Postmedieval that Jen Boyle and I are editing. A robust conversation quickly developed between Richard Burt, Eileen Joy and Karl Steel, with a bit from myself - and rapidly had little to do with the rest of my FB life, principally concerned as it is right now with whether my wife and I are actually going to be able to buy this amazing house in South Orange, NJ, my second life as a roller derby referee, and the joy that is my two-year old daughter, Hazel.
So I brought the conversation to ITM. Here it is, because it has grown to a point where it demands more voices and a broader public. And this, perhaps, is the hybrid model we can think about more - using private social networks, when warranted, as springboards over to the public blog. It is in some way, no different than the process by which we develop scholarship all the time: the thought that comes to the conversation, the conversation that comes to the writing, the writing which comes to the synthesis, the research, the argument, with all the recursive loops along the way. The topic is also deliciously meta, as the open review and comment of scholarly ideas is exactly what was going on in the FB conversation, and what happens here at ITM all the time. So have at it, friends.
------------
(If you click on the image above you can enlarge it, but many people will have a hard time reading it all the same. I'm therefore posting the exchange below -- JJC)
------------
Martin Foys "The quarterly’s experiment has so far inspired at least one other journal — Postmedieval — to plan a similar trial for next year." - this is the issue Jen Boyle and I are editing; we we're interviewed, but didn't rate a sound byte, I guess ;(
The Internet is calling into question one of academia’s sacred rites: the peer- reviewed journal article.
Martin Foys
Richard Burt An interesting article. I just forwarded a link to a Shakespeare listserv. But I think it is no accident it came out in August (the slowest month of the annual news cycyle) I question the (hyperbolic?) claims that "radical" shifts are happe...ning because of new (digital) media. Robert Darnton suggested putting various drafts of a dissertation online back around 1996 (in NYRB). that never happened. Now we have the reviewing process exposed in a single issue of SQ. But it is still peer reviewed. Rowe asked people in the field to participate in the comments. (I declined, on two occasions, out of disinterest.) But does anyone who reads SQ, accessible only through subscription or libraries, really care about the comments from non-specialists from space flights? Will anyone read this genetic, pre-publication material that usually drops away from publications like booster rockets do? To what end? Won't people just cite the published articles as they have before? More crucially, there seems something fraudulent about the SQ "experiment." I stress "seems" here, but I do also say "fraudulent" have to wonder if the fix was not in before the essays were officially accepted. None were rejected. How likely is that? The revision process of SQ's essays resembles the process of publication--contract with readers' reports, author responds, author turns in a revised mss to the press (which the editor does not actually look at to check to see if any changes have been made). The difference is that SQ now has a kind of dead blogging trail of the revision process of a faux peer-reviewed process. Did the academics who were invited by Rowe to comment really say all that they thought? Or did they censor themselves since their comments could be read by anyone, instead of just by members of the editorial board, as is usually the case? Letters of recommendation without student waivers aren't taken seriously, right?
As for the author, would you really want to have 350 people commenting on your essay as you were revising it? Would you seriously respond to all of the comments? How critical, in every sense of the word, are these comments? Do they really rise to the status of criticism? Might it not be better to proceed with the customary dialogic standards already in place (the author if an article is already engaged in dialogue with other critics)? Can anyone explain to me the value of this experiment? Or why anyone should want to adopt it? Speed seems to be the only stated advantage, but in fact, nothing is sped up in the SQ publication process. The journal will still publish in the same quarterly way it did before. But why can't we go at our own pace? Is there a NEED to go faster when we already use email and can send our written work to people who will give us useful feedback before and after we we submit it? And why would editors want to give non-specialists as much authority as non-specialists? What does the editor do, then? Accept essays by plebiscite? Authors understand themselves to be competing for Academic idol? Martin? Please help me out here.
Richard Burt Here's another concern: “our crowd sourcing.” Is crowding a good thing when it comes to the publication process? Isn't the reason academic books have such small print runs (500 to 1,000 copies) that they do not have mass appeal (however much we wish they did)? Are we not each other's fit audiences, though few? We few, we happy few?
Karl Steel the size of a crowd is relative. For peer review, 5's a crowd. How many eyes typically see an article before it goes to press? The author; the journal's editor; two peer reviewers (maybe two). And that's it. Even if only 10 or so qualified and interested people look at the prepublication material, commenting on the sections they know best, we're in territory of 'crowd.'
Martin Foys @ Richard: early days, remediation, digital incunable, blah blah blah, time still needed for all this to shake out :) BUT - as editor on this, one thing we are excited about is how all the contributors will be more easily plugged into all the other contributions, should they desire. This is something much rarer, and logistically obdurate, in traditional modes of publishing.
Richard Burt Karl, Yes, "the size of a crowd is relative. For peer review, 5's a crowd." But my basic questions stand: Who benefits from the crowd of five or more? The author? if so, why don't editors require even more peer reviewed reports? The rea...der of the published and still peer reviewed publication? I don't see the point of publishing pre-publication material. Unless you want to make a sort of DVD edition with a "making of" extra / appendix. Is there a logic to the current pre-production, production, post-production publication process? I don't see the logic of altering it. The statements about peer-review being bad in the NY Times struck as being entirely fabricated. I have heard of no actual case where peer reviewing per se has been a scandal. I'm just sayin': it ain't broke, so why fix it?
Martin:
You wrote: "This is something much rarer, and logistically obdurate, in traditional modes of publishing." I think the rarity is a good thing. There is no limit to the number of readers an author can send his pre-published mss to (in theory). WHy not let the author determine who reads it before it is ready to be published? We already know how the process works. So why publish what usually gets flushed? I just don't get it. Sorry.
Martin Foys Ah, Richard - I was writing there specifically to the idea of all contributors (not any reader) having easy access to all other essays in an issue, and being able to revise in discourse as the process moves forward. Not necessarily *the* model that one needs to have in all academic publishing, but I do think one worth exploring in these early-ish days of born-digital scholarship
Eileen A. Joy @Richard: there are all sorts of ways in which the traditional anonymous, double-blind peer review process does not work well. It may not have caused any large "scandals," as you call them, but that does not mean that it is not a system in... some need of reform. I also think there is immense value, for all sorts of reasons, in having a more transparent review process, regardless of what eventually gets "flushed" or not. Scholarship IS a process, not just an endless series of supposedly static "products," and I think we should take better account of that.
Karl Steel "if so, why don't editors require even more peer reviewed reports? "
briefly, time: more reader wrangling would lead to even longer review times for submissions, which would be deadly for pre-tenure proffies. Whereas an online prepublicatio...n open comment period could, I think, speed things up compared to the the status quo.
Richard Burt Martin,
Now I get you. Contributors to same issue makes sense. I think that process has already ben informally in place for years, however, when it comes to editing book anthologies (contributors are often asked to read and refer to other e...ssays in the volume as they are relevant).
Eileen,
I'm all for new media publishing possibilities and for reforms that improve the quality of whatever gets published (in codex form and in digital form). But I still don't see what is so bad about the peer review system as presently constituted or why it is need of reform. Can you say more? I agree that publishing is a process, but the "product" is also part of a process: books get forgotten and many are never even read). So everything gets flushed at some point. The bad publication to be used as toilet paper analogy goes back to Ben Jonson (who cribbed it from Martial) and then gets picked up again with the Augustans (Swift and Pope both use it). So much of what gets published even when peer reviewed isn't worth reading. In my experience, pretty much everything gets published somewhere even that means a mss starts at Princeton UP and ends up at Duquenes UP. Adding more comments to the pre-published publication to come seems to be to gather more moss, to shift metaphoric gears, rather than make things more transparent. And what counts as discursive transparency, anyway? Isn't transparency a phantasm, part of the academic imaginary (which of course traverses the academic symbolic)? All I care about is the product and the process of responding to it with another product, and so on. Can you tell me what the value of creating a new intraprocess is and why adding it is a desirable reform rather than more junk? All I care about is the product and the process of responding to it with another product, and so on.
I have no investment in the peer review status quo. As I said above, I just don't see what the problems with it are (the ones mentioned in the NY TImes article seemed totally bogus to me).
Richard Burt Karl,
Why is speed a value when we do not produce (supposedly) assembly line products? Isn't speed a word administrators like? How will things be sped up if the journal is published the same number of times a year with the same number of p...ages? Won't the backlog just get longer?
Martin Foys Karl, Eileen, Richard - any objections to moving this convo wholesale to "In the Middle"? Jeffrey and I would love to break this discussion open to a larger audience . . .
26 comments:
It's great that the NYTimes piece has generated so much interest in the subject of open review, here and elsewhere in the blogosphere. But, as typical with them, it's not particularly nuanced. Jennifer Howard had a piece in the Chronicle of Higher Education about the SQ experiment, and it provides more context and is more thoughtful on the subject. It's behind a paywall, but this link might get you through:
I'm fairly invested in this experiment since, as an Associate Editor at Shakespeare Quarterly, I was part of the group that discussed whether to do it and how to go about it, and as the editor of a forthcoming SQ issue on Shakespeare and Performance, I am anticipating doing something like this again.
It's important that this deliberately was not an experiment with fully open review: that is, it did not open the doors to any and all submissions and any and all commentators; nor did the journal promise to let all decisions about what to publish be determined by the reviewers. SQ decided to do a modified process, combining some of the practices of open review with some of the journal's standard editorial practices: all submissions went through what is usual practice at the journal, including an initial review by the editorial board (and, in this case, primarily by the guest editor) to determine which submissions had a chance of being accepted and which could not make that initial cut. And SQ invited a large number of scholars to respond online to submissions, not because the journal wanted to cut out other commentators out of the process, but to ensure that this issue had the same sort of rigorous screening and dialogue that other SQ issues do. Given the newness of this practice in our field, I think the fear that scholars might not even notice the opportunity to participate, let alone actually put fingers to keyboard, is a very real one. Inviting scholars to participate helped ensure that there were reviewers ready to step into the conversation. Once the commenting period was over, the submitters had the opportunity to do revisions, and those revised pieces were the ones considered for publication. I do not see the fact that all the pieces posted were finally accepted as being fixed or fraudulent. There was no promise of publication, nor any behind-the-scenes maneuvering.
I'm excited that Postmedieval will be doing something similar for their issue on media. The benefits of the process are real. As Martin points out, one of the great things about the process is the ways it invests authors in each others' work. There was a great conversation happening among the submitters in the SQ comments, and that conversation both strengthened the individual articles and allowed the issue as a whole to speak with greater timeliness. That could happen with traditionally peer-reviewed issues, but the ease with which those conversations took place in this issue was definitely due in part to the ease of the medium.
There's been a lot of comments in response to the NYT piece about the matter of peer review and the possibilities of open review. One of the better was the Tenured Radical's post . Kathy Rowe's own thoughts are available pre-print at the Media Commons site we used to host the experiment. And Kathleen Fitzpatrick's work on Planned Obsolescence is the place to go to start thinking about scholarship as process, what digital media has to offer us, and what these things might look like in practice.
I think, myself, that the obvious problem with peer review is that articles are often not reviewed by people who know enough about a subject to say whether an article is good or not. In fact, academic peer review is too often confined to the editors' friends and acquaintances. The editor cannot ask someone he or she has never heard of to review, after all. The effect often looks like an old boy's club to the rejected.
Open review on the internet, then, would have this advantage, that the scholar you need comments from, but whom you don't know exists, may find you. But it is very unlikely to occur in a useful time frame! And meanwhile there will be a lot of unhelpful comments from people who don't know much but, like me, can't resist a comments box.
A way through this might be to allow only subscribers, who are presumably already interested in and at least slightly knowledgeable about their journal's field, to review. But this requires a well-established and large user base, and only really works for periodicals. I suppose some version for monographs might operate by having a 'reviewers' club' or similar, which cut you a reduced subscription to a publisher's whole range of subject catalogue if you reviewed a certain number of pieces each year or something, kind of like if Amazon paid for product reviews with coupons.
Great discussion. To me it seems clear that a hybrid mode is best: some kinds of peer review are better than others (I prefer modes that encourage mutual responsibility, and am not a fan of blind review); peer review can't be replaced by a wiki or unpremeditated crowdsourcing. But as numerous academic blogs have demonstrated, gathering experts to maximize careful feedback leads to conversations that disseminate research in progress more widely (increasing its impact) AND leads to a better published product (journal article, book).
Careful planning is the key: inviting a sufficient number of experts to weigh in, ensuring that the conversation doesn't get led off track by nonproductive or derailing comments, etc.
An even better outcome: open access publication of the final products ...
And one more thing. A friend of mine who is a veteran journal editor is about to publish a piece that notes how difficult it often is to get scholars to agree to peer review each other's work: everyone complains that they do not have the time, and many people observe that since they are not being paid to do it they won't commit their labor. That second excuse seems especially unfortunate, and my friend blames the current system by which junior faculty are shielded from committee and extracurricular work by their colleagues, so that when demands to reach outside their own scholarship arrive they are reluctant to be so community minded: they just don't see it as an obligation, like earlier generations did, and resent not being compensated for it because it is "extra."
I don't think that is quite true (even though I do often hear grumbling about the the unremunerated work scholars are asked to do, and do think many of us could be more generous in the public intellectual role we should be playing) -- BUT I also think it is wrong for the companies that publish journals to charge exorbitant subscription fees, make a huge profit from these fees, and still expect their for-profit endeavors to be underwritten by unpaid scholarly peer review. I have no problem with nonprofit and especially open access publishers not paying for this essential labor, but something rubs me the wrong way about journals being so expensive my library can't subscribe to them only because the CEO of the company that publishes the things likes his money wrapped in bags with dollar signs on them and stacked high in his vault.
the current system by which junior faculty are shielded from committee and extracurricular work by their colleagues
What? Really? Where can I get a gig like that?
More from my Facebook thread . . .
Richard Burt:
Martin (cc'ing Eileen and Karl),
Here a few thoughts that might have some bearing on the special issue of Post-medieval you are co-editing.
1. What would happen if you were to present the rationale in terms of what adding the pre-publication material does on its own terms to the publication rather than as a remedy to a non-existent problem in peer-reviewing as presently practiced / as a radically new way of getting comments (when it's really a variation on already existing electronic commentary practices)? What aspects of peer editing does it do better?
As an analogue, I am thinking of the American Historical Review. When it first went online, the online edition had a number of features the print edition didn't. Online articles were linked in the endnotes. This was a great idea that never caught on since we still have to use online notes to get other online articles the way we do footnotes of articles printed on paper. The lead article article was written by Robert Darnton. His essay linked to several others he had published as well as to an interview with him. He had maps and some other stuff online. Historians seem very open to printing two versions of the journal. Every other journal I can think of makes the online and paper editions the same (in terms of content). But AHR was and is concerned with publication, not peer reviewing or pre-publication material.
2. You called your special issue of Post-Medieval an "experiment," so I'm wondering what results you are expecting from the experiment. What decision or decisions about how peer editing works at Post-Medieval will follow from your experiment?
3. If nothing follows from your experiment (and certainly the special issue of SQ will change nothing about peer-editing at SQ), what is the point of performing it? Restated: If it is just a one time experiment that will leave things at P-M as they are, what's the point of performing it? If what you are doing as an experiment in your special is clearly better, why not just change the peer-reviewing process at Post-Medieval now, especially if it is a remedy to an existing yet widely unrecognized problem?
4. Finally, if P-M wants to do something radical, why not make it an open access journal? Your experiment will be performed, as far as I can tell, under the same constraints in place at all academic peer reviewed journals and all essays in the special issue you're doing will be peer-reviewed (as usual). So nothing really changes, no? Or does it?
Martin Foys:
Hey Richard - thanks for those thoughts. A few things:
Where do I call our P-M issue an "experiment"? I have no hypothesized expectations as I might in an experiment. Only curiosity and a desire to explore.
I get the feeling you feel I'm charging up a hill waving some kind of battle standard trying to change academe. I'm not. It wasn't even my original idea to provide colleagues access to submitted essays for the issue - it was my co-editor Jen Boyle's.
That said, I'm certainly not opposed to it - new technologies occasion new, untested methodologies, and I'm not adverse to seeing what happens when we implement them, since it is such relatively untested waters - if only (but not only, really) to let all contributors participate in a more immediate feedback loop with each other as they revise drafts, as I already noted in an earlier post - might be useful, might not.
And now, I'd like to bring your questions and my answer over to the ITM blog, as others might have better answers for you - do you mind?
Norman Hinton:
My feeling about peer review and about the absence of peer review are so complicated that I don't know HOW much room it would
take to detail them.
Richard Burt:
"And now, I'd like to bring your questions and my answer over to the ITM blog, as others might have better answers for you - do you mind?
Not at all, Martin. do ahead, by all means. Thx.
I can see the merits of expanding peer review to conclude contributors to an already peer-reviewed journal. I just don't understand why others see this process as not being within the purview of peer reviewed publishing.
I guess I hallucinated "experiment." Sorry.
Since Martin and I are editing this so-called experiment, I wanted to chip a few cents.
In part, I agree with the tenor of Richard's more general critique of all this: appeals to "radical" innovations and progressive dreams should draw skepticism.
That said, I am not sure that such utopianism exhausts all that is in play here. Two follow-up points, one on a practical/professional note (which, interestingly, seems to be where many of these exchanges come to rest); and another less practical observation.
"Open" peer review does not necessarily imply that everyone (upwards of 350 plus) can or will comment on material. The SQ instance is an example of a process where reviewers were asked to declare their interest and expertise before commenting. In other words, many open peer reviews function less like a crowd than a dinner party. The former implies shouting from the blind, the latter an event of interfacing that at least points to the possibility of taking “to” and “with” not “at.” While I don’t necessarily think that open peer review is more transparent (in the many ways we can imagine this term), it does create some opportunities for thinking about the “product” and the “process” in ways that we are not typically asked to consider.
Many of us are re-imagining ways/forums/mediations that allow some of the theoretical in the performative, and the performative in the theoretical, to flow through and with our work and production a little more freely. Open peer review is, perhaps, another possible event-space. Speculative realism, OOO, animal studies, “thingness” revisited, media/mediation/ posthumography are all ventures that insist on re-imagining the affordances of object-environments-affects, of re-imagining what we mean we speak of the form in formulation, of the surfaces that augment the “thing itself,” and yet we often retract into very comfortable articulations of practice and product when we get down to the business of the tools we USE.
Posthumography in the object list above refers to a special issue of Rhizomes , “Boundaries of Publication: Posthumography < http://www.rhizomes.net/issue20/ >. This is a tremendous issue – and Richard Burt has a fabulous essay in this collection on desks, files, machines, and death – and worth reading in full (Don’t flush). I mention this issue because I think that many of the points Richard makes in that article would enhance this conversation. How would we imagine the paratext of peer review? Is peer review its own kind of homage to the posthumous? While open peer review may not be more transparent, it is a paratext; it is a new/old kind of ghosting the text; it is a graph and trace that seems to me to deserve a try.
Returning to a practical matter, the above issue was obviously thought through, and the contributions all speak to one another beautifully. I can’t help imagining, however, another surface to the images, texts, and performances of that collection. I like the idea of Benjamin’s trunk, Derrida’s papers, Goethe’s desk, and the apparatus of the publications publication all hovering together as paratextual morass.
Re Richard re Eileen, the publication as pooblication topos predates Martial, actually. Catullus, in the first and last lines of carmen 36, calls the poetic output of voluminus Volusius "cacata carta", shitty sheets.
Michele Kennerly
I've been avoiding responding to any of these discussions, because I'm a graduate student with limited experience of the "traditional" peer review process (and then only as an author, not as an editor or reviewer); but, for what it's worth:
I participated in the SQ "experiment" (the word does deserve quotes, I think) as the author of a review of digital resources, which means my essay was not peer-reviewed so much as peer-commented. Although the stakes were low for me, I didn't think twice about participating: I already have a public academic persona that blogs and microblogs half-baked research on a daily, almost hourly basis. In fact, I posted scans of my notes for what will become my submission to the Postmedieval "becoming-media" issue over three months ago, and all my ongoing notes for the project are available on my public wiki. I regularly solicit and receive input on my reading lists, arguments, even prose style through twitter. Frankly, whatever the Postmedieval "open review" process throws at me will probably be less rigorous than what I've already put myself through.
Which is, of course, exactly the problem. After the rich, mutually fulfilling exchanges I've had online, the traditional journal article is freeze-dried fruit. It's why I keep falling back on self-publishing digital projects or "creative criticism" published as art, rather than squeezing my research into a form to which its methodology is antithetical. Until we start talking about changing what we accept as "essay" and "argument" to meet our current media ecology -- i.e., *making* the dreams of Benjamin's trunk a reality -- structural amendments to the publishing process end up building libraries and writing loan policies for books that don't exist. The conversation will continue to flop lop-sidedly until we start talking more about, yes, what Darnton imagined twenty years ago (what Mallarmé imagined 120 years ago, what Georg Philipp Harsdörffer imagined 380 years ago) -- the book-to-come, digital Goethe's desks.
Thanks to everyone for such a spirited conversation here re: the Shakespeare Quarterly experiment. I want to respond to some of Richard Burt's questions but I also find them, to be honest, somehow "off" in tone. I can't really respond to questions that are framed with supposedly factual *statements* that are in no way factual at all, and are actually the opinions of Richard. I say this to Richard with the utmost respect for his scholarship, which I have long admired. So let me start with these assumptions, before I even get to the questions, which are premised on *assumptions* that may or may not prove to be true in the long run, and in any case, they deserve further rumination and debate, otherwise we're just foreclosing this whole conversation to begin with, and why would we do that? And my thanks to Jen Boyle as well for contributing comments here, since SHE is the one who initially conceptualized and will be co-editing with Martin Foys the forthcoming special issue of "postmedieval" on "Becoming Media" [Vol. 3, Issue 1: March 2012]--and that's "postmedieval" with a small "p," by the way--and I hope everyone will keep in mind that what Jen has in mind for this issue will go beyond the SQ "experiment" because she has some very novel ideas about the presentation of the issue that are not about the peer review aspect at all [some of her comments here cover that]:
1. RB mentions the idea, put forward in 1996, of drafts of dissertations being put online, and then he says, "that never happened." NOT TRUE. Here at In The Middle, for example, Karl Steel shared more than several drafts of parts of his dissertation, and most importantly, his Conclusion. His brilliant book, forthcoming from Ohio State UP ["How to Make a Human: Animals and Violence in the Middle Ages"], graciously acknowledges the feedback he received along the way from blog readers and commenters. Karl can explain better than me how this process helped him, but I know intuitively that it did. There are countless examples of PhD students sharing portions of their dissertations via weblogs and the like and getting important feedback, from experts *and* from other student-peers, and I won't waste time providing more examples. Whether or not this process has "official" oversight from students' diss. committees is, frankly [in my opinion], beside the point.
2. Regarding the NY Times article on the Shakespeare Quarterly's partial/hybrid open peer review, RB wrote,
"The statements about peer-review being bad in the NY Times struck as being entirely fabricated. I have heard of no actual case where peer reviewing per se has been a scandal. I'm just sayin': it ain't broke, so why fix it?"
Later on, in a further comment, RB notes the "non-existent problem in peer-reviewing as presently practiced."
[to be continued]
[continuing]
I'm not sure where to begin here, but I'll try. Has peer review ever caused "scandals"? I honestly don't know [chime in if you do!], but is it a system that has no problems or capacity for error and abuse? My god, NO. Perhaps RB has been immeasurably blessed by the double-blind system as it stands now, but there are so many stories to relate about all of the ways in which it can fail miserably that I practically don't where to begin. For the reasons both Jonathan and JJC pointed out here, editors often have to scramble to find good so-called "expert" reviewers, and sometimes you really get the expert you need, and sometimes you don't. There is also the question of hostility toward certain methodological approaches, regardless of the ultimate quality of the piece in question. I have a friend, who shall remain nameless, who works in a sub-specialty of early medieval history that is so SMALL, expert-wise, that he has a total of about 3-4 readers for everything he sends out. He has been unable to be published in an north American journal in his field for his entire career [he is in his mid-fifties]--he publishes primarily in European journals--and it took him almost 15 years to finally get a book published [and he had all sorts of job woes & anxieties related to all of this] because it kept getting sent to ONE person who vehemently disagrees with everything my friend believes about their shared historical period. Yes, this person is my friend, so I may be biased, but I feel pretty strongly about the quality of his work: it's amazing, beautiful, and smart. Hardly anyone in historical studies writes as beautifully as he does, or with such conviction and high ethical standards. But career-wise, he's at the top of a certain expert-someone's blacklist, and that person has been able to hide behind his anonymous-but-not-really-anonymous-faux-anonymity. Is something wrong with that system? Uh . . . yeah! This may even qualify as "scandal" if someone's career is harmed, and I think we all know, deep down, that some careers *have* been harmed.
But it isn't just the expert's fault, it's also the fault of the editors, at journals and presses, who sometimes don't provide enough managerial oversight. Editors *could* make the peer reviewing system as it stands now better by exercising stronger oversight of the reviews solicited and received [I and my co-editor Myra Seaman, for example, have made it a policy--time constraints, notwithstanding--to review ALL reviews, and if they don't meet our standards of politeness and fairness and expertise, we send them back for revisions, and in some cases, we revise them ourselves to be as helpful as possible to the authors]. But again, going back to JJC's comments, who has the time?
[to be continued]
My first article was published in the British Library Journal [which is quite a coup in studies in bibliography--my diss. was in intellectual history and one chapter was on the intellectual history of bibliography in the early modern period, vis-a-vis the contents of the Cotton Library], but first it went through the peer review wringers at Studies in Bibliography [Univ. of Virginia] and then at Libraries & Culture [Univ. of Texas]. I got some REALLY helpful comments from 3 readers, but 1 reader wrote, in a manual typewriter no less, 2 sentences that went something like,
"She's got it all wrong. And: why did she even bother, the subject is not worth the time."
Is there something wrong with an editor sending that comment on to me, and ONLY that, as an "expert" review? Uh, yes. Everyone has at least one story of getting comments that are just plain out of left field, wrong, and rude, while we all also have stories of getting incredibly tough-yet-incredibly-beneficial feedback as well. At the same time, so much time is often wasted when the reviewer and author have no acquaintance with each other or with each other's work and preferred methodological/theoretical approaches, and how many more letters do we all need to read where the reviewer assumes, for example, we haven't read something we have, and so on? Going back to my British Library Journal article, I'm in Old English literary studies, but the article, if you didn't know better, looks as if it were written by an expert in studies in librarianship/bibliography, maybe a historian of library science, etc. So one reviewer, noting that I had quoted Kevin Kiernan [the editor of "The Electronic Beowulf," among other things having to do with codicology and manuscript studies--his area of expertise] said something to the effect of, "boy oh boy, if this author only knew how controversial and *wrong* Kiernan is, like everyone in manuscript studies *knows*, she/he would never cite Kiernan." Let's see: the "Electronic Beowulf" took up one whole chapter of my dissertation, I'm well aware of the debates over his dating of the "Beowulf" manuscript, I happen to agree with him [as a fellow expert], AND he was a reader for my article and a supporter of my own work, so I *will* cite him, thanks. I also thanked him in print with the BLJ article appeared. Yeah, these sorts of misunderstandings happen all of the time and they don't necessarily interfere with the publication of an article [after all, my article was eventually published and I got some terrific comments, too], but do we really have to make all of these mistakes along the way? I don't think so.
[to be continued]
[continuing]
More importantly for this discussion, however, vis-a-vis RB's comments, is the issue of what we might call "information overload." I think RB is right to bring up the question of whether or not we really need or *want* to wade through all of the comments that might come to us through an open or crowd-sourcing review processes [do we have the time? the ability to discriminate? etc.], and whether or not as scholars looking in from the outside we really want to, or have the time to, read all of the so-called "fodder" that goes into such an open review process. There is ample room for debate here, but again, I think RB makes some assumptions [stated as facts] that may or may not be true and/or desirable. As to the issue of "junk," trash/garbage, poop, etc. that could possibly be produced by an open review system, the world and its libraries and attics and studies already teem and flow over with thousands of small textual voices all vying for our attention, some of beautiful quality, some not. For every Proust there is a Hemingway, for every journal of a famous man the letters of a poor, unnoticed one, for every author an archivist, but not enough readers or visitors for everyone and everything [why else is Katherine Hayles so rightly excited by machine-reading?], and so on and so forth.
Nevertheless, I personally stand on the side of "more" over "less." For a long while now, I have been an advocate of open-access, digital publishing initiatives, and of all varieties, whether it is strictly regulated along the lines of traditional publishing standards [to whit: Open Humanities Press, with its eminent Editorial Board overseen by humanities luminaries such as Alain Badiou and Katherine Hayles] or whether it is Nicola Masciandaro using online "creative commons" publishing-on-demand sites to publish the proceedings of a symposium on heavy metal and theory that he hosted in NYC last year [and which event was covered by the NY Times, I might add].
Why do we cite the fact that we write for limited audiences as a reason to scale back what we do, or to limit who has a say in it? That makes no sense to me. If we really are read by no one, then stop writing altogether. Sheesh. I'm a card-carrying utopian idealist and actually think we should be working harder in the humanities to broaden our audiences, not narrowing them to willy-nilly and then being sarcastic/nonchalant about it. For all the jokes you can muster about books being used a toilet paper, I can tell stories of books that saved lives [including intellectual work]. Both stories are true, and I'll choose the latter as inspiration for my work, no matter how large, how small [if even one person] the audience. In the final analysis, for me, the traditional double-blind peer review process is a relic of a different age and I think we would all be lying a little bit if we did not admit that a lot of our professional anxieties and worries [especially for those of us more junior in the profession, or trying to get into the profession] stem from the non-transparent nature of this process.
Is the idea of transparency in academic publishing a complete mirage/fantasy, as RB implies? Well, maybe in government, or in corporations: yes. But in academic publishing? Of course the system can be more transparent: is there anything really to gain from keeping it shrouded behind the veils of anonymous "experts"? Even the idea of what constitutes "expertise" needs to be re-visited: it is, after all, a concept/state of reality we have inherited in the wake of the hyper-techno-professionalization of the humanities, post-the 19th-century.
[to be continued]
[continuing]
My final comments here, vis-a-vis RB's provocative questions, have to do with the issue that I think is the most important issue being raised here, and also connects with Jen Boyle's interest in "event-spaces," and that has to do with "product" vs. "process." RB is happy to say that all he wants is "product" [never mind the "process" that went into it, which he will happily see "flushed"--it's only fodder, after all], to which, if he wants to, he will respond to with more "product." And so on and so forth.
I actually think this gets to an issue that has been percolating as a series of discontents for a long while now in the academy, and which has also long been an important site for feminist work. In terms of prestige, getting tenure, etc., we have long valued the single-authored monograph over the supposedly more "messy" collaborative [perhaps never-ending, always ongoing] "project," and we prize the lone scholar who spends countless hours and years in his lonely study, emerging finally, like Nietzsche, with masterwork held aloft in one hand. The singular mind, the singular work, which will then be judged against all of the previous "important" singular works, and so on and so forth. No work *really* happens this way; it just looks that way, and also often gets rewarded as such.
Although I know RB and others would balk in horror at the idea [partly because, again: who has the time?], but I have long thought how cool it would be to have books published digitally that would remain "open" online indefinitely in order to allow the author to keep revising it ad infinitum, and for others [the "crowds"] to come in an annotate its margins and send "messages" to the author. What a hell of a mess it would be to keep track of all of it, citation-wise, vis-a-vis other also-continually-evolving digital publications, but it would be a much more honest representation of the intellectual process, which is always--come on!--the process of a crowd. Speaking of which, I have several of them in my head all of the time, and so do all of you [thanks recent discoveries in neuroscience!]. Just as there is no singular author, there is also no "Central Meaning Maker" [as Daniel Dennett puts it] in your head, and Jen Boyle's emphasis on event-spaces nicely highlights the what I would call the crowded, always-flowing/changing, "evential" nature of life, intellectual and otherwise. It's difficult as hell to represent, but why not try?
Speaking of trying: experiment, experiment, experiment. What else is there to do with this life we call creative and intellectual and imaginative? What a waste of time to always stay in place, just because we always have, and "if it ain't broke, don't fix it." Are you kidding me? I signed on to this life to fly, not to sit and wait.
Did I say my last comment was it? I lied [haha]. But seriously, just one codicil to all that precedes this comment:
I think one of the most valuable and important functions of the humanities, which we often overlook at our peril, is to provide sites for experimentation and new creative *designs* and models for the way(s) in which we think and work, and that these sites and spaces should be as inclusive as possible. For one reason or another our discipline has become rigidly hierarchized over time, with all sorts of gatekeepers posted here and there as well as rituals and ceremonies of "orders" and "systems": how dis this happen? Our work needs to be more collective, more democratic, more inclusive, and frankly, there shouldn't be any rules at all. How else can we move knowledge forward? Sure: who will pay for it, employ it, publish it, etc.? [are the questions I hear coming]. We will.
Oh, and in case no one got the joke, the dream I detailed about the books printed online that authors would keep revising and crowds would come in to make marginal contents, etc., that is also a good description of "publishing" in the Middle Ages, and hence, the "system" we medievalist have to deal with all of the time.
"hear hear" also to Whitney's comments here; I've been following her blog for a while now and it's fantastic.
Just to keep this fun Facebook to ITM to Facebook and back again dialogue rolling, here is another recent comment thread from Martin Foys' Facebook post pointing everyone to *this* ITM post:
From EILEEN to Richard Burt: @Richard: I've posted a long set of comments at In The Middle, responding to your criticisms/questions. I'm the one who used the term "experiment," by the way, to describe the special issue of "postmedieval" that Jen Boyle and Martin will be co-editing together. I stand by that term.
From Richard BURT: @Eileen. Great. Just be clear, I am not asking for a retraction or trying to stop the P-M experiment from going forward. Full steam ahead! I'm just asking questions based on my own peer editing experiences, experiences involving having ...my own work peer reviewed and serving on the editorial boards of ELR for 15 years and of the UMass Press for six years ( was chair for three) and for regularly reviewing manuscripts for University and trade presses. I think the current system works just fine. I have read blind submissions and non-blind submissions. No difference to me. No one I know has ever received preferential treatment (ELR was not blind submission, and to my astonishment, some excellent essays by SUPER FAMOUS were rejected). The criticisms I have read of peer-editing are on based on utterly mistaken assumptions about how the process works. (None of my questions has received a satisfactory answer, from my point of view.) So here is what I conclude and why I conclude it. First, I think peer reviewing is fine as is. Second, the SQ and P-M experimental special issues change nothing significant about peer-review editing. In both the SQ and P-M cases, the contributions are in fact peer-review edited (one could even say pre-peer-reviewed since all of the P-M essays for the special issue have already been accepted / solicited / contracted b y the editors, as was the case when I wrote an essay for a special issue of SQ on Shakespeare and film, and none will be rejected based on the experimental period; moreover, none of the contributors will have to take the comments into account the way a contributor would a reader's report). The only change is that more people without any authority over what gets published can comment on an early draft. Third, Adding the comments is not a good change, in my view. It just creates more work, extensd a process that is already quite extensive as is. (When there was a debate about an essay at ELR, the essay would be sent to 1 [ sometimes 2] more reader; the process could drag out for over a year.) Fourth, the experiment does not make peer-editing more transparent since all sorts of emails and phone discussions between the journals' editors as well as board meeting discussions about the issue and about specific essays in it will never be put online). (If you want to get an idea of what happens when "everything" gets published, check out the unusually heated responses to Sean Seasgreen essay on the Norton Brit Lit edition in Critical Inquiry; he quoted from emails.) Given all of the above reasons, I therefore have to conclude that the experiment is at best non-sensical, pointless, and a waste of everyone's time. That's just my personal review based on my experience and what I know of the experiment. I could be wrong, of course. I do not mean to offend you.
My response to Richard Burt on Facebook to his last comment: From EILEEN to Richard Burt: @Richard: I think you're partly mis-understanding the reasons behind this experiment--since the essays in "postmedieval" will have bee pre-solicited in advance, the aims for a hybrid/partial open review process are not necessarily pointed ...at changing something that is "wrong" the the typical double-blind peer review process that most journals have adopted, bur rather, is trying to highlight the development of scholarship as a certain sort of process, okay? My full comments are over at In The Middle, and I'll transfer yours there as well. I don't want to respond here at Martin's FB page to comments that are disconnected from the larger conversation over there, if that's okay with you.
I want to return to something that Whitney said, and that Ayanna Thompson said in the Chronicle piece: one of the disconcerting things about traditional publishing is the relative lack of feedback that authors get (both Whitney and Ayanna seem to be talking specifically about the writing process, but I would add that there's silence after it's been published, too). Ayanna praised the sheer amount of comments she got to her SQ piece when it went up on MediaCommons, and Whitney talks beautifully about the paucity of feedback she's gotten elsewhere. I see one of the valuable things about this platform of publishing is the level of conversation it encourages. I've read lots of reader's reports to journal submissions, and I've written them, too, and I can attest to the fact that many of them are only a paragraph or two and often written in fairly vague terms. There are others that are not, of course; there are readers who provide beautifully detailed and generous comments that really respond to and help strengthen a submission. But I think many of us agree that's not the norm. So in terms of thinking about ongoing conversations, put me in Eileen's column: what a great thing these messy conversations are!
And, again, should one wish to better understand what the process actually was at SQ, go to the MediaCommons site and read through the letter from Kathy Rowe and the many many layers of comments to the submissions.
I'd like to turn now to this comment from Whitney:
"After the rich, mutually fulfilling exchanges I've had online, the traditional journal article is freeze-dried fruit. It's why I keep falling back on self-publishing digital projects or "creative criticism" published as art, rather than squeezing my research into a form to which its methodology is antithetical. Until we start talking about changing what we accept as "essay" and "argument" to meet our current media ecology -- i.e., *making* the dreams of Benjamin's trunk a reality -- structural amendments to the publishing process end up building libraries and writing loan policies for books that don't exist. The conversation will continue to flop lop-sidedly until we start talking more about, yes, what Darnton imagined twenty years ago (what Mallarmé imagined 120 years ago, what Georg Philipp Harsdörffer imagined 380 years ago) -- the book-to-come, digital Goethe's desks."
I want to point out as well that I'm reading Whitney's comment here via a post she pointed me to via her Twitter feed, by Chris Kelty, on the ALLEGED closing of Rice University Press's, "How Not to Run a University Press," at the Savage Minds weblog:
http://savageminds.org/2010/08/31/how-not-to-run-a-university-press-or-how-sausage-is-made/
Key for me in Kelty's post was this:
"Most scholars in the humanities seem to have their heads very firmly buried in the sand when it comes to the problem of scholarly communication -- even my fellow board members seemed more eager to crow about the necessity of maintaining high standards of rigorous peer review than to face either the challenges or opportunities in scholarly communication."
[to be continued]
[continuing]
In short, our vision isn't radical *enough*. Much of the conversation here has been [perhaps unfortunately--and it's my fault, primarily] preoccupied with arguing with Richard Burt over whether or not the traditional system of double-blind, anonymous "expert" peer review is fine the way it is, or should be changed, and that's not really the primary issue to which the special issue of Shakespeare Quarterly or the forthcoming special issue of "postmedieval" are concerned. I think the larger concern [and I can only speak for myself here, not for Sarah Werner, or K. Rowe, or Whitney, or anyone involved in the SQ issue] has to do with, as Whitney puts it, creatively re-inventing what criticism is and can do [as an art form as well as a mode of so-called "scholarly" communication"]. It is also about, for me [and perhaps also for Jen Boyle], highlighting and/or representing our work as a type of collaborative "event-space" in which there are many con-fabulated intellectual [and other] comings and going as well as mediations and re-mediations of those comings and goings, and why shouldn't we work a little harder to better showcase this state of affairs? Also, why shouldn't we work a little harder, as Sarah indicates, to *enrich* and multiply the amount of feedback we all receive as we are working on/developing our projects?
Whitney is right to point out that, in some respects, these so-called "experiments" [whether engineered by SQ or others] still return a "paucity" of information/feedback/products in comparison to what she has accomplished on her own with her own digital/online projects which, although they may not yet have academic prestige [whatever the hell that is], they are certainly helping her thought and development as an intellectual & creative artist. I've learned the same lesson through my association since 2006 with In The Middle and other online sites. The lesson here, then, is that the academy [and traditional outlets & arms for academic publishing] need to CATCH UP with what has already been happening and stop debating whether or not their precious systems for supposedly guaranteeing quality and "high prestige" can be protected in this "brave new world." In other words: hearts and minds have already been captured [and mentored!], so: what next? Some things, in other words, have *already* happened. Now, how can we better make sense of this, represent it, and harness its energies for the greatest amount of good for the greatest amount of persons?
And how, further, can we FOSTER new visions for modes and production of scholarship and intellectual-creative work that, for lack of a better phrase at present, really go "outside the box"? Isn't that what the humanities should be about, anyway? As someone who studies the past, I've always felt, as I recently stated in a paper I presented in July [at the biennial meeting of the New Chaucer Society], that one of the tasks of medieval studies today would be to engage in a sort of subterfuge and resistance that draws on the resources of the deep past, hopefully formulated by us as, in the words of my friend and fellow medievalist Michael Moore, “*living* challenge[s], opening new perspectives which can lead us out of the impasse of our own times.” So, the past matters, but but not as static entity--more as matter always flowing, as we are always flowing, ever-changing. Holding on to "tradition" and an idea of "product" [over "process"] is just a kind of fearful way to work [and live]. If the past asks anything of us, it is to reinvent ourselves.
As a brief follow-up --and I think this a really productive exchange -- I'm disheartened a bit by the fact that not everyone has re-joined the conversation at ITM. As Martin and Eileen have indicated, open peer review was something I thought would add to the issue. And I still do. I also think there is some healthy wrangling to go on over the idea. That's difficult to do when there is not a good faith interest in having a real conversation.
it seems we have become more invested in controlled top-down criticism than critique.
Thanks to those who have contributed here.
I don't want to turn (even more) negative, but unless there's something Richard said on Facebook that didn't come over here, when he says:
So here is what I conclude and why I conclude it. First, I think peer reviewing is fine as is.
he doesn't actually explain why he concludes that the objections raised can be dismissed in a sentence. I'd like to see that explanation.
As to Eileen's many (many) points, the one that immediately calls a response from me is her tale of the scholar with a hostile expert in his or her subfield. As many of you know, I work on early medieval Catalonia. In the English language, that makes me one of about ten, only half of whom are senior enough to be asked to review. I think I have now identified two of their styles well enough that I know when they have reviewed something of mine. Similarly, several of them must recognise me by now, not least because of where I differentiate myself from the others. To say this system is blind is ridiculous. It's a rare scholar, for example, who can avoid mentioning their own work, referenced or unreferenced, in a review.
The other thing that now springs to mind is that doing this process online and openly enforces transparency. One doomed paper of mine was accepted for print in a certain journal on the basis of two reviews, one of which was in favour and the other uncertain of its author's qualification to comment. They had me make final revisions for press and then got cold feet and sent it to a third reviewer, who obviously came from a subfield I'd ignored and panned it. Somewhere in that, the system failed, whether the outcome was correct or not. (And, of course, I now have another place for the paper, so it's not as if the reviewing actually stopped it being published, just slowed it. The bomber will always get through.) Doing this sort of thing in public view means that rules of process will have to be clearly laid down and deviance from them will need accounting for.
I'm only being anecdotal, of course, but none of us have more than our own perspective, even those who write thus:
The criticisms I have read of peer-editing are on based on utterly mistaken assumptions about how the process works. (None of my questions has received a satisfactory answer, from my point of view.)
Surely the process works the way the people who are working it make it work... At PM the process is currently not working the way Richard expects. Alternatives are obviously possible. I thought we were talking about whether they were desirable, not defending a One True Way.
Post a Comment