Reflections on the process

I chatted briefly over oatmeal with my friend (the one who invited me to live in this house), about anthropology and sociological studies of science. I'll be working today on a "rebuttal" to some reviews for a co-authored paper that was sort of in that niche.

The paper is about "Massively Distributed Authorship of Academic Papers", and it has about 30 co-authors. Our approach was to write about the experiences we had writing the paper, if that makes sense. From a methodological standpoint, this approach is similar to something called the "talk-aloud protocol", where someone doing a task "talks aloud" about the thought process they are going through when solving a problem. This is written about by our old paragogical pal K. Anders Ericsson.

Anyway, two of the reviewers found the paper confusing and easy to dismiss ("It is a very 'Seinfeld-ish' paper, in that it is a a paper about a paper about a paper"). One of them, however, was much more complementary: "The whole point of science is to provide a base for others to build on, and this paper does so in a unique and excellent way. In short, this may be the most exciting CHI paper I've ever reviewed, and I reiterate my strong recommendation that it be accepted." In short, the reviews came back something like a 7-10 split. For whatever reason, I've personally taken on the task of trying to pick up the spare. Maybe the other co-authors have for the most part given up at this point, it's hard to tell.

What's so interesting about the paper? In the words of our star reviewer, "This new approach would break a lot of tacit knowledge and assumptions about how research works." Imagine working on research not as a small group, but in a large distributed pool, sort of like the way Wikipedia is authored. But in fact, this isn't fair as "what if" question. As we've noted several times, science is already perfectly paragogical. It's absolutely true that just reading works from someone else is better understood as andragogy, but when you contribute something back to the discourse, paragogy kicks in (even over massive distances and timespans). The assumptions that the "massively distributed authorship" approach breaks are not so much our grounding assumptions about scientific discourse, as assumptions about how people get credit for their work, and about the technology they use to do the work.

So the "what if" question shifts onto technology and something more like sociology or economics. We're not submitting the paper to a sociology journal, but writing for a conference on human computer interface issues. But in either case, we should be asking, how is this really going to work? For example, is Etherpad really sufficient for real-time collaboration, or do we need something else? Is real-time collaboration "better" in any genuine respect than asynchronous means, like a wiki or version control system? If we kept coming back to reflective questions like that, the paper would be a better paper.

As it was, we drifted quite frequently onto sociological questions like "how to distribute 'credit' for authoring papers like this". First and foremost it is important to realize that there will be no credit to distribute if the paper isn't published. But the issue we took up in the paper was more like "what does it mean to be 1st author on a paper with 30 authors? or 6th author?  or 27th?".

Experientially, working on the paper was something like participating in a seminar or a course -- perhaps a P2PU course where instead of having a course forum, we only had an Etherpad, and instead of each person having his or her own "assignment", we had a shared goal, putting together a paper. (Hey, our paragogy writings are kind of like that too.)

In a sense, the recursive nature of this particular paper was both its greatest strength and its greatest weakness. It's a strength because we got to examine the thought process of co-authors in detail, something we couldn't have done so easily had we been trying to study the work of (for example) Gowers et al. in the Polymath project. It's a weakness because people found it confusing and distracting, maybe sort of like a relationship where too many conversations are devoted to "processing".

But perhaps there is a fundamental statement about how science works in this kind of writing. We don't get away from reflections on the process (nor should we try), however, we would do well to work to make the reflections give us something useful and coherent at the end of the day.