PlanetMath Science

Thinking about "what works" for PlanetMath, and what the "science" is here. At the simplest level, what we're doing is building (or, more accurately, re-building) a website. It's not clear whether there is anything scientifically interesting about that.

Certainly there's an architecture to consider, and we may indeed be making a few innovations in this domain (which is a sort of "science" that doesn't subject itself to the usual double blind studies; namely, computer science). There are also users to consider, since we are (more accurately) re-building a site that has already existed for about 10 years -- we want to make sure people find the new tool at least as useful as the old one. So, again, there is a sort of science here (something like anthropology) where we can find user requirements and opinions, all of which can be directed back into making the site look better and feel better.

But "real" science won't be possible until we have use data -- not just designs (even implemented ones) and not just opinions (even if they are clearly and forcefully expressed). And on some level we do not know how people will use this tool until we see them using it. If we knew in advance, it wouldn't be science.

However, if we know nothing in advance, then what we're talking about here is more like art (say, Fluxus or what have you) than like science. So this makes me nervous, since I feel a little stuck coming up with good research questions or hypotheses. Maybe this is a sign not to go too far out on a limb making too many guesses until we have completed the computer science and anthropology phases to a reasonable degree.

Still, one of my guesses is that people using the site to study problems will end up improving the encyclopedia. "How to build a mathematical knowledge repository the peer-production way", to riff on PlanetMath president Aaron Krowne's similarly-titled essay about encyclopedia-building in D-Lib magazine. This is interesting to me, but what's interesting to government funding bodies is more like "How can we build a system that improves basic mathematics education?" -- and for that (very basic math), Khan Academy might already be the more or less definitive answer -- hopefully more research on its usefulness will show that. For more advanced math (which government funding agencies also care about, though perhaps less so), there isn't an answer yet. I feel I could go on repeating myself endlessly that I have a good intuition about peer production being the way to go here. But my opinion isn't based on science, at least, not yet.

The thing to do seems to be to start with the most general or generic topics and then work out to more specific ones: (1) website, (2) users, (3) data. Data is very specific. Analysis or interpretation this data if anything gets even more specific. Statements about what the data means or implies may move towards generality again. But, for better or worse, we are still at the first step.

And let me be the first to say that there is nothing wrong with computer science! It has a fairly different "peer production profile" than what I'm envisioning for PlanetMath. The website-building operation is mostly the effort of a small and close-knit group of about a half-dozen people, with another half-dozen or so participants playing smaller parts. Most of these people (with myself being the main exception) are co-located at Jacobs University in Bremen, Germany, in the "Knowledge Adaptation and Reasoning for Content" (KWARC) research group.

We are "nearly ready" to get something out the door and move on to phase (2). The schedule may be more determined by when I decide to move on to phase 2 than anything else, though frankly some important parts of the old system have yet to be re-implemented. (The problem being, we're trying to do things in a more general way, and while this means that we will be able to solve more problems in the long run, it makes the system architecting more complicated in the short run.)

Once we do hit phase (2), I plan to enlist the core user group from the "old" site as alpha testers. These will be the people whose opinions matter the most, at least initially. A follow-up set of studies could be done a half a year later with the same people, or whomever constitutes the core group in the "new" site.

All of this relates to the non-linearity of paragogy: how does a group of half a dozen researchers and research students relate to a group of two dozen mathematicians and mathematics students, and how do these groups relate to (potential) masses of math learners who are the "real" subjects of scientific interest? The layers may actually be (mostly) de-coupled (cf. Herbert A. Simon, "Near Decomposability and Complexity: How a Mind Resides in a Brain"). Nevertheless, some peer production principles may apply at each level.