This summer at Harvard’s Berkman Center for the Internet and Society, Still Water’s John Bell and Jon Ippolito demo’d two networks designed to help more people access and contribute to academic research and development. The Pool focuses on the creative process, stimulating and documenting collaborations among artists and programmers. (Rick Rinehart at UC-Berkeley was an early adopter.) ThoughtMesh, meanwhile, focuses more on sharing the products of academic research; its auto-generated tags connect essays on similar themes drawn from different sites across the Web. As an upcoming feature in Leonardo magazine argues, such “crowdsourcing” networks may change the way creative and scholarly research is recognized by universities across the world.

The Berkman Web site features a video of the talk and this description:

The Internet both attracts and repels art institutions. Curators wonder who could possibly ensure quality control in a world where 50,000 videos are added to YouTube each day. Fortunately, artists themselves were crowdsourcing long before the Internet: composer John Cage laid out the principles fourteen years before Richard Stallman founded the Gnu project and twenty-nine years before the term ‘open source’ was coined. In addition to collaborating on their own creative projects, artists have helped to build the very recognition networks necessary to find the Leonardos among the LOLcats. This month saw the public release of two social networks, The Pool and ThoughtMesh, designed to help collaborators and critics find and evaluate each other. Unlike existing publishing systems such as blogs and wikis, these networks aim to give ordinary users a ‘big picture’ as well, and include graphical and lexical tools that can help answer such questions as how networked creativity is enhanced or hurt by licensing choices, the number of contributors, and project lifespan.

The Berkman Center presentation followed on the heels of the May 30 issue of the Chronicle of Higher Education showcasing The Pool, ThoughtMesh, and John Bell’s Re:Poste distributed review network as potential mechanisms for discovering and appraising academic work. Adding to the evidence that a sea-change may be brewing for how higher education evaluates its researchers, MIT’s Leonardo magazine of art and technology will be publishing the official promotion and tenure criteria of the U-Me New Media Department in early 2009. The magazine will publish these criteria along with a white paper entitled “New Criteria for New Media,” which argues for the redefinition of traditional criteria for excellence in the age of networked scholarship.

Written by U-Me faculty Joline Blais, Jon Ippolito, and Owen Smith in collaboration with Steve Evans and Nate Stormer, the criteria and white paper underscore the potential of crowdsourcing networks such as The Pool and ThoughtMesh to provide alternative evaluation mechanisms for academics:

Peer-evaluated online communities may invent their own measures of member evaluation, in which case they may be relevant to a researcher who participates in those communities. Examples of such self-policing communities include Slashdot, The Pool, Open Theory, and the Distributed Learning Project. The MLA pins the responsibility for learning these new metrics on reviewers rather than the reviewed.[17] Given the mutability of such metrics, however, promotion and tenure candidates may be called upon to explain and give context to these metrics for their reviewers.

As if to reinforce the conclusions of the white paper, ThoughtMesh co-developers Craig Dietrich and John Bell have just launched a commenting system internal to the ThoughtMesh network with the provocative heading of “peer review.” Unlike the relatively uncontrolled comments at a site like YouTube, however, ThoughtMesh’s reviews are subject to a rigorous trust metric. Each reviewer must claim a level of expertise before rating an article, and the software holds them accountable in a way that goes beyond even the rigorous method of peer reviewers for academic journals.

As might be expected, a review by someone claiming expertise will have more effect on the overall rating of the essay than by someone who claims none. However, those who claim expertise have to live up to it. If an academic makes exaggerated claims and is then trashed by her peers, her credibility will plummet *faster* than if she claimed no expertise in the first place.

The Pool and ThoughtMesh are free and open to anyone to try. For more information, please contact John Bell, Craig Dietrich, or Jon Ippolito.
(ThoughtMesh was built by Craig Dietrich and Jon Ippolito with John Bell; The Pool was conceived by Joline Blais, Jon Ippolito, and Owen Smith and built by John Bell with Matt James, Jeremy Knope, Justin Russell, and Mike Scott.)