August 19, 2002

Boosting the Signal-to-Noise Ratio on the Internet

Ray Ozzie believes that groups of weblogs are to discussion groups much as Google is to other more conventional search engines.

Let me explain: The big problem with conventional search engines was that there was no filter, no mind judging whether the web pages that contained the search words was worthwhile or not. Indeed, there were many minds devoted to fooling whatever filters conventional search engines set up by assigning false relevance to their own web pages. Google solved this problem through page rank: requiring that others--human others--have found a page worth linking to before it would rank it highly.

Similarly, conventional discussion groups produce either an appallingly low signal-to-noise ratio or rapid moderator burnout for largely similar reasons: flamers, spammers, et cetera have every incentive to post irrelevant information. Groups of weblogs, in Ozzie's opinion, solve this problem by a means similar to Google: if a flamer publishes a rant on his own weblog and nobody links to it, does it burn anything?

I don't know how right or relevant this argument is, but it is a very interesting one.


Architecture Matters: The Rebirth of Public Discussion: Over the years, I've been either a participant or a lurker on countless systems hosting public discussion forums.... And one thing has been a fact of life: that hand-in-hand with public discussion comes spamming, flaming, and other dysfunction that in large part progressively decreased the signal-to-noise ratio of any long-standing topic.... But what has struck me over the past few weeks is the fact that blogs represent a radical new approach to public discussion - one that, in essence, completely and naturally "solves" the signal:noise problem, and does so through creative exploitation of a unique architecture based upon decentralized representation of discussion threads.... blogs accomplish public discussion through a far different architectural design pattern... it "solves" the signal:noise problem is that nobody bothers to link to the "flamers" or "spammers", and thus they remain out of the loop, or form their own loops away from the mainstream discussion. A pure architectural solution to a nagging social issue that crops up online....

Architecture Matters: The Rebirth of Public Discussion

Having been an engineer and designer for most of my career, I often find myself trying to explain to others why I am so passionate about software, and in particular, why I believe that understanding something about architecture  is fundamental to anyone who makes decisions related to technology, and that choice of one design pattern  over another can dramatically impact outcome. Decisions such as centralized versus decentralized, hierarchical versus cellular, RPC versus MQ, API versus protocol, structured versus semi-structured, etc., are so very critical to understand in order to make an informed choice.  At the technology level, at the business level, architecture matters.

Over the years, I've been either a participant or a lurker on countless systems hosting public discussion forums - starting in the mid 70's at PLATO, but continuing with The Well, Echo, The Source, CompuServe, Prodigy, Usenet, Fidonet, eGroups, and many others. And one thing has been a fact of life: that hand-in-hand with public discussion comes spamming, flaming, and other dysfunction that in large part progressively decreased the signal-to-noise ratio of any long-standing topic.

Much study has been given to this topic, and although the most effective solution seems to have been human-moderated discussions, many creative folks have attempted to combat the problem with techniques such as community ranking and other such reputation systems. From my vantage point, I'd long accepted that dealing with "noise" is just something that goes with the territory in public forums.

And then came blogs.

(Pardon me if I'm simply rediscovering something that has been well-covered elsewhere, but it's truly fascinating to me.)

Over the past few months, I've lurked around a broad variety of blogs, trying to figure out in my own mind what exactly was going on, and watching others as they discovered the medium: trying to understand where and when to tag items and use permalinks, exploring the use of referrals as reverse-engineered backlinks, and experimenting the use of aggregation techniques to apply centralized organization to a decentralized web of activity. It has been truly fascinating for me, and, as discussed elsewhere, I realized that I would only truly learn if I joined into the fray.

But what has struck me over the past few weeks is the fact that blogs represent a radical new approach to public discussion - one that, in essence, completely and naturally "solves" the signal:noise problem, and does so through creative exploitation of a unique architecture  based upon decentralized representation of discussion threads. Let me elaborate.

In traditional discussion, topics and their responses are contained and organized within a centralized database. The relationship between topics and responses is generally maintained in a manner specific to the nature of the database - that is, in newsgroups the messages might be related by Message-ID hyperlinks or crudely by title, in Notes they are related by the $REF hyperlink, and so on. Summary-level "views" are generated through database queries. And that has been the general architectural design pattern of public discussions for quite some time.

But blogs accomplish public discussion through a far different architectural design pattern. In the Well's terminology, taken to its extreme, you own your own words. If someone on a blog "posts a topic", others can respond, but generally do so in their own  blogs, hyperlinked back to the topic's permalink. This goes on and on, back and forth. In essence, it's the same hyperlinking mechanism as the traditional discussion design pattern, except that the topics and responses are spread out all over the Web. And the reason that it "solves" the signal:noise problem is that nobody bothers to link to the "flamers" or "spammers", and thus they remain out of the loop, or form their own loops away from the mainstream discussion. A pure architectural solution to a nagging social issue that crops up online.

The downside? Well, part of why people like getting together is that unintended consequences  can be quite rewarding. And there's a danger that the self-selecting environment of a given blogging community might limit unintended outcomes. But, then again, I could argue quite the opposite: in a traditional public discussion, a good idea might get lost in the noise.

The bottom line: architecture matters, and we've yet to discover the limits of where this fascinating medium will take us.

Posted by DeLong at August 19, 2002 12:38 AM | TrackBack

Comments

Weblogs are a recent example of a breed of tools dubbed "reputation managers". Usability guru Jakob Nielsen came up with the term. It can also be applied to Google, Amazon and various other Web success stories. And it seems to me to be an economic answer to an economic problem, even if no money changes hands. An extract from an article I wrote last year (archived at http://www.shorewalker.com/pages/reputation_manager-1.html):

"'How do you sort out the good stuff on the Web?' ... The standard answer to the 'good stuff' question used to be: 'Go to a directory'. Yahoo, LookSmart and others used paid editors to build lists of the Web's best sites. But that model could't deliver quality and still remain economically viable. While Yahoo and its ilk turned to the business of being 'portals', the Web waited for other solutions.

"The first of those solutions came from an unexpected direction. Individual sites such as Amazon, Epinions and eBay developed systems to let users post reviews or items for sale, and let other users pass judgement on the quality of the people who'd done the posting. Rather than paying money, they paid for content and users rankings with the currency of user recognition, respect and goodwill. They created systems for giving their users reputations. It may not be coincidence that Amazon, Epinions and eBay all appear at least capable of surviving the dot-com collapse ...

"... While Google, Epinions, Amazon and eBay have asked large numbers of people to judge reputations, another reputation tool is approaching the problem from the other direction. Blogging software from firms like Dave Winer's Userland Software and Evan Williams' Blogger.com has allowed individuals to publish their own lists of sites, complete with commentaries. Weblogs' users don't vote within the site; they vote by choosing the site as a reliable source of guidance. In effect, they say to the site's author: 'you make the choices I'd make if I had time'. The Webloggers become the makers and breakers of reputations within their (usually narrow) areas of interest. And the mass of Weblogs becomes another reputation management system ..."

Posted by: David Walker on August 19, 2002 06:52 AM

Ah. Thanks...

Posted by: Brad DeLong on August 19, 2002 11:03 AM

I have a quasi-economic model of how blogs act as an information filter at

http://www.corante.com/bottomline/articles/20020621-875.shtml

Posted by: Arnold Kling on August 19, 2002 11:20 AM

In terms of filtering out flaming etc, it seems to me that a "good flame" is precisely what is more likely to be linked to than a sober piece of weblog analysis. If this was what weblogs actually do, wouldn't the same logic apply to, say, newspapers, and wouldn't the logic suggest that say, The Financial Times would be more widely read than The Sun? Or talkback radio? The right bloggers seem particularly prone to linking to the most viscious stuff. Something about this theory doesn't sound right to me.

Posted by: tim dunlop on August 19, 2002 02:37 PM

I have not seen any evidence of reduced tribalism in blogdom vs any other forum architecture. Opposing camps form, identify each other as "other", distance each other, throw virtual stick and leaves in the air, become progressively more reciprocally loyal internally and reciprocally hostile externally. One side's heroes are the other side's villains, one side's truths are the other side's falsehoods, and any middle ground is rapidly deforested.

I think architectures with different emergent dynamics are possible, but proof is lacking.

Posted by: RonK, Seattle on August 19, 2002 03:44 PM
Post a comment