Brad DeLong's Semi-Daily Journal (2004)

Technorati: Conquering the Web for Real-Time Transclusion

David Sifry conquers the web for the idea of real-time transclusion, and in the process gets a thorough education in the idea of scaling. Enormous kudos to him, and everybody else who keeps Technorati shambling along.

Wow: "The median time from when someone posts something to their weblog to when it is indexed and available for searches on Technorati is 7 minutes." 275,000 posts a day--that means that 140,000 weblog posts are being detected in 7 minutes or less: Technorati sees... all... knows... all... its eye pierces forest, stone, cloud, air, and... flesh:

Sifry's Alerts: Technorati tracks 3 million blogs: At 6:38PM PST on July 6, 2004, Technorati tracked its 3 millionth weblog. The growth of the service has been pretty remarkable - here's some stats: We're currently seeing anywhere from 8,000-17,000 new weblogs created every single day. On an average weekday, we're seeing over 15,000 new weblogs created per day. That means that a new weblog is created somewhere in the world every 5.8 seconds.

Of course, not all weblogs that are created are actively updated. Even though abandonment rates are high - our analyses show that about 45% of the weblogs we track have not had a post in over 3 months we are still tracking a significant population of people who are posting each day. The number of conversations are increasing. We're seeing over 275,000 individual posts every day. That means that on average, more than 3 blogs are updated every second. The median time from when someone posts something to their weblog to when it is indexed and available for searches on Technorati is 7 minutes. And we're striving to handle the load. But to be perfectly frank, it isn't easy. We've had some bugs and some outages - and for that I am truly sorry. I don't think the service is fast enough or stable enough. So, stability and fast response time is job #1, over new features and product developments. It has to work, 100% of the time.

I'll tell ya, it was a lot easier to ensure that when we were only tracking a couple hundred thousand weblogs, and we only had a few thousand page views per day. Those days are long behind us. The team and I (we're growing the team, btw) are working night and day to Be Of Service to you, the folks participating in those conversations. We're working on building out our backend infrastructure so that it can keep on scaling, as more and more people continue to create content on the web.

In the meantime, I beg your indulgence. Please be patient with us as we work on fixing our problems. But please be brutally honest and frank in your feedback. One of the things I love each day is reading through the comments we receive - and the best kind is the frank, honest kind that doesn't pull any punches. Yeah, sometimes it makes me squirm, but that just means that you're right. That helps me to keep the focus on our users, and how we can be of service to you, to not get complacent.

One of the things that drives me personally is that weblogs are turning us all into producers, creators, and participants in our society, not just consumers. As Doc Searls likes to say, "consumer is an industrial-age word, a broadcast-age word. It implies that we are all tied to our chairs, head back, eating 'content' and crapping cash." Of course, the act of producing, creating, and participating means that we're not doing something else - and here's the best news of all: A Forrester Research report asked Internet users which activities they were spending less time doing in order to spend time at their computers. 78% of the people polled said that they gave up television viewing. A study from The Georgia Institute of Technology's Graphic, Visualization and Usability Center showed a clear shift in media habits with more than one third of respondents saying that they "use the Web instead of watching TV on a daily basis."

Now for my Independence Day message: We're connecting with each other, we're talking to each other, finding people of similar interests, and we're having conversations. My dear hope is that this is the beginning of a rebirth of civics in America. Much in the same way that email revived the lost art of letter writing, Blogs are reviving the lost art of civilized civic dialogue - of argument, of well reasoned thought and response. And 3 million people (heck, even if you only assume that it is only 1.65 Million people, given the current abandonment rate) participating in worldwide civic discourse puts hope into my heart.

One last thing - I want to thank the team who have made this happen. You guys - our employees, friends, advisors - you folks made this happen. I can't believe how lucky I am to work with such a great group of people, who put blood, sweat and tears into making this happen. You guys made this happen. Thanks.

Update: Mary Hodder points out that not all blogs that are inactive are abandoned. In a private IM, she wrote that "people use them for very different reasons.. archive for annual event..conferences or vacations or whatever, that happen periodically and months may go by with little posting, but the postings are important and need to be searched.. until the next trip or event.."