February 18, 2003

Less Than $1 a Gigabyte

Moore's Law gets all the press. But should it? Dan Gillmor writes about the mass-storage equivalent of Moore's law: how hard disk prices have now dropped below $1 per gigabyte:


Mercury News | 02/16/2003 | Dan Gillmor: Disk-drive capacity continues to grow: The cost of disk-drive capacity has dropped below a buck a gigabyte. The news was anticlimactic in a sense, because it was predictable in a time of constant technological progress. It was one of those milestones that still means something -- not just a testament to the storage industry's achievements, but also a reminder of how the seemingly most prosaic of information technologies may have turned out to be the most disruptive.

Like others of my generation, I can remember my first computer with a hard disk, which I bought in the mid-1980s after years of personal computing with floppy disks and, before that, cassette storage. I wondered how I'd ever fill the 10 megabytes on the IBM PC, given that an entire novel's worth of text takes up less than 1 megabyte, or 1/60,000 of the space on today's typical 60-gigabyte hard drive. I soon found out. In a Digital Age corollary to Parkinson's Law, data expanded to fit the available space. I learned this repeatedly as the types of data changed. Text was efficient. Graphics took up more space. Microsoft created file formats for its Office software that took up even more.

Silicon Valley's disk-drive innovators kept pace. Then came multimedia, especially MP3s, the music format that routinely uses a couple of megabytes a song. My Apple iPod MP3 player will fill up one of these days. But the iPod has a measly (!) 10 gigabytes. By this time next year we'll see tiny media players with 40 or 60 gigabytes of storage.

Ah, but what about video? That's the ultimate space hog, right? Yes, for now. My hard-disk video recorder at home has what today is a laughably small capacity, 40 gigabytes, but it holds more than 30 hours of video I record from the satellite, ensuring that there's usually something I want to watch when I get home. The kinds of files we store keep getting bulkier, but the disk-drive wizards are moving fast enough to stay ahead. In the next few years, given their continuing innovation, they're likely to do something I didn't imagine possible until recently -- give us so much storage at such a low cost that we genuinely don't know how to use it all. But in the short term, disk space still will be somewhat constrained.

Here's one example of how we'll use it: I just installed the 2003 Encyclopedia Britannica on my laptop computer. It came on a DVD disk and took up about 2.4 gigabytes of space. This is the same encyclopedia, with multimedia additions, that used to take up a huge bookshelf. Now I carry it around. The immense storage capabilities of computer disk drives also make me wonder whether applications we once assumed should reside on central servers might migrate back down to the desktop. Corporations could install employee Web sites on laptop computers, for example.

I'd also like to have a home server that stored everything -- music, movies, reference materials, software, you name it -- for easy access by devices I use around the house. Of course, I'd want a backup of everything. But the huge capacities of drives for desktops and servers remind us of another aspect of the industry's progress. Disk drives are getting smaller, too. We'll soon embed huge amounts of storage into small devices.

Digital movie cameras are starting to be equipped with hard disks, a natural step. Soon enough, though, all kinds of other things we use every day will make a record of what they're doing. That will raise some new questions. For example, what are the privacy implications of our automobiles keeping track of where we drive and at what speed, as the car makers, insurance industry and government snoops will surely wish? In the medium term, we could fill up disks at the edge of networks in order to help spread multimedia content to people who want to use it. This will be necessary given the near-zero probability that the telecommunications industry will give us sufficient bandwidth -- the speed of network data connections -- for centralized multimedia delivery.

The furor over digital copying is a direct outgrowth of storage improvements. Not until large numbers of songs and videos could be easily stored on personal hard disks did the music and movie industries get seriously worried about the networks that began to connect those disks. Without mass storage, Napster would have been much less relevant. The expanding capacity of portable drives opens a new front in the entertainment cartel's war with its customers. Song traders don't need to use the Internet anymore. They can hold parties in friends' homes, swapping songs from disk to disk. Soon it will be movies.

I rely on futurists to help me understand what we'll store on disks or other storage memory, such as flash memory, in the long term. Gordon Bell, a computer industry legend, imagines wearable devices that hold everything we've ever seen, heard and said. I'm not crazy about that vision, but when disks hold a thousand times more than they do today, your entire life will fit -- plus more books, songs and movies than you could use in a thousand lifetimes. Yikes.

Posted by DeLong at February 18, 2003 08:45 AM | TrackBack
Comments

My sister, a CalPoly math prof, needed a quick and easy way to move large files between home and office, so I suggested one of those USB plug-n-play "keychain drives".

The 64Mbyte Sony DataVault she selected was the first mass storage device I've ever seen come with a paper tag warning "Harmful or Fatal if Swallowed - Keep out of the reach of small children"

Posted by: contract3d on February 18, 2003 09:07 AM

In our lifetimes, we will have a tablet pc and will have access to all of human knowledge, from books, to music, to movies, to self created media. But will it help our understanding? We have all of this nifty knowledge already (the entire Briticana you can carry around!) and yet we stupidly use our knowledge and treasure to bomb poor countries. Still, it is wonderful to see the cost of information plummet and wonder what free information will portend.

Posted by: Troy McClure on February 18, 2003 09:19 AM

the downside to carring everything you have is access and sync. you have to make sure that the memory device is accessable to all components or you will need to carry one device for your camera, one for you mp3 player, one for your computer, etc....

this is more or less already the case with the current generation of devices - what a pain in the ass.

Sync is another problem. Sync works well with palm pilots but few other applications. Just imagine managing five devices, four storage formats, two operating systems so that the file(s) you want are always available.

This is why I was such a big fan of X-drive and other now defunct internet based storage systems. Operating system neutral (access via a browser), available from any net connected computer and file format neutral systems made my life much easier.

I would guess the return of these business models (like the yahoo storage option) maybe mixed in with microsoft actually making your computer useful by bundling a controlled access web server.

For me anyways, it isn't about size but how you use it.

Posted by: jjj on February 18, 2003 09:34 AM

"...and yet we stupidly use our knowledge and treasure to bomb poor countries."

I'll bet if you took a completely guaranteed-anonymous poll of every single person in Afghanistan, a majority would say that their lives are better now than they were under the Taliban.

But the much more relevant outcome of our bombing of Afghanistan is that there is no longer a central government in Afghanistan that: 1) allows terrorist training camps to be openly run, and 2) openly protects Osama bin Laden.

Posted by: Mark Bahner on February 18, 2003 09:36 AM

"Digital movie cameras are starting to be equipped with hard disks, a natural step."

I don't see that as the *only* "natural step." There are currently flash memory cards that hold hundreds of megabytes. Soon it will be 1-10 gigabytes.

http://www.ultracompactflashcards.com/san-sdcfh-256.asp

It seems more logical to me to go to flash memory, unless one routinely records many hours of movies. If one just records a few minutes of various family get-togethers, the flash memory cards seem more logical to me. They're much more drop-resistant (to 10,000 G's). No moving parts, so they conserve power. Can be switched in and out.

Posted by: Mark Bahner on February 18, 2003 09:47 AM

Mark, you could extend that to say that there is no longer a central government in Afghanistan at all. We destroyed is (or at least our hired mercenaries from the Northern Alliance did) and replaced it with a puppet that has no authority at all.

I'll bet if you polled every person in Afghanistan they'd say that things are changed very little, if at all.

All we accomplished was to replace a dictatorial central government (the Taliban) with a set of dictatorial local governments (the drug-producing warlords).

We never did much more than start the job in Afghanistan, and we will come to regret the fact that we left it like that.

Posted by: Chuck Nolan on February 18, 2003 09:47 AM

>>hard disk prices have now dropped below $1 per gigabyte<<

...and the hard-disk companies have been losing money for years. I don't understand why they continue to innovate at such a blistering pace if they can't make a dime from it.

Posted by: YM on February 18, 2003 10:33 AM

"...and the hard-disk companies have been losing money for years. I don't understand why they continue to innovate at such a blistering pace if they can't make a dime from it."

The guys in the lab with the propellor beanies innovate just because of the innate coolness of it all. (I know. In a small way, I'm one of them).
Whenter there's a dime to be made from it concerns them not at all.

That said, the business types in the corner office are supposed to be paying attention to that stuff. What's their motivation? "We'll make it up on volume"?

Posted by: Chuck Nolan on February 18, 2003 10:41 AM

"I'll bet if you took a completely guaranteed-anonymous poll of every single person in Afghanistan, a majority would say that their lives are better now than they were under the Taliban."

What about those polls that claim that a majority of Americans think that most of the 9-11 hijackers were Iraqi? (This is strictly "IIRC".)

Or that we spend a considerable amount of the Federal budget on foreign aid?

Ah, the Information Age...

Best,

Posted by: Stephen J Fromm on February 18, 2003 11:04 AM

Not that this has anything to do with the post but...

In fact we did such a good job in Afghanistan that the budget Bush submitted to Congress contained $0.00 for rebuilding efforts there.

We're gonna pay for this.

Posted by: biz on February 18, 2003 11:59 AM

"That said, the business types in the corner office are supposed to be paying attention to that stuff. What's their motivation? "We'll make it up on volume"?"

Their motivation is called "competition." IBM, Seagate, Maxtor....if they don't double their disk drive space every year, for the same price, their competitors will.

Still, the pace of the development is spectacular. I remember very clearly, in the small business I was working at, circa 1991, we were discussing keeping data on floppies versus hard drives. I remember very clearly that the cost of a hard drive was around $2 per MEGAbyte (i.e., $2000 per gigabyte). Now, a little more than a decade later, it's less than $1 per gigabyte. Truly amazing.

Posted by: Mark Bahner on February 18, 2003 02:34 PM

This paper has a lot of neat data on computer trends...both storage, and processors.

Check out page 11 of 43, with the graph of processing power over time, and compared to various types of animals.

Currently, computers are about as powerful as a spider or lizard brain. But, by 2020, $1000 will by you a computer with the power of a human brain. *That's* when things will really get interesting! :-)

Posted by: Mark Bahner on February 18, 2003 02:43 PM

Oops!

Forgot to include the link:

http://www.tomcoughlin.com/Techpapers/Data%20Storage%202001%20Talk,%20110601.pdf

Posted by: Mark Bahner on February 18, 2003 02:45 PM

Biochips and Venter's Law

Just as the microprocessor enabled the cost of computation to drop from $75 to less than a billionth of a cent for a million computations between 1970 and 2003, so too will the cost of sequencing an individual's genotype drop through the floor over the next twenty years.

With the completion of the Human Genome Project just three years behind us, the cost has already fallen from approximately $3 Billion to sequence an individual's genome to several hundred thousand dollars today. Recent, viable, claims have been made that a $1000 genotyping system could be possible in the next 5-10 years. (See: http://www.bio-itworld.com/news/100302_report1277.html )

Makes me wonder when genotyping will become all the rage as a Holiday gift? 2014?

Posted by: Zack Lynch on February 18, 2003 04:33 PM

Avoiding the Afghanistan side-thread (it's more like a return to 1992-1996, after which the Taliban was considered a Good Thing) the issue of the diminishing cost of storage raises a different issue, which is one of data retrieval.

a. we need better ways to store and search for data on our own hard drives
b. we need future-proof file formats.

As others have said, there's a growing need for home users to have the kind of search interface and underlying algorithms of a Google. We're getting there, slowly -- David Gelernter's Lifestreams has been on the go for years, and it's part-way there -- but it'll take a brave step to abandon the stupid desktop metaphor and start implementing ad-hoc organisation based upon metadata, implied metadata and usage patterns

Posted by: nick sweeney on February 18, 2003 06:00 PM

AltaVista (the search king before Google) *did*
have a personal computer tool for indexing one's hard drive and speeding up text searches. It did more than HTML too, indexing dozens of different PC file formats. Not sure it had the same kind of proximity intelligence of even old AltaVista, let alone the link usage rating of Google.

I'm a bit surprised that RAID (redundant disks) didn't get mentioned in the parent article. Is that too obvious? Once I've put 60 gigabytes of important data (or even 20) on a disk, I want it automatically backed up by at least one other drive. So I guess we're back up to $2/GB unless performance is an issue whereas striping... no matter it's still amazingly cheap. Regarding disk manufacturers not making money: hmmm, smells like movie accounting. They also don't make money.

Posted by: Dennis Slough on February 18, 2003 07:28 PM

"and the hard-disk companies have been losing money for years. I don't understand why they continue to innovate at such a blistering pace if they can't make a dime from it."

The only profits come from being first-to-market. Everyone is going to reach the next level of innovation so the business guys push the innovators to get there first. They receive the profits until the competition catches up and the product is commoditized (again). Then you race to be first-to-market in the next level of innovation.

How is Afghanistan doing in that race anyway?

Posted by: Dan Jordan on February 18, 2003 09:15 PM

I believe the hard drive companies have actually turned profitable again. It's just that there had been too many competitors so there was not usually pricing power... not good for a capital and R&D intensive industry where much equipment becomes obsolete in a year. I guess everybody tried to gain market share to achieve economy of scale, and that led to overproduction and low/negative returns and that led to consolidation. Well, that plus the fact that Samsung decided to muscle its way in some years ago.

Posted by: snsterling on February 18, 2003 10:58 PM

Some people think the most scary part is that everything will be recorded by these tiny self contained "endless" recorders. Society will become greatly changed by recording of almost all illegal activity, and possibly by an expected loss of privacy (use your imagination to figure out for example what high school/college students would do with these things... not that video cameras, hidden cameras and X10's don't already exist, but they are not small, self contained, endless, cheap, and in common use).

Anyway, that's not really the scary part. The scary part is that as processing power advances, it will be possible to fake any picture and eventually to fake a video of anything. Now this will be a very serious problem because no picture or video will even be accepted as court evidence unless it comes from a verified tamper proof recorder. In fact, even a news reporter would not be able to trust video images which today one not even think of verifying.

Posted by: snsterling on February 18, 2003 11:26 PM

"...it will be possible to fake any picture and eventually to fake a video of anything..."

Unless every pixel has a serial number. Now there's a way to soak up a lot of disk.

What you say *does* make the mind boggle.

Posted by: Dennis Slough on February 19, 2003 04:38 AM

In related news, the capacity of the human brain remains roughly the same as it was a thousand years ago.

>>The scary part is that as processing power advances, it will be possible to fake any picture and eventually to fake a video of anything<<

All the technology in the world can't make Keanu Reeves able to act convincingly; why do you think that this will change in the future?

Posted by: dsquared on February 19, 2003 08:37 AM

"All the technology in the world can't make Keanu Reeves able to act convincingly"

dsquared overlooks Reeves brilliant role in "Bill & Ted's Excellent Adventure". Or was that acting?

Posted by: Dan on February 19, 2003 09:24 AM

"All the technology in the world can't make Keanu Reeves able to act convincingly;..."

Look at Gollum in the Two Towers. In another decade, Keanu won't even need to show up on the set. They'll just take his head, and paste it on someone else's body. (Someone who can act.) And in another decade after that, they won't even need the body. It will just be a completely computer-generated KR. From the battle scene in the Two Towers, I'm not sure they can't do that already (at least for shots from a distance).

Posted by: Mark Bahner on February 19, 2003 09:42 AM

There's still a wide range of estimates on how much processing power you need for a human brain equivalent. Seeing how toilet-cleaning robots are way out of our grasp, I'm not holding my breath.

Posted by: Jason McCullough on February 19, 2003 10:12 AM

Faking a video scene can be accomplished with only improved graphics. This includes improved textures, physics (especially of light), plus other things. To attain realistic motion you can use actors with motion sensors like they already use (Homer Simpson once did a demonstration). Even better, it should not be hard to analyze a video of someone and record their individual mannerisms.... combine this with high level scene direction and some physics calculations and you have a fake video. Fake your death, fake someone elses death, fake some inappropriate behavior by your boss, whatever you like. And you can guarantee the necessary technologies will be developed because of the size of both the movie and video game business.

20 years ago, a "person" on a video game was a collection of a handful of pixels where a few were varied to give the appearance of motion!

Posted by: snsterling on February 19, 2003 12:41 PM

"There's still a wide range of estimates on how much processing power you need for a human brain equivalent."

Hmmmm. I guess that would depend on what you consider to be a "wide range."

Ray Kurzweil, in the Age of Spiritual Machines (highly recommended reading, for those interested in artificial intelligence, or computer progress) calculates 20 quadrillion (i.e., 20 million billion) operations per second:

http://felix.openflows.org/html/kurzweil.html

This other site calculates 10^17 operations per second...or 100 quadrillion operations per second.

http://www.sciforums.com/archive/32/2001/07/1/3224

The difference is a factor of 5 in operations per second. But with the number of operations per second doubling every 1-2 years, the factor of 5 will be achieved in 2-5 years.

Whether it's 2020, or 2022, or 2025 (or even 2030, as the second site calculates)...computers will still achieve the power of a human brain within the lifetimes of most of the readers of this post.

In fact, I personally hope to retire by hiring a $1000 computer to do all my work for me. ;-)

And if one lives another 50 years (which I also hope to do), $1000 computers will have computing power on the approximate scale of all the human brains on earth. Pretty amazing stuff!

Posted by: Mark Bahner on February 19, 2003 02:44 PM

"Seeing how toilet-cleaning robots are way out of our grasp, I'm not holding my breath."

Interesting comment, as toilet-cleaning robots have always been an interest of mine (ever since reading Heinlein's The Door Into Summer, 30-odd years ago).

The problem of a toilet cleaning robot is much more one of indexing or machine vision, than brain power.

I'm sure you could buy a perfectly good toilet-cleaning robot with a $30 electronic brain. But it would cost you $1000 for the mechanical stuff...and toilets would be *all* it was capable of doing.

Or, you could get a Lego set, and build your own:

http://www.occdsb.on.ca/~proj4632/evans.htm

Lego! Heh, heh! Heh, heh! C-c-cool! :-)

Posted by: Mark Bahner on February 19, 2003 02:51 PM

"20 years ago, a 'person' on a video game was a collection of a handful of pixels where a few were varied to give the appearance of motion!"

And I *still* say "Asteroids" is the best video game of all time! "Grand Theft Auto," indeed! ;-)


Posted by: Mark Bahner on February 19, 2003 03:01 PM

Well, AI is nowhere near enough of a mature subject. A thousand-fold increase in processing power really doesn't get you much with our current understanding of the science of thinking.

Posted by: Jason McCullough on February 19, 2003 05:53 PM

"Faking a video scene can be accomplished with only improved graphics. This includes improved textures, physics (especially of light), plus other things. To attain realistic motion you can use actors with motion sensors like they already use (Homer Simpson once did a demonstration). Even better, it should not be hard to analyze a video of someone and record their individual mannerisms.... combine this with high level scene direction and some physics calculations and you have a fake video."

Ever see the Final Fantasy movie? I don't think we are all that far from that.

And something I have not seen addressed here - at some point, it seems that we would hit a physical limitation.

Posted by: kevin on February 20, 2003 11:59 AM

Yea! You guys are all assuming exponential growth 4ever. It doesnn't work that way. If Aircrafts had continued their exponential growth that they had from 1902 to 1970 we would be taking that vacation on the moon that we heard so much about. But it didn't. It platoed. The same will happen to computers eventually. We just have no idea when.

Posted by: Callmeindy on February 21, 2003 09:59 PM

Yea! You guys are all assuming exponential growth 4ever. It doesnn't work that way. If Aircrafts had continued their exponential growth that they had from 1902 to 1970 we would be taking that vacation on the moon that we heard so much about. But it didn't. It platoed. The same will happen to computers eventually. We just have no idea when.

Posted by: Callmeindy on February 21, 2003 09:59 PM

>> But it didn't. It platoed. The same will happen to computers eventually. We just have no idea when. <<

Well, just like birds were proof of the possibility of flight before we ever invented a flying machine...

Try this experiment to demonstrate the feasibility of a device with a million times the computational power of the current generation desktop computers: have sex, consume food and water in increased quantities as desired, and in about 9 months out of you comes an advanced computational device (please note, experiment might not work for men. there have already been some reports that the advanced computational device might emerge instead from sex partner.)


Posted by: snsterling on February 23, 2003 01:57 PM

"And something I have not seen addressed here - at some point, it seems that we would hit a physical limitation."

With single-molecule computing, especially single-molecule that physical "limitation" is so mind-bogglingly unlimited, that it doesn't really matter.

There are 6x10^23 molecules in one gram-mole of a compound. In other words, since styrene (to pick a molecule at random) has a molecular weight of 104, 104 grams of styrene contains 6x10^23 molecules.

If every one of those molecules is a transistor, you've got 104 grams of styrene (about 4 ounces) that has more potential computing power than all the human brains on earth right now, combined.

http://www.sciam.com/article.cfm?articleID=00017C07-D8E3-1D07-8E49809EC588EEDF

http://sci.newsfactor.com/perl/story/13110.html

And chemical companies produce styrene by the thousands of tons!

Posted by: Mark Bahner on February 24, 2003 09:44 AM
Post a comment