Metadata: Archive Entry From Brad DeLong's Webjournal

July 12, 2004

Metadata

Daring Fireball assures us that Apple's Spotlight will be a real and a well-thought-out product:

Daring Fireball: Spotlight on Spotlight: ...two years ago... Apple hired Dominic Giampaolo, renowned file system design expert and creator of the highly-regarded, metadata-rich Be File System.... [W]hat then has Giampaolo been working on?... Spotlight — which is, in the words of one WWDC attendee, Giampaolo’s “baby”.... [T]he aforementioned source who attended the Spotlight session at WWDC sent me the following report:

Spotlight is completely, relentlessly focused on files and files’ metadata. Files are the only object returned to Spotlight queries. Two aspects of Jobs’ keynote were thus misleading: The “spotlight” effect on System Preferences was wholly unrelated to Spotlight. Spotlight’s ability to show results from Apple Mail archives on Jobs’ machine was tantamount to a sham. Believe it or not, Tiger Mail has switched to an “exploded” Maildir-like storage format with a single message per file.

One implication of Spotlight’s file-centricity is that its ability to search “email” might not apply to clients other than Apple Mail — it’s the fact that the new Tiger version of Mail stores each message as a separate file that allows Spotlight to effectively return individual mail messages as search results. No other major mail client uses a one-message-per-file storage format.

Spotlight’s full-text search is outsourced to SearchKit, which will be considerably faster in Tiger (“3x indexing, 20x incremental search” over Panther). So, Spotlight has three places to look for information about files: its own hand-tuned substring-matching metadata store (built by Giampaolo, not part of Core Data or anything else), Carbon’s HFS+ catalog calls (so Spotlight will respond to searches for type and creator), and SearchKit’s full-text index.

Both metadata collection and full-text indexing depend on cooperating per-file-format Importers, either written by Apple or by third parties. Like Google, no matter how much text an Importer provides, Spotlight only cares about the first 100K of raw text. Importers are fired on every file the moment it is created, saved, changed, or moved, including when files are made available through a newly mounted drive. Performance is said to be excellent in every case except network-mounted home directories, which are bedeviling on several levels and on which they’re still working.

It’s through the default set of Importers that Spotlight is able to index and search format-specific metadata, such as the ID3 tags in MP3 files. What’s cool about this architecture is that Spotlight’s indexes will thus stay up-to-date automatically. All you need to do is save, move, or copy a file, and Spotlight’s metadata and content indexes will note the changes on-the-fly. Compare and contrast to the full-content file searching previously provided via Sherlock, which required periodic monolithic re-indexing of the content of your drives.

Posted by DeLong at July 12, 2004 05:02 PM | TrackBack | | Other weblogs commenting on this post

Comments

[Smartass moment]
Is that WWDC-AM (1260 khz) or WWDC-FM (101.1 Mhz)?
[/Smartass moment]

Sorry, couldn't help myself. I used to listen to the Washington Senators on the aforementioned WWDC-AM. (Smartass moment, senior moment, what's the difference?)

Carry on.

Posted by: RT on July 12, 2004 06:16 PM

____

RexxMail is another email client that stores each message as a seperate file. It is open source but the only port that know of is for EcomStation or OS/2.

RexxMail

Article from Dutch users
http://www.os2ug.be/artikels/rexxmail.html

Authors site
http://www.degeus.com/rexx/rexxmail_nederlands.html
http://www.degeus.com/rexx/rexxmail_english.html

eComStation
www.ecomstation.com

Posted by: george on July 12, 2004 06:23 PM

____

Please with this hype: MH has used single-file format for ever. This is yet another instance of Apple taking the fittest of existing formats/projects/code for Darwin.... Re: the point of their new os....

See the LWN commentary on mail clients for more info.

http://lwn.net/Articles/91308/

Posted by: B. Koz on July 12, 2004 09:00 PM

____

So, they've finally done it!

I worked with dbg and was part of the Be team for several years, and this sounds like an excellent improvement over what we had with the BeOS.

Be's native file format was one-file-per-message, with the mail_daemon (which picked up the mail via POP3) responsible for tagging metadata onto the files as attributes (for some reason, I always think of it as gluing attributes to the sides of files, but that's just a colorful and useful analogy).

The Be filesystem was very cool in that not only was there metadata, but it was indexable. This meant that the mail_daemon just created files and you could get an Inbox view just by doing a query for new mail.

Anyway, this is a logical extension: they've abstrated the attribute creation and created a way to register them to the system. I assume that not every Importer is triggered on every file, which could really slow things down: I hope there's a filter based on extension or other metadata.

Posted by: Adam on July 12, 2004 09:18 PM

____

Ha! I went to school with Dominic! He was always cool, and rumor had it that he had cracked most of the systems on campus, but wasn't doing anything bad with his secret knowledge.

He introduced me to Jean-Louis Gassee at MacWorld Boston in 1997.

Posted by: josh brandt on July 13, 2004 05:59 AM

____

Oh, joy. Apple has made a very, very smart move. The Be filing system was very much ahead of its time, and I suspect that its translation to OS X may produce better results than the WinFS component promised for Longhorn.

The traditional problem with Maildir vs. mbox storage has been one of speed-of-access. The mbox format is relatively portable, though not robust -- if anyone's ever suffered from a large, corrupted mbox file, they'll know just what kind of anguish it causes -- but it's fast to process.

( Old discussion on formats here:
http://slashdot.org/askslashdot/01/01/27/0138202.shtml
though the canonical discussion is Jamie Zawinski's here:
http://www.jwz.org/doc/mailsum.html )

Processor speeds and memory increases, though, have taken away some of the traditional performance hit associated with Maildir. And to have a seamless mail/document search driven by HFS+ metadata has me drooling just thinking about it.

Posted by: nick on July 14, 2004 09:30 PM

____

Nice site Ever wonder how cigarettes are made? These are then cut to proper length for cigarettes. Filter-tipped brands have a double-length filter inserted between every two cigarettes; filters are sealed to the rods and then cut in half to produce two cigarettes, each one having a filter on one end.

Posted by: cigarette on August 3, 2004 06:39 AM

____

interesting idea

Posted by: Adam Smith (Hotel Font D' Argent Manager) on August 3, 2004 01:02 PM

____

cool

Posted by: Adam Smith (Apartments Chinasol Manager) on August 3, 2004 01:02 PM

____

Most of them smoked cigarettes Save money on cheap cigarettes Cigarette smoking has been buy cigarettes. order cigarettes the most popular method of taking offer discount Camel cigarettes nicotine the year a report offer discount marlboro cigarettes that concluded that buy cigarettes. order cigarettes cigarettes and other Save money on cheap cigarettes forms of tobacco smoking cigs for all the Buy cigarettes and pack of smokes for you rolling tobacco would you like cigarettes with Free delivery of cigarettes years of smoking can cause cigarettes Chesterfield cigarettes bar talking with people Lucky Strike cigarettes Marlboro

Posted by: cigarettes on August 4, 2004 07:44 AM

____

nice site i really like it http://www.a1-cheap-cigarettes.com

Posted by: cheap cigarettes on August 5, 2004 08:36 AM

____

nice site i really like it http://www.my-discount-cigarettes.com

Posted by: discount cigarettes on August 6, 2004 06:32 AM

____

7804 You can buy viagra from this site :http://www.ed.greatnow.com

Posted by: Viagra on August 7, 2004 05:16 PM

____

300 Why is Texas holdem so darn popular all the sudden?