April 23, 2003

Things That Make You Go "Hmmm...."

If I were going to talk about advances in human-computer interfaces, I certainly would not talk about how great Microsoft is: about how "...[t]he move to a true advance in user interface is long overdue and, with Microsoft?s success with the thumb mouse, we just might soon be seeing the software-only company leading the way." I would talk about David Gelernter's Lifestreams or about AquaMinds's Notetaker or I would find something else--find some other intriguing, useful, actually-shipping products to praise.

But, then, I'm not consulting for Microsoft.

I'm afraid I'm going to have to take everything in AskTog with several grains of salt in the future...


While the battle lines of pro- and anti-Aqua have been drawn in blood, no one seems to be looking at the real problem, and that is that Apple, in OS-X, is doing nothing other than running a 10-year-old and 20-year-old interface together at high speed.

Apple has one advantage shared with Microsoft—control of the interface—and one advantage that is unique—control of the hardware. Apple seems unwilling or unable to take advantage of either.

The next big movement in interface technology, just as with the last, will result from a marriage of hardware and software changes. The current interface resulted from two important hardware innovations, the bit-mapped display and the mouse. Both were achieved by 1965, with the first successful commercial GUI arriving some 19 years later.

In the ensuing years, little has happened to the displays, except they’ve gotten bigger (and, in the case of palmtops, smaller). Lots has happened to input devices, but you wouldn’t know it looking at a Macintosh. The only changes to input technology that have taken place recently around Apple are the crippling of the keyboard and the farcical round mouse that has made Apple the laughingstock of the industry.

Ironically, the only real advance in input technology has come from the software-only company, Microsoft, with their introduction of the wheel mouse. (We had a similar mouse prototyped at Apple more than ten years ago. Didn’t seem worth marketing it, though. After all, the standard one-button mouse is perfect, isn’t it?)

Where could Apple be next year if it were truly interested in developing a next-generation interface? Let’s take a look.


Easy wins. Anyone who uses the mouse has faced the frustration of having to repeatedly press the Return key while flipping from dialog to dialog. Why isn’t there a Return key on the left side of the keyboard, so we wouldn’t have to keep dropping the mouse? Why aren’t there, likewise, an extra Del and Delete key? These could all occupy a new column on the left edge of the keyboard.


The clutter of words, icons, and buttons that obscure our screens today are the result of the severely limited vocabulary of the mouse. The only word it knows is "click," so you have to find an instance of the word you want to convey to the computer, then say "click" while you hover over it.

Consider how many words you would have to paste up in your office to be able to carry out a conversation using this method. And consider how slow and frustrating that conversation would be.

Gesture is the obvious next step in input. Companies, such as Palm Computing, with far weaker hardware than Apple have been using it now for years. Why is it missing from the Macintosh, and why is it missing from OS-X? Apple brags of having the hardware advantage; it’s time they used it.

OS-X should ship with a tablet or horizontal flat panel that will enable gestural input. The mouse is dead; it’s time for a change.

Integrated Phone/Voice Recognition

By having integrated telephony, not only can users eliminate one source of clutter on their physical desktop—the phone—they will suddenly start wearing a lightweight headset. That headset is the entry into vocal interaction. Where gesture leaves off, spoken commands can begin. Is voice-recognition up to the task? It is barely up to it for continuous speech recognition, but most of what we need doesn’t require either continuous speech or a rich vocabulary. Most of what we need to do involves issuing one of a few dozen commands in any one context. For example, in the context of Photoshop, the user might speak anything on a menu or recite the name of a tool. Several dozen words define the entire vocabulary.

Imagine the flow that would occur were you able to just work, freed from the constant task of changing or modifying the action of tools.

Reader Erik Neu sees an additional synergy:

In my observation, the average (or even the better-than-average) office worker does not master even the more basic features of their telephone (like conference calling and transferring) because of the clunky, non-intuitive and non-standard interface. It would be SOOOO much better and easier to invoke these features from a PC interface.

Screen Objects

Apple is feeling pressure to eliminate the spatial finder not because someone repealed people’s natural affinity for spatial interfaces, but because the 1984 Macintosh model has run out of steam. Why? Because, in those days of 128K machines with a single floppy and no hard disk, no one thought to provide complex objects. Instead, people got documents, applications, and folders. Can you imagine an office that only had documents and folders? Where would you sit? Where would you store things? It is little wonder that people can become lost.

What are needed are a series of new, more powerful objects.


Apple holds a patent on this one. Developed by Gitta Salomon and her team close to a decade ago, a pile is a loose grouping of documents. Its visual representation is an overlay of all the documents within the pile, one on top of the other, rotated to varying degrees. In other words, a pile on the desktop looked just like a pile on your real desktop.

To view the documents within the pile, you clicked on the top of the pile and drew the mouse up the screen. As you did so, one document after another would appear as a thumbnail next to the pile. When you found the one you were looking for, you would release the mouse and the current document would open.

Piles, unlike today’s folders, gave you a lot of hints as to their contents. You could judge the number of documents in the pile by its height. You could judge its composition very rapidly by pulling through it.


Apple has one new kind of folder in OS-X, the Package, which, in essence, enables an application to contain within it all the little tiny files necessary for it to run.

However, regular files appear to continue to be as weak and ineffective as ever. I should be able to tell, by looking at the outside of a file, how much material is within, how old the folder is, and how long it has been since I’ve opened it. I’m not talking about surrounding it with text; I’m talking about using that 24-bit color space to add visual attributes. For example, the more the folder contains, the thicker it should appear. The older it is, the deeper the color should become, with age cracks appearing after several years. The time since I’ve opened it could be represented by cobwebs or dustiness.

File Cabinets

Many of the larger projects I work on could take good advantage of a file cabinet, a sort of project-level structure. Different drawers might hold email vs. graphic designs vs. specifications vs. prototypes. Or I might divide it by the people working on it, with a drawer for each. By displaying labels on the drawers when there are few or tool-tips when there are many, people could access the sets of folders within. It would knock out one layer of hierarchy by bringing the second layer to the surface.


I proposed scrapbooks in Tog on Software Design as a handy place to organize the materials for projects, etc. Building on the concept of piles, scrapbooks offer the ability to casually organize incoming material and to provide notes and other context-enhancements.

As outlined in Tog on Software Design, a scrapbook can be:

• created by placing items on top of one another

• thumbnailed

• flipped through in its thumbnail form, revealing individual documents

• opened to reveal a single page

• unfolded to reveal any number of pages at once

• browsed through like a book

• merged with another scrapbook

• shared synchronously or asynchronously


A scrapbook can contain:

• whole documents

• fragments, such as clippings or scraps

• annotations (text, graphics, other types?)

• icons and thumbnails

• table of contents

• index

• other scrapbooks

• tool sets

• editing history

• items from other scrapbooks

• pointers (references) to items not in the scrapbook 


Notebooks would be a step up from a scrapbook, perhaps occurring as a project nears fruition.


People should also be first-class objects in a modern interface. I should be able to drag a document onto an icon-sized photo of a colleague and have it automatically sent to the default address. (Of course, I could option drag a document to it to reveal all addresses, or I could just double click it to do the same.)

Active Documents

Microsoft has been inching closer and closer to compound documents, and Apple itself took a brief run at it several years ago, but it is time for the real thing. The large-corporation monopoly will not be broken in the computer industry until we have the kind of cooperative development in the end-user space that we are seeing in the Linux system software space. This calls for a document platform on which different developers can supply various engines for text editing, for graphics editing, for spell checking, for "spreadsheeting" (to coin a phrase), etc., etc., etc.


Up until now, I've been discussing objects that might appear on the desktop. The desktop itself, however, has shrivelled a bit. We need a new, larger desktop.

Sun has done this by introducing a "virtual desktop" that is several times bigger than the portion visible on the screen, along with a little map viewer that lets you see where you are and drag to where you want to go. In my experience, users find this troubling.

Quicken has what I believe to be a much more successful model, with a series of master tabs, always visible across the top, pulling up one environment or another. The user easily moves from the checking/savings account environment to the investment environment to the report environment. Quicken was also clever enough to enable the user to open any account in any environment, so if you need to compare stock purchases to checks written for same, you can open your checking account in the investment environment, etc.

I would love to bring up the AskTog environment every month and find all my windows and tools just lying there waiting for me.


This is not an exhaustive list of objects and capabilities that should be incorporated in the interface. These are just a few of the more obvious.

OS-X may work out and it may not, but let’s not confuse it with an advance in user interface. At best, it will catch up with itself, should they decide not to cripple the Finder.

The move to a true advance in user interface is long overdue and, with Microsoft’s success with the thumb mouse, we just might soon be seeing the software-only company leading the way.

Posted by DeLong at April 23, 2003 08:42 PM | TrackBack


Heh. I think Apple had better rename what they're currently calling 'piles' when they try to sell software making use of the concept into the UK.


Posted by: Tom Runnacles on April 23, 2003 08:59 PM


Posted by: Brad DeLong on April 23, 2003 09:13 PM

What, you think Tog's been Scobleized or something?


Posted by: Kynn Bartlett on April 23, 2003 09:30 PM

What, you think Tog's been Scobleized or something?


Posted by: Kynn Bartlett on April 23, 2003 09:32 PM

Heh. OneNote is interesting but doesn't deliver, at least in its beta incarnation. Mitch Kapor's 'Chandler' may reincarnate the organisational principle behind Lotus Agenda, but the 0.1 alpha doesn't do that right now.

What's needed, of course, is a kind of emergent organising interface. Not necessarily metadata-based (like Microsoft's proposed Longhorn file system) but something closer to MIT's Remembrance Agent. There are obviously data-types that have built-in organising elements (dates, names, phone numbers) and you can use these types as 'pivots' for datasets, but we're still a bit away from getting something that does for your hard drive what Google does for the Web.

Posted by: nick sweeney on April 23, 2003 10:30 PM

If you like NoteTaker, also check out NoteBook by Circus Ponies (www.circusponies.com, I think).

They're very similar applications for OS X, which would be expected since the people behind the two applications worked together on an identical application for NeXT computers, which was called NoteBook.

It's an interesting competitive dynamic, to say the least.

The CircusPonies NoteBook people are offering a $10 competitive upgrade price for users of NoteTaker.

Posted by: Jon h on April 24, 2003 11:13 AM

A critique of Apple can begin and end with the one-button mouse. The two-button-plus-wheel mouse is easy to use and greatly enhances usability. However, Apple sticks with the one-button mouse because THEY PUT STYLE OVER SUBSTANCE. Their unwillingness to move to a modern mouse is a damning indictment of their current design philosophy.

Other evidence of this can be seen in their decision to roll out the clear plastic, gelcap-shaped mouse. You know, the one that defies all that we have learned about ergonomics and carpal tunnel in the last decade?

I'd balance this with a concession that the iPod is very cool. Perhaps Apple's real problem is an inability to update their initial bold design choices. Arrogance? Design religion? Anyway, if this theory is correct then we can be confident that the iPod will be out-iPod'ed within a few years.

Posted by: Magik Johnson on April 24, 2003 11:21 AM

"The two-button-plus-wheel mouse is easy to use and greatly enhances usability."

And they work fine with OS X, with right button and wheel support.

Posted by: Jon h on April 24, 2003 11:52 AM

Very Interesting.

But I doubt that voice-based interfaces will be with us anytime soon. The reason is simple: most computing is done at work, and most office workers work in common rooms, seperated from their co-workers only by thin cubicle walls, if at all. Even relatively high level executives work in cube farms in many progressive companies these days (For example, at Chicago-based investment research firm Morningstar, the CEO has a cubicle that is the same size as the customer service workers'), and its the most progressive companies where people are most likely to need high end computing applications.

Having a large room full of people barking out short, staccato commands to their computers would be maddening. Its hard enough to work in a cube farm with people talking on the phone all around you, but at least then their speach is natural, and somewhat soothing.

Further, I suspect that even those with private offices and those working at home would feel very self-conscious about talking to their machines. Our social habits are hard to break, and there are few social habits as strong for most people as the habit of being quiet when there's nobody around to talk to.

That being said, the things I could make Excel do if it would just listen to me, dammit... : )

Posted by: sd on April 24, 2003 11:59 AM

Yes, it's obvious that Tog has an axe to grind, given that he's Apple's consumate disgruntled former employee. He does have a few legitimate concerns-- such as, yes, wheel mice are great.

On the other hand, shut the hell up about the 1-button mice. A, you can just plug your preferred third-party mouse in. B, Have you tried Ctrl-Click, Cmd-Click, Shift-click, or Click-hold? No? The mouse button is like a key. It accepts modifiers like a key.

People disliked the round mouse-- who were absolutely dedicated to their old mice. If you used the round mouse like it was supposed to be used, squeezing rather than pushing it, it was fine.

And the worst, most annoying criticims, were those of the ibooks and iMacs: "it's a toy." "it's for girls." Shut up! Are you afraid of this or something? It looks easy to use so it's suddenly upsetting to you? The Real Men actually complained about the sissified UI. That's just not a legitimate criticism. It's stupid, reactionary, and fearful.

As to actual innovation, I think search/indexing is the way to go, and use something based on folders. "Filing Cabinets" my ass-- that's called a folder, with other folders in it.

And most of the technology described ("Piles" for example, and let me remind you that's a word meaning hemmorhoid) without a bold new metaphor. For example, I'd like to see something for documents that is equivalent to the 'vFolders' in the Ximian Evolution (ximian.com/products/evolution) email tool. They're like a fast saved search of messages, so you can keep all your messages stored in mailboxes based on who sent them, but have a vFolder based on the subject, or whether you've marked a message as important, or whether it arrived in the past 24 hours.

Just a cross between a search and a folder. A search folder, a virtual folder, a metafolder, a saved search. A set of symlinks based on slocatedb and cron, for you unix types. That'd be cool. And very within reach.

Posted by: verbal on April 25, 2003 12:34 PM

Tog does seem to have gotten a bit...shrill...lately. (Can I use that term for a non-NYT-columnist?)

The round mouse was in fact hideous, and I'm not that thrilled with the new finder. But the notion that you would want to put a return key on the left so you could keep a hand on your mouse...that makes no sense for anybody who types well. The big advance of OS X for me is precisely that I can go hours or even days without using the mousel...at which point I don't exactly care how many buttons it has.

But the one bizarre and basically unfogivable oversight in this column is somehow ignoring the IBM trackpoint. Some people do hate it while other like it, but it was a truly new development in human interfaces.

Posted by: Jonathan King on April 25, 2003 12:43 PM

Re: vFolders

That sounds kinda like a database View from SQL. It looks and acts like a regular table, but really its contents are computed on the fly by evaluating a particular SQL query.

Re: the iMac mouse - i thought it was awful. Too small and too light. I use a logitech 2 button wheel mouse.

Re: the left-side return, I'd like that.

In fact, I'd like all the extra keys on the right side
moved to the left. So the numeric keypad, arrow keys,
etc would be on the left.

I'm right-handed. I have my mouse to the right of the keyboard. I also like to have a pad of paper handy for
writing notes. Usually, this means putting the paper
between keyboard and mouse, which puts the mouse
pretty far out to the right. By contrast, a mouse and a pad of paper work pretty well with my iBook's compact keyboard.

(Strictly speaking, the amount of space required for keyboard, paper, and mouse would be the same if the numeric keypad was moved. But if the items are arranged to be centered on the space bar, then the paper and mouse would be closer. If that makes any sense.)

Alternatively, I'd like someone to invent a little stand for a pad of paper that would put the paper above the mousepad, with room underneath for mousing.

Posted by: Jon h on April 25, 2003 03:49 PM
Post a comment