The Magic Keyboard

2019-10-25T12:56:30+09:00

What if your keyboard had access to your memories? What if it could complete your thoughts, not based on what you'd previously typed or learning generic responses or dumb rules, but based on a personal database of places, books, web pages, and notes of all kinds?

A keyboard should help you go places.

To give an idea of what this would look like, when you type in Japanese, what you type on the keyboard isn't exactly what shows up on the screen. Instead you type a romanized version of the text you want a few words at a time. After each span you lock it in, and if you're converting to Chinese characters you choose one of a few alternatives from a drop-down list. In some input systems, short definitions may be shown next to the current selection.

An example of me using Google's Japanese IME.

Autocomplete in Japanese is based on a dictionary supplied with your operating system, plus a user dictionary. If you frequently have to type something not in the dictionary you can register it in a personal dictionary to have it come out just like any other word. You can also get creative and register things that aren't normal words, like converting "home" to "123 Mt. Pleasant Drive".

This mediated input opened up potential for cleverness in software keyboards, acronymically known as Input Method Engines (IMEs). Introducing emoji was just a matter of adding them to a font and the dictionary - you didn't have to tell people how to use them, as they'd just be another completion option when you typed "beer" 🍻 or "love" 😍 or what have you.

Now we all have smartphones, and we all have mediated input, and emoji are everyhwere. Hooray. 🎊

But what if mediated input were smarter? What if it were more personal?

What if you could type "Here's that gyoza place I went to last week" and autocomplete a map link?

A mockup of a magic keyboard search interface.

"Here's that book I mentioned at lunch." "This is that game with indirect controls." "This is the page for that meetup you should go to." "This is where you can get tickets for that museum show."

Of course you can already share this information thanks to the universal power of links, but what if it didn't need a context switch? What if taking notes on things made it easier for you to share them with people? (This is, of course, part of the dream of the Memex.)

Five years ago when I started work on Deltos for personal note taking, one of my goals was to use the notes in Deltos as fodder for a magic keyboard, but that ended up falling by the wayside. It turns out working with IMEs is difficult - every operating system has a completely different way of handling them, and they often aren't well documented.

In particular, IMEs often work on the principle that for any given output, the input is clearly defined. In a dictionary file these are just short bits of text. But if you can search through all your notes, then inputs aren't well defined, and just updating the dictionary isn't enough - you need search functionality.

One idea is to avoid using the IME interface directly. A small application, just a search box where a selection is copied to the clipboard, would be enough to realize the main goal of the magic keyboard, while still minimizing context switching. It would also make it easier to create a cross-platform app. Maybe that's good enough.

The other problem is that most people don't have an explicit personal database, just a bunch of personal data locked up in silos. Fixing that is another and honestly bigger step, but it can be handled one person at a time.

I don't think the magic keyboard exists yet. But I hope it will soon. Ψ