Apple-Newton Handwriting Recognition

Despite the abysmal recognition accuracy in the first generation Newton, most Newton afficianados or people interested in handwriting recognition will tell you that the second generation, "Print Recognizer" in Newton OS 2.x was a vast improvement, offering fast and surprisingly accurate recognition. Unlike the first generation software, this second generation recognition engine was developed in-house at Apple, in the Advanced Technology Group (ATG), later (and briefly) renamed the Apple Research Laboratories (ARL). I served as Technical Lead for the project, and together with a core team of three Apple engineers and two contractors, plus a host of other contributors (most of whom are listed in the slides mentioned below), we managed to produce what many have called the first genuinely usable handwriting recognition system. The technical papers, articles, and slides below document a lot of the key technological hurdles that were overcome and the innovations that were made in order to make this possible.

The core recognition technology from the Newton has gained a new lease on life in the Jaguar release of Mac OS X (10.2). Together with a different team of engineers I have helped integrate handwriting recognition into Mac OS X in such a way that it just works with all existing apps; i.e., applications are not required to rev in order to support ink and the routine input of text by a pen and graphics tablet. This technology has been dubbed "Inkwell". (Partly it just seemed like a good name, plus I have a long-standing fondness for the Fleischer Brothers' animations, including their "Out of the Inkwell" series.) The Apple Computer page on Inkwell is (well, was) here: http://www.apple.com/macosx/jaguar/inkwell.html

For a silly paragraph concocted entirely out of words in the original Newton's limited dictionaries (for benchmarking our recognizer against the old one), check out Test Drive1.

A favorite story from the early days, that still makes me proud, was when Yann LeCun (who had led the development of a handwriting recognizer at AT&T Bell Labs, and is one of the most well known and respected names in the field of neural networks) and I got to talking at NIPS ('96, I think) and decided to see whose recognizer was better. We decided we would pick some random paragraph of text, and both of us would write that paragraph on our own devices, using our own software. We felt that we each knew our own system best, and also recognized that the correction mechanism could be almost as important as the quality of the recognition, so we agreed to write the text and correct it until it was 100% correct, and time how long it took each of us, shorter being better. Our handwriting recognizer, running on the Newton, handily beat Yann's by a significant margin!

For detailed technical info, please refer to:

[Note: there was an extended abstract for the IWFHR5 workshop here for a while, but it was removed when we withdrew our paper from that conference.]

Though many people contributed to this effort, the core group consisted of: