Koko Chapter

Koko's Mac II: A Preliminary Report

Kim Rose, Mike Clark, Larry Yaeger, Tom Ferrara and Ann Marion

Vivarium Program
Apple Computer, Inc.
Project Koko

Since 1987, Apple Computer's Vivarium Program has been working with the Gorilla
Foundation to provide Koko with a voice. Apple has designed and built a
special Mac II enclosure and written software which will allow Koko to activate
a voice by touching icons on the screen. With this tool, the Gorilla
Foundation can modify the information presented to Koko, and collect and
analyze Koko's use of the system to gain further insights into language
acquisition by higher-order primates.

Koko, the Gorilla Foundation, and the desire for a voice

Since she was one year old (1972), Koko has been learning a version of American
Sign Language (Ameslan), and now has a vocabulary of over 600 words. The
Gorilla Foundation was formed to continue this training and the related
research, and Project Koko is now the longest uninterrupted study of ape
language abilities ever undertaken. The research findings of the Foundation
offer strong support for the notion that, when immersed in an environment rich
in interactions with speaking humans, gorillas can not only comprehend and
react to spoken English, but can also invent new gestures, translate from
English to Ameslan, and converse at a high level of abstraction. The
dedication of researchers Dr. Francine Patterson and Dr. Ron Cohn is no less
impressive - since lowland gorillas have a life span of more than 50 years, the
researchers have basically committed the rest of their lives to studying Koko
and a male gorilla, Michael.

At one time, when Project Koko was still on the Stanford campus, Koko had
access to a voice synthesis unit which she used to enhance her communication
skills. However, when the Foundation moved to Woodside Koko lost access to
this device. The loss was traumatizing - when Koko was asked what she wanted
for her birthday, she signed, Voice! The researchers were also disappointed to
lose this basic tool, and have been seeking a way to reinstate this capability.
With the system we've designed, Koko's researchers from the Gorilla Foundation
can display icons for Koko to select, manipulate the icons available to her,
and record both her and the researchers' use of their systems. This last
function will give the Gorilla Foundation its first tool to completely record
and analyze Koko's communications. Dr. Francine Patterson (Executive Director
of the Gorilla Foundation) serves on Vivariums Advisory Board, and is providing
us with regular feedback on Koko's use of the system. Dr. Patterson believes
that by enabling Koko to use spoken language, we can break new ground in
interspecies communication.

Koko's Macintosh

Kokos uses a standard Mac II enclosed in a special gorilla-proof housing. The
enclosure is constructed of 1/2" polycarbonate and is bolted to the floor of
Koko's trailer. Access to the computer is provided through a special touch
screen designed to withstand the 2,000 pounds of force an excited gorilla can
potentially generate. The computer has two video cards and is shared by Koko
and the researcher.

What we hope to learn

Alan Kay notes that, "Since we're doing a project simulating animals, we should
have at least one animal on our advisory board!" We further expect to gain
some insights into what kind of iconic representation scheme will allow Koko to
do sophisticated retrievals. Will she be able to use the interface we've
designed to navigate successfully? Are there any ways in which we can
extrapolate from the results of this experiment to make systems more accessible
to novice human users?

A continuing exposure to Koko's use of language can provide insight into the
nature of her cognition and brain function; we expect this to continue to be of
interest and value, especially once we provide the Foundation with copies of
some of the time/date-stamped files of her speech and the researchers' notes.
The nature of the world model that Koko has built in her brain/mind is very
much of interest; presumably, use of language requires some internal world
model. Larry Yaeger, Principle Engineer on the Vivarium, has embarked on a
project to build simulated critters that build and use internal models of the
world. Larry is basing his work on neurophsyiological data and neural network
models, but we believe that phenominological input regarding this issue from
natural sources (such as Koko) will be of great value.

What we have already learned

In the process of developing the tools, Vivarium engineers Mike Clark (Systems
and Electronics), Tom Ferrara (Mechanical) and Larry Yaeger (Software) have had
a number of challenging experiences. Above all, the personal experience of
participating in this research - meeting Koko and communicating with her - has
been truly awesome.

Systems and electronics

Mike Clark has spearheaded system design since the project's inception. The
Gorilla Foundation had an existing proposal to develop a large keyboard for
Koko, but during initial discussions Mike Clark determined that everything that
the Gorilla Foundation wanted to do (and more) could be done with a Mac II, a
large touch screen, and a HyperCard-style button interface. At the first
meeting Mike demonstrated a HyperCard stack that spoke all of the words from
the Koko's Kitten book, spoken by his niece and digitized by his son.

Mechanical engineering

The most important consideration in all mechanical design aspects was for the
safety of the gorilla. A typical gorilla response to anger or excitement is to
run full speed and backhand the object of the anger. For Koko, this means a
260-pound animal running at about 20 miles per hour, swinging her arm with a
force comparable to a 10-pound shotput traveling at 100 miles per hour.
Accordingly, Tom Ferrara faced a significant design challenge in developing the
workstation - a Mac II computer, a 19" Sony monitor, and a touch screen - into
a package which could withstand these extreme forces. Failure of the physical
system could only be tolerated if it did not endanger Koko in any way.
The most difficult design consideration was the part of the system with which
the gorilla actually comes in contact: the touch screen. Most touch screens
are thin, to minimize the visual parallax between the image on the screen and
the point of contact. But we needed a thick piece of glass to withstand Koko's
potential force. Mike spent over a month talking to MicroTouch, the Gorilla
Foundation, and the Los Angeles Zoo to determine the right combination of
materials. A standard MicroTouch capacitive screen optically bonded to a
one-inch thick piece of tempered glass was finally selected. It is interesting
that the parallax through a one-inch piece of tempered glass is not as bad as
one might predict when operating the system from the floor. Behind the touch
screen is mounted a standard 19" Sony Trinitron from SuperMac Technology.
Design of the computer's enclosure proved almost as challenging. We needed an
enclosure which would withstand Koko's excitement and absorb the shock of any
blows she might deliver. After debating and testing a number of designs, Tom
finally built the enclosure of an inner framework made from 1" x 2" solid
6064-T6 aluminum, covered by a 3/8" polycarbonate sheet. The Monitor is
mounted on a sliding assembly that is dampened with gas struts with 3" of
travel. This provides shock absorption in the event that Koko should throw her
weight into the screen area. The final unit as delivered is 28 5/8" tall x 21
1/4" wide x 39 1/4" long. Passive ventilation is provided by slots which are
designed to channel any foreign materials (bananas, feces, etc.) away from the
CPU. (For more information on the design's evolution, see Appendix 1.)
Koko has a standard Mac II CPU consisting of 5 megabytes of memory, a
40-megabyte hard disk, and a SuperMac Spectrum/8 color card, connected through
the floor of Koko's trailer to the researcher's monitor and keyboard.
In the process of designing the physical system, Tom Ferrara had to concern
himself with both durability and ergonomics. He felt that the project helped
him to gain a better understanding of human-computer interfaces, especially in
terms of accommodating users who may not have the physical ability to use a
keyboard or manipulate a mouse, but who can point.

Software user interface

The Koko project inspired Mike Clark to learn SuperCard so that he could create
a large-screen color interface. So far, three applications have been developed
to familiarize Koko with the touch screen interface. The first is four
digitized animal pictures that make the animals sound when touched. The second
shows a picture for each letter of the alphabet (developed by the Gorilla
foundation) and speaks the name of the picture when it is pressed. The final
application is called KokoPaint. It is a simplified color painting program
which allows Koko to select from a few colors and finger-paint with them.

Linguistic research

Larry Yaeger experimented with HyperCard and SuperCard as candidates for
implementation of the primary research tool, but he ultimately opted to write a
custom Mac application for Koko entitled "Lingo". The interface presented to
Koko is simple, consisting primarily of uniformly shaped, regularly spaced
buttons - rectangular regions on her touch-screen display - that, when pressed,
cause the playback of a sampled-sound resource corresponding to the selected
button. Each button corresponds to a single word and may display text, a
full-color picture, or both. Sampled sound was selected over phonemic speech
generation early on, primarily for the improved clarity of speech. This
rapidly led to a discussion of who would serve best as Koko's voice, and we
decided to find a way to let her make the selection herself. The voices of
four human females were recorded with MacRecorder and later played for Koko,
who ultimately selected one of them as her own.

Anticipating a maximum vocabulary of between 500 and 2000 words (Koko's
common-use signs and the total number of uniquely identifiable signs she has
evidenced, respectively), specific, strategically placed buttons on the screen
were set aside to perform a forward- or backward-paging function, hopefully
allowing Koko to flip between screens of words. Additionally, at the request
of the Gorilla Foundation, a sentence buttonwas incorporated that accumulates
the individual words as they are pressed and when pressed itself produces a
sentence consisting of these words played rapidly in sequence.
The interface just described is presented to Koko on a "Subject screen" which
she uses to make her word selections. A separate "Researcher screen" displays
Koko's word selections and permits the recording of contextual notes by
observers. The Researcher screen is a standard Apple RGB monitor fed by a
cable out of the bottom of Koko's workstation. The keyboard cable also extends
out the bottom, and the keyboard, mouse, and second monitor form a second
workstation for the researcher.

Through various menu options, the Researcher can employ a number of tools for
laying out and manipulating the Subject screen and for saving and restoring
these layouts in a convenient fashion. Several characteristics of the
interface could not be predetermined - for instance, most useful button size,
optimal number of buttons per screen, the relative usefulness of text versus
images for the buttons, the desirability of playing a sound immediately upon
each buttons press versus delaying until the sentence button is pressed, and so
on. The solution was to make such features easily modifiable through menu
selections and simple dialogs from the Researcher screen. Tools for editing
the words, sounds, and images associated with the various buttons are provided
directly within Lingo to help the researchers design and experiment with the
Subject screen layout. Alternate layout configurations may be saved and opened
in the usual Macintosh manner through the File menu.

The system displays the subject's and researcher's entries and voicing Koko's
selections and records them in a pair of archival data files - the Subject
Archive file and the Researcher Archive file - for subsequent linguistic and
behavioral analyses. Every utterance (button press) made by Koko is separately
date/time-stamped; the researcher's notes are date/time-stamped and archived
once per line. The configuration of the Subject screen along with any changes
made to it is also archived in simple TEXT files which may be examined in any
word processor. Fresh transcript files are automatically opened and closed on
a daily basis, or whenever a new configuration is opened.

What has the Gorilla Foundation learned?

At the time of this writing, Koko has not yet used her complete workstation;
Ron and Penny are hard at work gathering all the images and sounds for the new
workstation software, and the final screen layout is being designed. Koko has
only used the stand-alone Mac II for voice selection. Even with only two brief
sessions totaling less than one hour with the Macintosh, however, significant
results have already been achieved. Through the voice-selection exercises,
Koko was able to solve a complex conceptual problem presented to her in spoken
English by a computer, without prior training or the aid of visual markers of
any kind. The accomplishment of this task illustrates that a gorilla has
cognitive capacities for:

1. comprehension of spoken language: Koko responded to the recorded statements
and grasped the task requirements.

2. self-concept: Koko employed the sign "myself" to refer to one of the voices
when it spoke both the query and concept sentences.

3. auditory discrimination of human voices: Koko responded differently to the
four voices.

Next phases of the computer-language project will allow Koko to use the voice
she has selected to create her own spoken statements [for further reading see
reference of G.F. Journal article].

What does the Gorilla Foundation hope to learn?

The first research project is to develop a user interface that will allow Koko
to navigate between screens. The entire vocabulary of over 600 words cannot
effectively be displayed for Koko on her screen all at the same time. We must
therefore design a way to represent to Koko where she is in relation to the
other things available to her and how to get to them. In addition, we need to
provide Koko with a way to indicate thought completion. After the user
interface issues have been resolved, then the main research task of
communications research can begin.

Once Penny and Ron saw the Koko's Macintosh, they began to see possibilities
for using it that extend beyond those originally imagined. They began
thinking about the possibility of empowering Koko to control her environment.
Could our engineers make it possible for the Mac to control the lighting in
Koko's room so that Koko could control lighting herself? Could she learn how
to operate the interface they designed? Could Koko control her curtains,
raising and lowering them at will? Currently she orders around researchers to
do these tasks for her - would she perform them herself if she could?

In conclusion. . .

Designing Koko's workstation and software has already led to some insights
regarding user interface issues in hardware and software design. And the
manner in which Koko now interacts with her Macintosh, its rather special
workstation, and its software will undoubtedly yield even greater insights into
these issues. But perhaps the most enlightening aspect of this work will be
the greater understanding we obtain of the nature of intelligence through this
ongoing study of language and its use. Through such studies of alternate
cognitive systems we should be able to develop a better understanding of human
cognition and communication. Perhaps we may then apply this understanding to
the development of functional artificial intelligence, or, at the very least,
come to know ourselves better. And if we are able to see the human species as
part of a continuum, rather than uniquely set apart, perhaps we will become
more aware of our place in the world's ecosystem - and the need to act
accordingly.

Appendix 1

A recap and timeline of the design and building phases for the workstation:
1st pass, February 1987:

o Screen set at 52-degree angle
o CPU located at bottom of enclosure parallel to the floor
o Overall height to center of screen 22 in.
o No open areas where foreign matter (gorilla poop) can be forced into
enclosure
2nd pass, August 1987:
o Screen set at 30 degree angle
o CPU located at top of enclosure parallel to floor
o Overall height to center of screen 17 in.
o Top surface of enclosure to be flat so Koko can sit on it - she weighs 270
lbs.
o Material to be 3/8" 6064-T6 aluminum with internal structural supports to
allow CPU mounting with isolation pads
o Front panel to be 3/8" lexan.
3rd pass, February 1988:
o Provided recess of 6" at bottom of front panel to allow Koko's legs to fit
underneath when she sits on the floor
o Redesigned enclosure to utilize thinner material and incorporate an inner
sub-structure which would add stability as well as allowing more efficient
design of isolation set-up inside
4th Pass, September 1988:
o Incorporated 3/8" plexiglass cover over the screen area to insure that she
could not put her hand through the monitor when not in use
o Redesigned front panel to accept either 17" or 19" display to be removable
from the front to provide easier access for service
o Changed cable ports from the back to the bottom of the unit so the cables
would be located underneath the raised floor of the trailer.
5th Pass, January 1989:
o Deleted the Plexiglass screen cover and designed front panel to accept a high
impact laminated glass touch screen
o Designed a filtration system utilizing open cell 1" thick foam. It will
require 1 piece at the bottom of the unit and another piece 1/2" thick at the
upper back.
o Two muffin fans will be used to circulate air through the filters and to
allow better cooling inside the enclosure.

Appendix 2: Gorilla Language Bibliography

1. Patterson, F.G. "The Gestures of a Gorilla: Language Acquisition in Another
Pongid." Brain and Language, 1978, 5, 72-97.
2. Patterson, F.G.P. "Human Communication with Gorillas." In G.B. Stone (Ed.),
In the Spirit of Enterprise. San Francisco: W.H. Freeman and Co., 1978.
3. Patterson, F.G. "Linguistic Capabilities of a Young Lowland Gorilla." In
F.C. Peng (Ed.), Sign Language and Language Acquisition in Man and Ape: New
Dimensions in Comparative Pedolinguistics. Boulder, Colorado: Westview
Press,1978. Reprinted in R.L. Schiefelbusch and J.H. Hollis (Eds.), Language
Intervention From Ape to Child, Baltimore, Maryland: University Park Press,
1979.
4. Patterson, F. "Conversations with a Gorilla." National Geographic, October,
1978,154 (4), 438- 465. Condensed in Reader's Digest, March,1979, 114, (683),
81-86.
5. Patterson, F.G. "Linguistic Capabilities of a Lowland Gorilla." Ph.D.
Dissertation, Stanford University, Psychology, 1979. University Microfilms
International, Ann Arbor, Ml 48106 (#79-172-69). Abstracted in Dissertation
Abstracts International, August 1979, 40-B, 2.
6. Patterson, F. "Comment on Terrace" (Letters), NYU Educational Quarterly,
1980, 11 (3), 33.
7. Patterson, F.G. "Creative and Innovative Uses of Language by a Gorilla: A
case study." In K.E. Nelson (Ed.), Children's Language, Vol. 2. New York:
Gardner Press, 1980.
8. Patterson, F. "Gorilla Talk: Comment on Monkey Business, " The New York
Review of Books, October 1980, 27(15), 45-46.
9. Patterson, F. "In Search of Man: Experiments in Primate Communication." The
Michigan Quarterly Review, Winter 1980,19 (1), 95-114.
10. Patterson, F. "Can an Ape Create a Sentence? Some Affirmative Evidence."
Science, 1981, 211, 86-87.
11 . Patterson, F. and Linden E. The Education of Koko. New York: Holt,
Rinehart and Winston, 1981.
12. Patterson, F. "Gorilla Warfare." American Psychological Association
Monitor, January 1981,12 (1),16 and 41.
13. Patterson, F. "More on Ape Talk." The New York Review of Books, April
1981, 28 (5), 43.
14. Patterson, F. "Generalized Language Ability in the Gorilla." Paper
presented at the annual meeting of the American Psychological Association, Los
Angeles, August 1981.
15. Patterson, F. "Gorilla Language Acquisition." National Geographic Society
Research Reports, Vol.17 (1976 Projects). Washington, D .C .: National
Geographic Society, 1984.
16. Patterson, F. Koko's Kitten. New York: Scholastic Books, 1985.
17. Patterson, F., Share, E. and Cornwall, C. "Interspecies Communication and
Conservation." Primate Conservation (IUCN/SSC), January 1985, 5, 39-40.
18. Patterson, F. "The Mind of the Gorilla: Conversation and Conservation." In
K. Benirschke (Ed.), Primates: The Road to Self-sustaining Populations. New
York: Springer-Verlag,1986.
19. Patterson, F., Tanner, J., and Mayer, N. "Pragmatic Analysis of Gorilla
Utterances: Early Communicative Development in the Gorilla Koko." Journal of
Pragmatics, 1988,12, (1), 35-55.
20. Patterson, F. Koko's Story. New York: Scholastic Books, 1987.
21. Patterson, F., Patterson, C. H., and Brentari, D.K. "Language in Child,
Chimp, and Gorilla." American Psychologist, March, 1987, 42 (3), 270-272.
22. Patterson, F. and Patterson, C.H. "Review of Ape Language: From
Conditioned Response to Symbol." American Journal of Psychology, 1988,101, (4)
(in press).

Other Publications of Interest

1 . Gorilla, a journal published semiannually by the Gorilla Foundation, Box
620-530, Woodside, CA. 94062.
2. Crail,Ted. Apetalk and Whalespeak. Los Angeles: J.P. Tarcher,1981.
3. Vessels, Jane. "Koko's Kitten." National Geographic, January, 1985. 167
(1), 110-113.

Audio-Visual References

1. Harrar, L. "Signs of the Apes, Songs of the Whales" (Videotape). Boston,
MA: WGBH Educational Foundation, Nova. Available through Vestron Video, Box
4000, Stamford, CT 06907.
2. Johnson, Anne C. "Are They Really Dumb? - Intelligence and Language Among
the Animals" (Filmstrip). 4 News Currents, Knowledge Unlimited 1984, Box 52,
Madison, WI 53701
3. Moses, Harry. "Talk to the Animals." A 1 2-minute 16 mm film produced for
60 Minutes CBS. Available through CRM McGraw-Hill Films, 110 Fifteenth Street,
Del Mar, CA 92014.
4. National Geographic Society. "Gorilla" (Videotape), 1986. Vestron Video,
Box 4000, Stamford, CT 06907.
5. Schroeder, Barbet. "Koko, a Talking Gorilla." (90-minute 16mm film) 1978.
New Yorker Films, 16 West 61st St., New York City, 10023.