Koko's Mac II: A Preliminary Report
Kim Rose, Mike Clark, Larry Yaeger, Tom Ferrara and Ann Marion
Apple Computer, Inc.
Since 1987, Apple Computer's Vivarium Program has been working with the
Foundation to provide Koko with a voice. Apple has designed and built a
special Mac II enclosure and written software which will allow Koko to activate
a voice by touching icons on the screen. With this tool, the Gorilla
Foundation can modify the information presented to Koko, and collect and
analyze Koko's use of the system to gain further insights into language
acquisition by higher-order primates.
Koko, the Gorilla Foundation, and the desire for a voice
Since she was one year old (1972), Koko has been learning a version of American
Sign Language (Ameslan), and now has a vocabulary of over 600 words. The
Gorilla Foundation was formed to continue this training and the related
research, and Project Koko is now the longest uninterrupted study of ape
language abilities ever undertaken. The research findings of the Foundation
offer strong support for the notion that, when immersed in an environment
in interactions with speaking humans, gorillas can not only comprehend and
react to spoken English, but can also invent new gestures, translate from
English to Ameslan, and converse at a high level of abstraction. The
dedication of researchers Dr. Francine Patterson and Dr. Ron Cohn is no
impressive - since lowland gorillas have a life span of more than 50 years,
researchers have basically committed the rest of their lives to studying
and a male gorilla, Michael.
At one time, when Project Koko was still on the Stanford campus, Koko had
access to a voice synthesis unit which she used to enhance her communication
skills. However, when the Foundation moved to Woodside Koko lost access
this device. The loss was traumatizing - when Koko was asked what she wanted
for her birthday, she signed, Voice! The researchers were also disappointed
lose this basic tool, and have been seeking a way to reinstate this capability.
With the system we've designed, Koko's researchers from the Gorilla Foundation
can display icons for Koko to select, manipulate the icons available to
and record both her and the researchers' use of their systems. This last
function will give the Gorilla Foundation its first tool to completely record
and analyze Koko's communications. Dr. Francine Patterson (Executive Director
of the Gorilla Foundation) serves on Vivariums Advisory Board, and is providing
us with regular feedback on Koko's use of the system. Dr. Patterson believes
that by enabling Koko to use spoken language, we can break new ground in
Kokos uses a standard Mac II enclosed in a special gorilla-proof housing.
enclosure is constructed of 1/2" polycarbonate and is bolted to the
Koko's trailer. Access to the computer is provided through a special touch
screen designed to withstand the 2,000 pounds of force an excited gorilla
potentially generate. The computer has two video cards and is shared by
and the researcher.
What we hope to learn
Alan Kay notes that, "Since we're doing a project simulating animals,
have at least one animal on our advisory board!" We further expect
some insights into what kind of iconic representation scheme will allow
do sophisticated retrievals. Will she be able to use the interface we've
designed to navigate successfully? Are there any ways in which we can
extrapolate from the results of this experiment to make systems more accessible
to novice human users?
A continuing exposure to Koko's use of language can provide insight into
nature of her cognition and brain function; we expect this to continue to
interest and value, especially once we provide the Foundation with copies
some of the time/date-stamped files of her speech and the researchers' notes.
The nature of the world model that Koko has built in her brain/mind is very
much of interest; presumably, use of language requires some internal world
model. Larry Yaeger, Principle Engineer on the Vivarium, has embarked on
project to build simulated critters that build and use internal models of
world. Larry is basing his work on neurophsyiological data and neural network
models, but we believe that phenominological input regarding this issue
natural sources (such as Koko) will be of great value.
What we have already learned
In the process of developing the tools, Vivarium engineers Mike Clark (Systems
and Electronics), Tom Ferrara (Mechanical) and Larry Yaeger (Software) have
a number of challenging experiences. Above all, the personal experience
participating in this research - meeting Koko and communicating with her
been truly awesome.
Systems and electronics
Mike Clark has spearheaded system design since the project's inception.
Gorilla Foundation had an existing proposal to develop a large keyboard
Koko, but during initial discussions Mike Clark determined that everything
the Gorilla Foundation wanted to do (and more) could be done with a Mac
large touch screen, and a HyperCard-style button interface. At the first
meeting Mike demonstrated a HyperCard stack that spoke all of the words
the Koko's Kitten book, spoken by his niece and digitized by his son.
The most important consideration in all mechanical design aspects was for
safety of the gorilla. A typical gorilla response to anger or excitement
run full speed and backhand the object of the anger. For Koko, this means
260-pound animal running at about 20 miles per hour, swinging her arm with
force comparable to a 10-pound shotput traveling at 100 miles per hour.
Accordingly, Tom Ferrara faced a significant design challenge in developing
workstation - a Mac II computer, a 19" Sony monitor, and a touch screen
a package which could withstand these extreme forces. Failure of the physical
system could only be tolerated if it did not endanger Koko in any way.
The most difficult design consideration was the part of the system with
the gorilla actually comes in contact: the touch screen. Most touch screens
are thin, to minimize the visual parallax between the image on the screen
the point of contact. But we needed a thick piece of glass to withstand
potential force. Mike spent over a month talking to MicroTouch, the Gorilla
Foundation, and the Los Angeles Zoo to determine the right combination of
materials. A standard MicroTouch capacitive screen optically bonded to a
one-inch thick piece of tempered glass was finally selected. It is interesting
that the parallax through a one-inch piece of tempered glass is not as bad
one might predict when operating the system from the floor. Behind the touch
screen is mounted a standard 19" Sony Trinitron from SuperMac Technology.
Design of the computer's enclosure proved almost as challenging. We needed
enclosure which would withstand Koko's excitement and absorb the shock of
blows she might deliver. After debating and testing a number of designs,
finally built the enclosure of an inner framework made from 1" x 2"
6064-T6 aluminum, covered by a 3/8" polycarbonate sheet. The Monitor
mounted on a sliding assembly that is dampened with gas struts with 3"
travel. This provides shock absorption in the event that Koko should throw
weight into the screen area. The final unit as delivered is 28 5/8"
tall x 21
1/4" wide x 39 1/4" long. Passive ventilation is provided by slots
designed to channel any foreign materials (bananas, feces, etc.) away from
CPU. (For more information on the design's evolution, see Appendix 1.)
Koko has a standard Mac II CPU consisting of 5 megabytes of memory, a
40-megabyte hard disk, and a SuperMac Spectrum/8 color card, connected through
the floor of Koko's trailer to the researcher's monitor and keyboard.
In the process of designing the physical system, Tom Ferrara had to concern
himself with both durability and ergonomics. He felt that the project helped
him to gain a better understanding of human-computer interfaces, especially
terms of accommodating users who may not have the physical ability to use
keyboard or manipulate a mouse, but who can point.
Software user interface
The Koko project inspired Mike Clark to learn SuperCard so that he could
a large-screen color interface. So far, three applications have been developed
to familiarize Koko with the touch screen interface. The first is four
digitized animal pictures that make the animals sound when touched. The
shows a picture for each letter of the alphabet (developed by the Gorilla
foundation) and speaks the name of the picture when it is pressed. The final
application is called KokoPaint. It is a simplified color painting program
which allows Koko to select from a few colors and finger-paint with them.
Larry Yaeger experimented with HyperCard and SuperCard as candidates for
implementation of the primary research tool, but he ultimately opted to
custom Mac application for Koko entitled "Lingo". The interface
Koko is simple, consisting primarily of uniformly shaped, regularly spaced
buttons - rectangular regions on her touch-screen display - that, when pressed,
cause the playback of a sampled-sound resource corresponding to the selected
button. Each button corresponds to a single word and may display text, a
full-color picture, or both. Sampled sound was selected over phonemic speech
generation early on, primarily for the improved clarity of speech. This
rapidly led to a discussion of who would serve best as Koko's voice, and
decided to find a way to let her make the selection herself. The voices
four human females were recorded with MacRecorder and later played for Koko,
who ultimately selected one of them as her own.
Anticipating a maximum vocabulary of between 500 and 2000 words (Koko's
common-use signs and the total number of uniquely identifiable signs she
evidenced, respectively), specific, strategically placed buttons on the
were set aside to perform a forward- or backward-paging function, hopefully
allowing Koko to flip between screens of words. Additionally, at the request
of the Gorilla Foundation, a sentence buttonwas incorporated that accumulates
the individual words as they are pressed and when pressed itself produces
sentence consisting of these words played rapidly in sequence.
The interface just described is presented to Koko on a "Subject screen"
she uses to make her word selections. A separate "Researcher screen"
Koko's word selections and permits the recording of contextual notes by
observers. The Researcher screen is a standard Apple RGB monitor fed by
cable out of the bottom of Koko's workstation. The keyboard cable also extends
out the bottom, and the keyboard, mouse, and second monitor form a second
workstation for the researcher.
Through various menu options, the Researcher can employ a number of tools
laying out and manipulating the Subject screen and for saving and restoring
these layouts in a convenient fashion. Several characteristics of the
interface could not be predetermined - for instance, most useful button
optimal number of buttons per screen, the relative usefulness of text versus
images for the buttons, the desirability of playing a sound immediately
each buttons press versus delaying until the sentence button is pressed,
on. The solution was to make such features easily modifiable through menu
selections and simple dialogs from the Researcher screen. Tools for editing
the words, sounds, and images associated with the various buttons are provided
directly within Lingo to help the researchers design and experiment with
Subject screen layout. Alternate layout configurations may be saved and
in the usual Macintosh manner through the File menu.
The system displays the subject's and researcher's entries and voicing Koko's
selections and records them in a pair of archival data files - the Subject
Archive file and the Researcher Archive file - for subsequent linguistic
behavioral analyses. Every utterance (button press) made by Koko is separately
date/time-stamped; the researcher's notes are date/time-stamped and archived
once per line. The configuration of the Subject screen along with any changes
made to it is also archived in simple TEXT files which may be examined in
word processor. Fresh transcript files are automatically opened and closed
a daily basis, or whenever a new configuration is opened.
What has the Gorilla Foundation learned?
At the time of this writing, Koko has not yet used her complete workstation;
Ron and Penny are hard at work gathering all the images and sounds for the
workstation software, and the final screen layout is being designed. Koko
only used the stand-alone Mac II for voice selection. Even with only two
sessions totaling less than one hour with the Macintosh, however, significant
results have already been achieved. Through the voice-selection exercises,
Koko was able to solve a complex conceptual problem presented to her in
English by a computer, without prior training or the aid of visual markers
any kind. The accomplishment of this task illustrates that a gorilla has
cognitive capacities for:
1. comprehension of spoken language: Koko responded to the recorded statements
and grasped the task requirements.
2. self-concept: Koko employed the sign "myself" to refer to one
of the voices
when it spoke both the query and concept sentences.
3. auditory discrimination of human voices: Koko responded differently to
Next phases of the computer-language project will allow Koko to use the
she has selected to create her own spoken statements [for further reading
reference of G.F. Journal article].
What does the Gorilla Foundation hope to learn?
The first research project is to develop a user interface that will allow
to navigate between screens. The entire vocabulary of over 600 words cannot
effectively be displayed for Koko on her screen all at the same time. We
therefore design a way to represent to Koko where she is in relation to
other things available to her and how to get to them. In addition, we need
provide Koko with a way to indicate thought completion. After the user
interface issues have been resolved, then the main research task of
communications research can begin.
Once Penny and Ron saw the Koko's Macintosh, they began to see possibilities
for using it that extend beyond those originally imagined. They began
thinking about the possibility of empowering Koko to control her environment.
Could our engineers make it possible for the Mac to control the lighting
Koko's room so that Koko could control lighting herself? Could she learn
to operate the interface they designed? Could Koko control her curtains,
raising and lowering them at will? Currently she orders around researchers
do these tasks for her - would she perform them herself if she could?
In conclusion. . .
Designing Koko's workstation and software has already led to some insights
regarding user interface issues in hardware and software design. And the
manner in which Koko now interacts with her Macintosh, its rather special
workstation, and its software will undoubtedly yield even greater insights
these issues. But perhaps the most enlightening aspect of this work will
the greater understanding we obtain of the nature of intelligence through
ongoing study of language and its use. Through such studies of alternate
cognitive systems we should be able to develop a better understanding of
cognition and communication. Perhaps we may then apply this understanding
the development of functional artificial intelligence, or, at the very least,
come to know ourselves better. And if we are able to see the human species
part of a continuum, rather than uniquely set apart, perhaps we will become
more aware of our place in the world's ecosystem - and the need to act
A recap and timeline of the design and building phases for the workstation:
1st pass, February 1987:
o Screen set at 52-degree angle
o CPU located at bottom of enclosure parallel to the floor
o Overall height to center of screen 22 in.
o No open areas where foreign matter (gorilla poop) can be forced into
2nd pass, August 1987:
o Screen set at 30 degree angle
o CPU located at top of enclosure parallel to floor
o Overall height to center of screen 17 in.
o Top surface of enclosure to be flat so Koko can sit on it - she weighs
o Material to be 3/8" 6064-T6 aluminum with internal structural supports
allow CPU mounting with isolation pads
o Front panel to be 3/8" lexan.
3rd pass, February 1988:
o Provided recess of 6" at bottom of front panel to allow Koko's legs
underneath when she sits on the floor
o Redesigned enclosure to utilize thinner material and incorporate an inner
sub-structure which would add stability as well as allowing more efficient
design of isolation set-up inside
4th Pass, September 1988:
o Incorporated 3/8" plexiglass cover over the screen area to insure
could not put her hand through the monitor when not in use
o Redesigned front panel to accept either 17" or 19" display to
from the front to provide easier access for service
o Changed cable ports from the back to the bottom of the unit so the cables
would be located underneath the raised floor of the trailer.
5th Pass, January 1989:
o Deleted the Plexiglass screen cover and designed front panel to accept
impact laminated glass touch screen
o Designed a filtration system utilizing open cell 1" thick foam. It
require 1 piece at the bottom of the unit and another piece 1/2" thick
o Two muffin fans will be used to circulate air through the filters and
allow better cooling inside the enclosure.
Appendix 2: Gorilla Language Bibliography
1. Patterson, F.G. "The Gestures of a Gorilla: Language Acquisition
Pongid." Brain and Language, 1978, 5, 72-97.
2. Patterson, F.G.P. "Human Communication with Gorillas." In G.B.
In the Spirit of Enterprise. San Francisco: W.H. Freeman and Co., 1978.
3. Patterson, F.G. "Linguistic Capabilities of a Young Lowland Gorilla."
F.C. Peng (Ed.), Sign Language and Language Acquisition in Man and Ape:
Dimensions in Comparative Pedolinguistics. Boulder, Colorado: Westview
Press,1978. Reprinted in R.L. Schiefelbusch and J.H. Hollis (Eds.), Language
Intervention From Ape to Child, Baltimore, Maryland: University Park Press,
4. Patterson, F. "Conversations with a Gorilla." National Geographic,
1978,154 (4), 438- 465. Condensed in Reader's Digest, March,1979, 114, (683),
5. Patterson, F.G. "Linguistic Capabilities of a Lowland Gorilla."
Dissertation, Stanford University, Psychology, 1979. University Microfilms
International, Ann Arbor, Ml 48106 (#79-172-69). Abstracted in Dissertation
Abstracts International, August 1979, 40-B, 2.
6. Patterson, F. "Comment on Terrace" (Letters), NYU Educational
1980, 11 (3), 33.
7. Patterson, F.G. "Creative and Innovative Uses of Language by a Gorilla:
case study." In K.E. Nelson (Ed.), Children's Language, Vol. 2. New
Gardner Press, 1980.
8. Patterson, F. "Gorilla Talk: Comment on Monkey Business, "
The New York
Review of Books, October 1980, 27(15), 45-46.
9. Patterson, F. "In Search of Man: Experiments in Primate Communication."
Michigan Quarterly Review, Winter 1980,19 (1), 95-114.
10. Patterson, F. "Can an Ape Create a Sentence? Some Affirmative Evidence."
Science, 1981, 211, 86-87.
11 . Patterson, F. and Linden E. The Education of Koko. New York: Holt,
Rinehart and Winston, 1981.
12. Patterson, F. "Gorilla Warfare." American Psychological Association
Monitor, January 1981,12 (1),16 and 41.
13. Patterson, F. "More on Ape Talk." The New York Review of Books,
1981, 28 (5), 43.
14. Patterson, F. "Generalized Language Ability in the Gorilla."
presented at the annual meeting of the American Psychological Association,
Angeles, August 1981.
15. Patterson, F. "Gorilla Language Acquisition." National Geographic
Research Reports, Vol.17 (1976 Projects). Washington, D .C .: National
Geographic Society, 1984.
16. Patterson, F. Koko's Kitten. New York: Scholastic Books, 1985.
17. Patterson, F., Share, E. and Cornwall, C. "Interspecies Communication
Conservation." Primate Conservation (IUCN/SSC), January 1985, 5, 39-40.
18. Patterson, F. "The Mind of the Gorilla: Conversation and Conservation."
K. Benirschke (Ed.), Primates: The Road to Self-sustaining Populations.
19. Patterson, F., Tanner, J., and Mayer, N. "Pragmatic Analysis of
Utterances: Early Communicative Development in the Gorilla Koko." Journal
Pragmatics, 1988,12, (1), 35-55.
20. Patterson, F. Koko's Story. New York: Scholastic Books, 1987.
21. Patterson, F., Patterson, C. H., and Brentari, D.K. "Language in
Chimp, and Gorilla." American Psychologist, March, 1987, 42 (3), 270-272.
22. Patterson, F. and Patterson, C.H. "Review of Ape Language: From
Conditioned Response to Symbol." American Journal of Psychology, 1988,101,
Other Publications of Interest
1 . Gorilla, a journal published semiannually by the Gorilla Foundation,
620-530, Woodside, CA. 94062.
2. Crail,Ted. Apetalk and Whalespeak. Los Angeles: J.P. Tarcher,1981.
3. Vessels, Jane. "Koko's Kitten." National Geographic, January,
1. Harrar, L. "Signs of the Apes, Songs of the Whales" (Videotape).
MA: WGBH Educational Foundation, Nova. Available through Vestron Video,
4000, Stamford, CT 06907.
2. Johnson, Anne C. "Are They Really Dumb? - Intelligence and Language
the Animals" (Filmstrip). 4 News Currents, Knowledge Unlimited 1984,
Madison, WI 53701
3. Moses, Harry. "Talk to the Animals." A 1 2-minute 16 mm film
60 Minutes CBS. Available through CRM McGraw-Hill Films, 110 Fifteenth Street,
Del Mar, CA 92014.
4. National Geographic Society. "Gorilla" (Videotape), 1986. Vestron
Box 4000, Stamford, CT 06907.
5. Schroeder, Barbet. "Koko, a Talking Gorilla." (90-minute 16mm
New Yorker Films, 16 West 61st St., New York City, 10023.