A Hybrid Neural Network and Virtual Reality System
for Spatial Language Processing
Guillermina C. Martinez1, Angelo Cangelosi1, Kenny R. Coventry2
1School of Computing and 2Department of Psychology
University of Plymouth
Drake Circus, Plymouth PL4 8AA, UK
[email protected], [email protected], [email protected]
Abstract
This paper describes a neural network model for the study
of spatial language. It deals with both geometric and
functional variables, which have been shown to play an
important role in the comprehension of spatial prepositions.
The network is integrated with a virtual reality interface for
the direct manipulation of geometric and functional factors.
The training uses experimental stimuli and data. Results
show that the networks reach low training and
generalization errors. Cluster analyses of hidden activation
show that stimuli primarily group according to extra-
geometrical variables.
1 Introduction
The aim of this work is to develop a hybrid neural network
(NN) and virtual reality (VR) system for the study of spatial
language and cognition. It will also be tested as a prototype
natural language interface for virtual environments.
Spatial language, and in particular the use and
understanding of spatial terms such as over, above, under,
and below, has proven to be an important experimental field
for the investigation of cognition [3,13]. The use of an
expression involving a spatial preposition in English
conveys to a hearer where one object (figure) is located in
relation to a reference object (ground). Understanding the
meaning of spatial prepositions is of particular importance
in semantics as they are among the set of closed class terms
which are generally regarded as having the role of acting as
organizing structure for further conceptual material [14].
Recently, both experimental research and computational
models have investigated the use of spatial prepositions,
and their role in spatial cognition.
1.1 Psychological Literature on Spatial Language and
Function
In the experimental psychological literature it has been
shown that both geometric (e.g., the distance between two
objects and their relative orientation) and extra-geometric
variables (e.g., the function of an object and its size and
shape) play an important role in the comprehension of
spatial prepositions.
Traditionally, geometric constructs have been invoked to
underpin prepositions’ lexical entries (e.g., [10,11]). For
example, in the sentence, “The pear is in the bowl,” the
figure (the pear) is located in the region described by the
prepositional phrase “in the bowl”, with the spatial relation
expressed by in corresponding to “contained interior to the
reference object.”
Clearly, while geometry is important in the use and
comprehension of spatial prepositions, other extra-
geometric variables need to be invoked in order to account
for use and comprehension. For example the expression, the
man is at the piano, implies that the man is playing the
piano, not just that he is in close proximity to it. There have
been a number of empirical demonstrations showing that
extra-geometric factors play an important role in the use
and comprehension of spatial prepositions. Functional
relations have been postulated as key components
underlying the meaning of the spatial prepositions in, on
and at [1, 3, 4].
Functional relations have to do with how objects interact
with each other, and what the functions of objects are. For
example, with in, Garrod and Sanford [7] and Coventry [3]
propose that the lexical entry is: in [functional containment
- in is appropriate if the ground is conceived of as fulfilling
its containment function]. Whether or not in is appropriate
depends on a number of factors which determine whether
the container is fulfilling its function. Empirical evidence
for the importance of this functional analysis has been
forthcoming for topological prepositions.
It has also recently been shown that prepositions are
influenced differentially by geometric and extra-geometric
variables. Coventry, Prat-Sala and Richards [5] found that
the comprehension of over and under was more influenced
by function than above and below, while the comprehension
of above and below was better predicted by geometry than
over and under. In addition, effects of extra-geometric
variables have been shown to influence use and
comprehension even when the prototypical geometric
constraint holds. For example, they found that
appropriateness ratings of expressions such as the umbrella