A Hybrid Neural Network and Virtual Reality System for Spatial Language Processing



Figure 1: Examples of experimental conditions in the second experiment of Coventry et al. [5]. The three scenes differ in the level of
variable F
unction. In the control condition (left) there is not rain, in the non-functional condition (center) the umbrella does not protect
the man from the rain, and in the functional condition (right) the umbrella is fulfilling its function of protection the man from the rain.


can be rotated at 90, 45, and 0 degrees) Function
fulfillment of protection from the rain (3 levels: yes, no,
control), Appropriateness of object for protection
function, e.g. umbrella or suitcase (2 levels: yes, no) and
Object type (4 levels). This results in 72 experimental
scenes/conditions. An example of three scenes is presented
in Figure 1. The scenes differ in the level of the variable
Function.

Three network architectures are used. They only differ in
the number of input units and the way input scenes are
encoded. The five hidden units and the four output units are
the same in all networks (Figure 2).

Orientation:



yes U

Functionfulfill: no U
control O


OVER
UNDER
ABOVE
BELOW


Object Properties O
(Iocalist or feature) O

Figure 2: Neural network architecture

Network A: Localist experiment encoding

In this network, the number of input units exactly reflects
the number and levels of the four experimental variables.
This architecture has a total of 12 localist input units. We
use the term localist to indicate that for each variable only
one unit is active.

Three input units are used to encode the three levels of
Orientation of the protecting object. Three localist units
are used for the three levels of the Function independent
variable. Two units encode the levels of Appropriateness,
and four units the types of Object.

Network B: Localist Object Encoding

This network does not have an explicit representation of the
object appropriateness, because eight localist units are used
to represent all objects. There are also three localist units
for Orientation and three for Function. This architecture
has a total of 14 input units.

Network C: Feature-based Object Encoding

In this network the objects are encoded according to their
geometrical and functional features. Each object is
represented using eight feature-based units. Three units
encode the dimension of the object in the three dimensions
(x, y, z) and three encode the major shape components
(hemispherical, conical, cuboid). Two units refer to the
lexicalized function of the object (i.e. Appropriateness).
For example, the object umbrella is encoded as x=1, y=1,
z=.67, hemispherical=1, conical=0, cuboid=0,
appropriate=1, inappropriate=0.

There are three localist units for Orientation and three for
Function. This architecture has a total of 14 input units.

Training

A standard error backpropagation algorithm was used, with
a learning rate of .01, momentum of .9 and 10000 epochs.
Of the total of 72 scenes, 71 were used for each training
epoch, and 1 for the generalization test. The training of each
network type A/B/C was replicated ten times, by varying
the initial random weights and the stimulus randomly taken
out for the generalization test.

The subjects’ mean ratings for the use of the four
prepositions were normalized in the range 0-1 and were
used as teaching input for the backpropagation training.

2.2 Virtual Reality Environment

The VR module consists of an interface for the
manipulation of 3D objects in the scene. For example, in
the umbrella scene there are three objects that the user can
manipulate: the man, the protecting object (e.g. umbrella or



More intriguing information

1. Should informal sector be subsidised?
2. The name is absent
3. The Composition of Government Spending and the Real Exchange Rate
4. INTERACTION EFFECTS OF PROMOTION, RESEARCH, AND PRICE SUPPORT PROGRAMS FOR U.S. COTTON
5. PROTECTING CONTRACT GROWERS OF BROILER CHICKEN INDUSTRY
6. POWER LAW SIGNATURE IN INDONESIAN LEGISLATIVE ELECTION 1999-2004
7. The name is absent
8. The name is absent
9. The name is absent
10. The name is absent
11. The Role of Trait Emotional Intelligence (El) in the Workplace.
12. Personal Experience: A Most Vicious and Limited Circle!? On the Role of Entrepreneurial Experience for Firm Survival
13. Transfer from primary school to secondary school
14. Conditions for learning: partnerships for engaging secondary pupils with contemporary art.
15. Income Mobility of Owners of Small Businesses when Boundaries between Occupations are Vague
16. Deletion of a mycobacterial gene encoding a reductase leads to an altered cell wall containing β-oxo-mycolic acid analogues, and the accumulation of long-chain ketones related to mycolic acids
17. Return Predictability and Stock Market Crashes in a Simple Rational Expectations Model
18. The name is absent
19. The name is absent
20. Giant intra-abdominal hydatid cysts with multivisceral locations