Context-Dependent Thinning 18
Associative-Projective Neural Networks paradigm (Kussul, 1988, 1992; Kussul, Rachkovskij, & Baidyk,
1991a).
9.1 Comparison to other representation schemes
Let us compare our scheme for representation of complex data structures using the CDT procedure (we
will call it "APNN-CDT" below) with other schemes using distributed representations. The best known
schemes are (L)RAAMs (Pollack, 1990; Blair, 1997; Sperduti 1994), Tensor Product representations
(Smolensky, 1990; Halford, Wilson, & Phillips, in press), Holographic Reduced Representations (HRRs)
(Plate, 1991, 1995), Binary Spatter Codes (BSCs) (Kanerva, 1994, 1996). For this comparison, we will
use the framework of Plate (1997) who proposes to distinguish these schemes using the following
features: the nature of distributed representation; the choice of superposition; the choice of binding
operation; how the binding operation is used to represent predicate structure; the use of other operations
and techniques.
9.1.1. The nature of distributed representation
Vectors of random real-valued elements with the Gaussian distribution are used in HRRs. Dense binary
random codes with the number of 1s equal to the number of 0s are used in BSCs. Vectors with real or
binary elements (without specified distributions) are used in other schemes.
In the APNN-CDT scheme, binary vectors with randomly distributed small number of 1s are
used to encode base-level items.
9.1.2. The choice of superposition
The operation of superposition is used for unstructured representation of an aggregate of codevectors.
In BSCs superposition is realized as a bitwise thresholded addition of codevectors. Schemes
with non-binary elements, such as HRRs, use elementwise summation. For tensors, superposition is
realized as adding up or ORing the corresponding elements.
In the APNN-CDT scheme, elementwise OR is used.
9.1.3. The choice of binding operation
Most schemes use special operations for binding of codevectors. The binding operations producing the
bound vector that has the same dimension as initial codevectors (or one of them in (L)RAAMs) are
convenient for representation of recursive structures. The binding operation is performed "on the fly" by
circular convolution (HRRs), elementwise multiplication (Gayler, 1998), or XOR (BSCs). In
(L)RAAMs, binding is realized through multiplication of input codevectors by the weight matrix of the
hidden layer formed by training of a multilayer perceptron using the codevectors to be bound.
The vector obtained by binding can be bound with another codevector in its turn. In Tensor
Models, binding of several codevectors is performed by their tensor product. The dimensionality of
resulting tensor grows with the number of bound codevectors.
In the APNN-CDT scheme, binding is performed by the Context-Dependent Thinning procedure.
Unlike the other schemes where the codevectors to be bound are not superimposed, they can be
superimposed by disjunction in the basic version of the CDT procedure. Superposition codevector z (as
in equation 4.4) makes the context codevector. The result of the CDT procedure may be considered as
superimposed bindings of each component codevector with the context codevector. Or, it may be
considered as superimposed paired bindings of all component codevectors with each other. (Note that in
the “self-exclusive” CDT version (section 5) the codevector of each component is not bound to itself. In
the hetero-CDT version, one codevector is bound to another codevector through thinning with the latter).
According to Plate's framework, CDT as a binding procedure can be considered as a special kind
of superposition (disjunction) of certain elements of the tensor product of z by itself (i.e. N2 scalar
products ziZj). Actually, (zi) is disjunction of certain ziZj=zi∧Zj, where Zj is the j-th element of permuted z