Context-Dependent Thinning 19
(equation 4.10). CDT can be also considered as a hashing procedure: the subspace to where hashing is
performed is defined by 1s of z, and some 1s of z are mapped to that subspace.
Since the resulting bound codevector {z} is obtained in the CDT procedure by thinning the 1s of
z, (where the component codevectors are superimposed), {z) is similar to its component codevectors
(unstructured similarity is preserved). Therefore to retrieve the components bound in the thinned
codevector, we only need to choose the most similar component codevectors from their alphabet. This
can be done using an associative memory.
None of the mentioned binding operations, except for the CDT, preserves unstructured
similarity. Therefore to extract some component codevector from the bound codevector, they demand to
know the other component codevector(s). Then rebinding of the bound codevector with the inverses of
known component codevector(s) produces a noisy version of the sought component codevector. This
operation is known as decoding or "unbinding". To eliminate noise from the unbound codevector, a
"clean-up" memory with the full alphabet of component codevectors is also required in those schemes. If
some or all components of the bound codevector are not known, decoding in those schemes requires
exhaustive search (substitution, binding, and checking) through all combinations of codevectors from the
alphabet. Then the obtained bound codevector most similar to the bound codevector to be decoded
provides the information on its composition.
As in the other schemes, structured similarity is preserved by the CDT, i.e. bindings of similar
patterns are similar to each other. However the character of similarity is different. In most of the other
schemes the similarity of the bound codevectors is equal to the product of similarities of the component
codevectors (e.g. Plate, 1995). For example, the similarity of a*b and a*b' is equal to the similarity of b
and b'. Therefore if b and b' are not similar at all, the bound vectors also will not be similar.
The codevectors to be bound by the CDT procedure are initially superimposed component
codevectors. So their initial similarity is the mean of the components' similarities. Also, the thinning
itself preserves approximately the square of similarity of the input vectors. So, the similarity for
dissimilar b and b' will be >0.25 instead of 0 for the other schemes.
9.1.4. How the binding operation is used to represent predicate structure
In most of the schemes predicate structures are represented by role-filler bindings. Halford, Wilson, &
Phillips (in press) use predicate-argument bindings. The APNN-CDT scheme allows such
representations of predicate structures as role-filler bindings, predicate-argument bindings, and also
offers a potential for other possible representations. Both ordered and unordered arguments can be
represented.
9.1.5. The use of other operations and techniques.
- Normalization. After superposition of codevectors, some normalizing transformation is used in various
schemes to bring the individual elements or the total strength of the resulting codevector within certain
limits. In BSCs, it is the threshold operation that converts a non-binary codevector (the bitwise sum of
component codevectors) to a binary one. In HRRs, it is the scaling of codevectors to the unit length that
facilitates their comparison.
The CDT procedure performs a dual role: it not only binds superimposed codevectors of
components, but it also normalizes the density of the resulting codevector. It would be interesting to
check to what extent the normalization operations in other schemes provide the effect of binding as well.
Clean-up memory. Associative memory is used in various representation schemes for storage of
component codevectors and their recall (clean-up after finding their approximate noisy versions using
unbinding). After the CDT procedure, the resulting codevector is similar to its component codevectors,
however the latter are represented in the reduced form. Therefore it is natural to use associative
memories in the APNN-CDT scheme to store and retrieve the codevectors of component items of various
complexity levels. Since component codevectors of different complexity levels have approximately the
same and small number of 1s, an associative memory based on assembly neural networks with simple
Hebbian learning rule allows efficient storage and retrieval of a large number of codevectors.