Context-Dependent Thinning 2
1. Introduction
The problem of representing nested compositional structures is important for connectionist systems,
because hierarchical structures are required for an adequate description of real-world objects and
situations.
In fully local representations, an item (entity, object) of any complexity level is represented by a
single unit (node, neuron) (or a set of units which has no common units with other items). Such
representations are similar to symbolic ones and share their drawbacks. These drawbacks include the
limitation of the number of representable items by the number of available units in the pool, and
therefore, the impossibility to represent the combinatorial variety of real-world objects. Besides, a unit
corresponding to a complex item only represents its name and pointers to its components (constituents).
Therefore in order to determine the similarity of complex items, they should be unfolded into the base-
level (indecomposable) items.
The attractiveness of distributed representations was emphasized by the paradigm of cell
assemblies (Hebb, 1949) that influenced the work of Marr (1969), Willshaw (1981), Palm (1980),
Hinton, McClelland & Rumelhart (1986), Kanerva (1988), and many others. In fully distributed
representations, an item of any complexity level is represented by its configuration pattern over the
whole pool of units. For binary units, this pattern is a subset of units which are in the active state. If the
subsets corresponding to various items intersect, then the number of these subsets is much more than the
number of units in the pool, providing an opportunity to solve the problem of information capacity of
representations. If similar items are represented by similar subsets of units, the degree of corresponding
subsets' intersection could be the measure of their similarity.
The potentially high information capacity of distributed representations provides hope for
solving the problem of representing combinatorially growing number of recursive compositional items in
a reasonable number of bits. Representing composite items by concatenation of activity patterns of their
component items would increase the dimensionality of the coding pool. If the component items are
encoded by pools of equal dimensionality, one could try to represent composite items as superposition of
activity patterns of their components. The resulting coding pattern would have the same dimensionality.
However another problem arises here, known as "superposition catastrophe" (e.g. von der
Malsburg, 1986) as well as "ghosts", "false” or “spurious” memory (e.g. Feldman & Ballard, 1982;
Hopfield, 1982; Hopfield, Feinstein, Palmer, 1983). A simple example looks as follows. Let there be
component items a, b, c and composite items ab, ac, cb. Let us represent any two of the composite items,
e.g. ac or cb. For this purpose, superimpose activity patterns corresponding to the component items a
and c, c and b. The ghost item ab also becomes represented in the result, though it is not needed. In the
"superposition catastrophe" formulation, the problem consists in no way of telling which two items (ab,
ac, or ab, cb, or ac, cb) make up the representation of the composite pattern abc, where the patterns of
all three component items are activated.
The supposition of no internal structure in distributed representations (or assemblies) (Legendy,
1970; von der Malsburg, 1986; Feldman, 1989) held back their use for representation of complex data
structures. The problem is to represent in the distributed fashion not only the information on the set of
base-level components making up a complex hierarchical item, but also the information on the
combinations in which they meet, the grouping of those combinations, etc. That is, some mechanisms
were needed for binding together the distributed representations of certain items at various hierarchical
levels.
One of the approaches to binding is based on temporal synchronization of constituent activation
(Milner, 1974; von der Malsburg, 1981, 1985; Shastri & Ajjanagadde, 1993; Hummel & Holyoak, 1997).
Though this mechanism may be useful inside single level of composition, its capabilities to represent and
store complex items with multiple levels of nesting are questionable. Here we will consider binding
mechanisms based on the activation of specific coding-unit subsets corresponding to a group
(combination) of items, the mechanisms that are closer to the so-called conjunctive coding approach
(Smolensky, 1990; Hummel & Holyoak, 1997).
"Extra units" considered by Hinton (1981) represent various combinations of active units of two
or more distributed patterns. The extra units can be considered as binding units encoding various