Introduction
How does the brain allow us to bind ideas together? Coming upon an object such as a rose, for example, one is able to correctly recognize it as a particular rose. But how do we recognize complex objects? It is well known that information in the brain is segmented; thus when one is confronted with the rose, the olfactory system (located in one region of the brain) recognizes the associated smell. The visual system (located in another region) takes the visual input stimuli and further segments them so as to recognize shapes and colors. Somehow, within the visual system, the shapes and colors come together, allowing us to recognize the form of the visual object. Further, the smell and the visual image are somehow able to fuse together, or bind, so that one is able to recognize the rose in its fullness from either component. How can this happen? This question is known as the binding problem. As Valerie Hardcastle has recently put it, ``given what we know about the segregated nature of the brain and the relative absence of multi-modal association areas in the cortex, how [do] conscious percepts become unified into single perceptual units?'' [2]
Some think that there are three steps to recognition: segmentation, binding, and association. The easiest way to differentiate between these three components is to consider a visual perception problem. Consider the task of recognizing a pink circle. First, the brain segments the circle into at least two parts--color and shape. The area of the visual system that recognizes color labels the color; the area of the visual system that recognizes shape labels the shape. Somehow, color and shape are bound together into a cohesive mental object. The next step is association - searching in memory for objects that have appeared to be similar in form to the input--namely, a circle that is pink.
But even William James had difficulty imagining how the brain could perform such a feat, finding the theory of mental units ``compounding with themselves'' or ``integrating'' to be logically unintelligible. [9]
In other words, no possible number of entities (call them as you like, whether forces, material particles, or mental elements) can sum themselves together. Each remains, in the sum, what it always was; and the sum itself exists only for a bystander who happens to overlook the units and to apprehend the sum as such; or else it exists in the shape of some other effect on an entity external to the sum itself. Let it not be objected thatand O combine of themselves into ``water'' and henceforward exhibit new properties. They do not. The water is just the old atoms in the new position, H-O-H; the ``new properties'' are just their combined effects, when in this position, upon external media, such as our sense-organs and the various reagents on which water may exert its properties and be known.
To resolve this problem, some turn to the assumption that the brain contains a homunculus or that the world comes partitioned into prearranged categories. Others have searched for an explanation in the ability of 40 Hz cortical firing patterns to integrate lower-level neuronal events that are related to perceptual experiences [1]. Looking at the matter from the perspective of modern nonlinear dynamics, however, we suggest that James was unduly pessimistic: there is no binding problem.
In 1949, Donald Hebb published a book entitled The Organization of Behavior, in which he attempted to explain how the brain's information might be organized. In this classic work, Hebb introduced the idea of a cell assembly with the following words [4]:
Any frequently repeated, particular stimulation will lead to the slow development of a ``cell-assembly,'' a diffuse structure comprising cells...capable of acting briefly as a closed system, delivering facilitation to other such systems and usually having specific motor facilitation. A series of such events constitutes a ``phase-sequence''-the thought process. Each assembly may be aroused by a preceding assembly, by a sensory event, or--normally--by both.
Since it has long been known that an assembly of neurons has a critical firing level for ignition and behaves in the same ``all-or-nothing'' way as a single neuron [16,21,17,18], we can talk about the threshold of an entire assembly--where an individual neuron can participate in more than one assembly, just as a student can participate in more than one class. Similarly, assemblies of assemblies of neurons are expected to share the same properties. Each subassembly of an assembly has its own particular minimal trigger value that causes it to fire, and once a sufficient (threshold) number of subassemblies fire, the assembly ignites. This picture can be extended to assemblies of assemblies of assemblies of neurons and upward. Thus the threshold for a first order assembly would be a certain number of active neurons, the threshold for a second order assembly would be a certain number of active first order assemblies, the threshold for a third order assembly would be a certain number of active second order assemblies, and so on [5,3,19].
The development of a complex assembly can be related to the problem of learning to read. First, we must learn our letters, a task that involves recognizing patterns of shapes, lines and and angles. We then memorize words, which can be thought of as assemblies of letters; then phrases (assemblies of assemblies of letters), sentences, paragraphs and so on. In Hebb's terms, binding is a correlation of the activation of a complex assembly from a number of lower order assemblies.
In quite different ways, both Charles Legéndy [10] and John Hopfield [8,7] have used Hebb's concept of a cell assembly to estimate that there can be up to a thousand million different stored memory states, or complex assemblies, available to our brain. (To put this number into a human perspective, note that it is about equal to the number of seconds in thirty years.) In viewing the brain as a dynamical system--where each point in a phase space represents a state of the brain--Hopfield showed that such memory states are stable. If so, then a bound memory would be one of those stable states and therefore reliably stored and readily recalled. In this paper, the approaches of both Hopfield and Legéndy are reviewed, expanded upon, and related to the binding problem.
Before embarking on this quest, it is necessary to introduce some mathematical notions: What is a dynamical system? What is an attractor? What is a stable state?
Some dynamics
Consider a water filled bathtub with one drain. Obviously, the water will flow down the drain and eventually the tub will be empty. Let us assume that water is flowing into the tub at a rate that keeps the level constant. Now, the water will roam over the area of the tub until it gets sucked into the vortex located above the drain. This vortex will grab nearby water and cause it to flow into the pipe below. In such a situation, all of the water will be pulled into that vortex. We can say that the water is attracted to the vortex, or in other words that the vortex is an attractor for the water in the tub. As all the water in the tub is attracted to that vortex, the whole tub is called its corresponding basin of attraction.
Now imagine the same tub but let it contain two drains. Again, the water in the tub will roam all around--but when the water flows close enough to one drain or the other, it enters a critical region and will be attracted to one vortex (lying above one drain) or to the other. Since there are two attractors to the system, there are two basins of attraction. We can extrapolate this analogy to a tub containing as many drains and attractors as we wish.
There is another important property that this system exhibits. Not only is the vortex an attractor, but it is also stable. We can physically distort the vortex by pulling our fingers through it, but as soon as we withdraw our hand, the vortex will return to its previous shape. As a more vivid example, think of a tornado. A tornado is nothing but a giant vortex existing in the air and attracting most any object in its path. Throughout its existence, it will be distorted by objects in its path, but it will always return to that funnel shape that we all can recognize. The tornado is a stable dynamic entity or ``thing.''
Lastly, we wish to introduce the mathematical notion of a dynamical system. In order to define a dynamical system, one must define the space in which the system evolves. Each point in the space--known as phase space--determines the state of a system at a given time. Secondly, one must define the dynamics, that is, how a state in a given space evolves in time.
For example, to define the motion of a pendulum, we use a two dimensional phase space (for example, the angle of the pendulum and its angular velocity). Given one point in this space and Newton's dynamics second law of motion, we can predict the evolution of the pendulum for all time. (Note that knowing the angle alone is not enough to describe completely the state of the pendulum.)
As mentioned in the introduction, the phase space considered in this paper will be the combined states of the neurons in the brain, and every point in that phase space will represent a state of the brain.
How many ideas can a brain hold?
Consider a simple model of the neocortex that is composed of N McCulloch-Pitts neurons. Briefly, a McCulloch-Pitts neuron can be thought of as a switch with inputs of varying strength [13]. Recently Hopfield has numerically estimated the number of attractors A--which are stable by definition--in a such a system to be [8]:
Thus for 
, a conservative
estimate [20], the maximum number of
stable mental states is
More than a quarter century ago, in a relatively unknown paper, Legéndy estimated the number of Hebb's cell assemblies C that can form from y subassemblies of n neurons each to be:
where further details are provided in the section on The hierarchical structure of memory. Taking, for example, y=60 and n = 10,000, Legéndy estimated the maximum number of assemblies to be about
We are intrigued by the correspondence between the estimates of Legéndy and Hopfield.
First of all it is interesting to note that 
is about equal to
the number of seconds in 30 years. Thus if one supposes that the brain
records its awareness of a distinct perceptual state every two seconds,
is about the number of impressions that are actually experienced.
It is not difficult to see where Legéndy's formula come from.
Consider a model brain of N McCulloch-Pitts neurons
[13, 15]. Its organization is embodied in the
interconnections among these neurons, and there are 
of these.
If a single memory trace involves m neurons, each interconnected
with a probability of f, then it will fix a fraction 
of
the possible connections and leave the others undisturbed. Thus
is the probability of an interconnection being
undisturbed by the memory. By storing more memories, there will be
fewer free connections left to be used. If K traces are entered, the
chance of a particular interconnection being undisturbed is reduced to
What is the value of this expression for maximum storage? Certainly not unity, for then nothing is stored. Also it cannot be equal to zero for then all of the neural interconnections are determined and the individual memory traces interfere with each other. Thus we suppose that the maximum number of traces--C--is fixed by the condition
Since is much less than unity, the left hand side of this
expression is approximately equal to 
so
which is essentially the result obtained by Legéndy [10].
The Stability of Memories
As has already been mentioned, Hopfield found the storage capacity of
the brain to be approximately 
for 
The
storage capacity refers to the number of possible patterns that the
brain can store and accurately recall.
In his 1982 paper [8], Hopfield sought a method of solving the associative memory problem using a neural network model. This problem poses the question of how one can store a set of memories (or patterns) in a system so that when the system receives a stimulus resembling one of the stored memories, the system is able to recognize that stimulus as one of the stored memories. A natural extension of this problem is how many memories can be stored so that they are all recallable (that is, how many memories are stable)? For example, let us assume that we store a pattern representing the letter A into our system:
Now, suppose the following image is presented into the system:
We would hope that the system could respond by recognizing and reproducing the stored letter A. We would like to know how many such memories can be stored and recalled in this fashion.
In order to determine which pattern an input probe (S) most closely resembles, we need to define what ``close'' means. We measure closeness by counting the number of bits between two states that are not equal. For example, assume that a memory is composed of five neurons, that is, N = 5 and let
Notice that the first unit in each memory takes on the value +1.
That means that in both memories, neuron #1 is firing. Similarly, in
both memories, neuron #2 is not firing. In fact, the memory states
M and S differ only in the state of neuron #5. In the memory
labeled M, the neuron #5 is firing whereas in the memory labeled by
S, that neuron is not firing. Thus, M and S are separated by 1
unit of distance since the two patterns differ only in the 
entry.
Specifically, we wish to store a set of p different patterns (or
memories) composed of McCullough-Pitts neurons. We take a single
pattern to be composed of N McCullough-Pitts neurons. By letting , we are assuming that each memory is spread over the
entire brain.
Each neuron participating in a memory is designated a value of -1 or +1 depending on its firing state; thus, -1 represents a neuron that is not firing and +1 represents a neuron that is firing.
Now, going back to the problem, as the brain receives a partial memory, we want the brain to recognize the memory state that is closest to that initial partial memory. How did Hopfield propose that the brain might do this?
First, a random set of p memories was produced, each composed of N
binary units taking of the values of +1 or -1 with equal
probability. Once these memories were produced, an 
weight
matrix was computed representing the relationship between the neurons
throughout all memory states.
Each component 
in the matrix represents the strength of the
synaptic connection between neuron j and i and can be given a
positive, negative or zero value. If two neurons (i and j) are
connected in such a way such that i tends to fire (or not fire)
whenever j fires (or not), would be very positive. If i
does not perform the same action as j, would be very
negative. If i and j are not connected, their synaptic strength
would be zero. Hopfield considers the case were no neuron is connected
to itself, that is, 
for 
For example, assume that we wish to store 3 memories ( 
and 
), each memory consisting of 5 neurons, where
Let us determine the strength of the synaptic connection between neuron #1 and #2. By just looking at the memories, we can see that neuron #2 fires whenever neuron #1 fires. We thus expect that the weight value would be positive. For each memory, we compute products the states of neurons #1 and #2 and then sum over all the memories. Thus
In computing the synaptic connection
between neuron #4 and #5 we find that 
This is the
value we would expect for whenever #4 fires, #5 does not, and vice
versa. In this way, we can compute the weights between all five
neurons in the three memory system (see figure 1).
Figure 1: A diagram representing the synaptic strength between neurons
#1 through #5 in the memories of equation 5. Note that
Notice that the weight matrix will be symmetric (that is, 
for 
), for the synaptic strength connecting
neuron j to neuron i will be equivalent to the connection strength
between neuron i and j. This way of computing the weight matrix
has been termed ``Hebb's rule'' because of its similarity to the
hypothesis made by Hebb in the way the synaptic strengths between
neurons change in response to experience [8,7]. Two
neurons firing together (or not) tend to strengthen their connection
whereas two neurons not acting in the same fashion tend to weaken
their connection. (Since Freud and James, among others, had suggested
this simple learning rule, Hebb was amused that it came to bear his
name. It was the only aspect of his theory that he did not consider to
be original [12].) Also, it is important to note that Hopfield
assumed no learning in his model; that is, the memories have already
been stored and thus the coefficients of the weight matrix were fixed.
Once the weight matrix is computed, a random input probe S of length N is constructed. In trying to determine how many possible memories are stable, S is set equal to each memory pattern.
Each neuron in the model then computes a weighted sum of its inputs from other neurons and outputs a -1 or a 1 depending on whether or not the sum is below or above its threshold; the threshold for every neuron in this system is taken to be zero. That is,
where
sign( 
) means ``calculate the algebraic sign of.'' The above
formulation is interpreted in the following manner. The 
neuron in the altered pattern S' is computed by taking the algebraic
sign (where the sign of 0 is +1) of a sum of weighted inputs
received by neuron i.
For example, let us assume that N=5. Then if we want to compute the total input to neuron 1, we compute:
where 
represents the input to neuron i from neuron j. To determine
whether or not neuron #1 should be in a state of firing or not, we
then take the algebraic sign of the resulting sum.
Now, for all those entries of S' that differ from S, one neuron (or component) is randomly chosen and changed. The resulting changed probe is then labeled as the new S. This procedure is repeated until the distance between S and S' is 0, after which a comparison is made between the resulting S and the original p memories.
Hopfield estimated that approximately .1 N patterns could be stored,
enabling an input of a partial memory to converge to its closest memory state.
(This estimate has been recently confirmed by a detailed statistical argument
[7].) That is, approximately .1 times the number of neurons are
stable states, or the number of attractors can be approximated by:
Thus if a partial memory (S) is presented into the system, S will be attracted to the closest stored memory that is stable under this algorithm.
In the appendix, we present an example of a one hundred neuron model of a system storing five, ten, fifteen, twenty and twenty-five memory states.
The hierarchical structure of memory
Legéndy assumed that the storage capacity of the brain depends on the number of complex assemblies of neurons it contains. What are complex assemblies? Briefly, one complex assembly could be thought of as a memory (or pattern) composed of various levels of interacting subassemblies. For example, if a complex assembly recognizes the word ``consciousness,'' then one of its subassemblies could recognize or store the letter ``c'' (see figure 2).
Figure 2: The hierarchical view of assemblies
Let us see how a developing brain might organize its knowledge in such a
hierarchical structure. At birth (and perhaps even before) the young
brain accumulates impressions in primary assemblies--lines and angles
in the primary visual cortex (V1), musical tones and clicks in the
temporal lobes, etc. Palm has estimated that there are about 200 neurons in a
primary assembly [14]. If we conservatively assume that there are
neurons in the neocortex [20] and that primary
assembly neurons have a probability of about unity of being
interconnected, Legéndy's Equation (4) tells us that the
maximum number of primary assemblies 
.
But there is not enough time in the life of a human being to store
all these assemblies. To see this assume that one new assembly is
recognized and learned each second and note that 
seconds is
about equal to the age of the earth. Thus we must conclude that the
number of primary assemblies will be limited to the number of seconds
in (say) ten years or 
. If we set 
in Equation (3), almost all of the neural
interconnections are undisturbed.
Now we pass on to a consideration of second order assemblies
[5, 6], which are assemblies of assemblies rather
than assemblies of sensory neurons. If we suppose that ten primary
assemblies of 200 neurons each will combine to form a secondary
assembly, Equation (2) tells us that the
maximum number of secondary assemblies 
that can form before exhausting the number of available neural
interconnections. Since 
is about equal to the number of
seconds in three million years, we must again conclude that human life
is not nearly long enough to fill the brain with secondary assemblies.
Thus the number of secondary assemblies will again be limited to
something like , and again the number of available
interconnections will not be exhausted.
Moving on to tertiary assemblies--each composed of ten secondary
assemblies--we find from Equation (2) that the
maximum number of tertiary assemblies 
are possible, but this is the number of seconds in 30,000 years, again
too large. For fourth order assemblies--each composed of ten tertiary
assemblies--the
maximum number 
or the number of seconds in 300 years, which is still significantly more that one has time to learn.
From such considerations, one sees that the number of assemblies that can be stored in 30 years is about equal to the number of attractors that can be stably stored in them for assemblies of the fourth or fifth order.
Conclusions
Both Hopfield and Legéndy consider memory traces to be spread over entire neocortex, but they make complementary estimates. While Legéndy estimates the maximum number of complex assemblies that can be stored without being concerned about their stability, Hopfield estimates the number of stable attractors without being concerned about their structure as complex assemblies. Since the two estimates are equal, Legéndy's complex assemblies can be assigned--one by one--to Hopfield's stable attractors. Upon making this assignment, one sees that the binding problem simply disappears.
From the perspective of modern nonlinear dynamics, this result is not unexpected. Nonlinear dynamics typically generate ``things'' (hydrodynamic vortices, chemical molecules, cell assemblies, etc.), which can be treated as new dynamic objects at higher levels of organization. Upon meeting others, such things typically attract or repel; seldom do they fail to interact. Why did William James fail to see this? Because he was thinking in terms of linear dynamics, where the whole is equal to the sum of its parts, rather than in terms of nonlinear dynamics, where the whole is more than the sum of its parts.
As a final note, we emphasize that the estimates of Legéndy and Hopfield were made for model neurons that are much simpler than real neurons, so they have established only a conservative lower bound on the ``number of things a brain can know''.
Appendix
We have tested the Hopfield algorithm for the case of a model of the brain containing only 100 neurons, running simulations of the program by storing 5, 10, 15, 20 and 25 memories in order to investigate their stability. The algorithm was as follows:
- (a)
- We randomly produced p memories (p = 5,
10, 15, 20, 25) of length 100 composed of binary units taking on
the values of -1 or +1.
- (b)
- The weight matrix W was constructed according to the rule:
where
represents the value of the neuron of the
memory M. (Recall that the neurons can take on the values
of 
.)
- (c)
- The input vector S was set to one of the stored memories,
represented by
.
- (d)
- We then computed:
- (e)
- For all those entries of S' that differed from S, we
randomly chose and changed only one of those entries. The resulting vector was
labeled as the new S.
- (f)
- Items (d) and (e) were repeated until the distance between
S and S' was zero.
- (g)
- The resulting vector S was compared to the original memory
to which it was set ( ), and the distance between S and
was recorded.
Items (a) - (f) were repeated 50 times for each memory. For example, for the case of p = 5, 250 different trials were run (50 trials per memory). We then constructed the following three dimensional bar graph (see figure 3) showing the stability of the memories.
Figure 3: The stability of storing 5, 10, 15, 20 and 25 memories in a 100
neuron brain. The y-axis represents the data compiled for storing 5,
10, 15, 20 and 25 memories. The x-axis represents the final distance
h calculated in step (g) of the algorithm. The z-axis represents the
probability that a memory will land within h units of its original
state at the end of its evolution.
The results from this numerical experiment can be analyzed as follows:
- p = 5: All the memories tested were stable under the algorithm. That is, by starting at the any of the five memories stored, the algorithm returns to that same memory. What does this mean? If our ``brain'' receives a stimulus that roughly resembles any of the five memories stored, it is able to recognize the stimulus as one of those memories.
- p=10: In this case, the algorithm returns to the same memory
we started with over
of the time. For the remaining 
, the resulting
memory differs only in at most 2 places from the initial memory. These slightly
distorted memories will perhaps still be distinguishable as the original
memory; and since these occurrences occur infrequently, we conclude that all 10
memories are stable. - p=15: For this case, the algorithm returns to the same memory
we started with over
of the time. Approximately 
of the time, the
resulting memory only differs in 1 to 4 places from the initial
memory. These slightly distorted memories again may still perhaps be
recognizable as their original memory, but their image maybe not as
clear as one might hope. Approximately 
of the results land at a
distance up to 20 units away. Thus it cannot be guaranteed that all
of the 15 memories are stable. - p=20 and p=25: It is evident that as we increase the
storage of memories, the
ability to recall them deteriorates. (In other words, even if we
entered the memory the picture of a pink circle, our little brain could
not be assured of recognizing it again.) In fact, we cannot be
assured of being able to correctly recognize any of the 20 or 25 memories
that were previously stored. As p is increased, this lack of stability
continues to increase.
The storage capacity of our 100 neuron system is 10--meaning that we can store approximately 10 memory states and have them be recallable (or stable). What happens when we overload the storage capacity (of approximately .1 N) of the system? It responds by not remembering anything. This is intuitively reasonable because, as we increase the number of memories we attempt to store, the basin of attraction for each stored memory states must become smaller. Thus if we are checking for the stability of a particular memory, the algorithm will carry it to another state that might be in the basis of attraction of an entirely different memory state. As the storage capacity is exceeded, some stable states exist, but they lie far from the memory states that were originally stored (see figure 4).
Figure 4: When too many memories are stored, the basin of attraction
for each memory becomes too small. Illustratively, we store memories 
into a model brain. We hope that when confronted with a
distorted memory S, the brain recognizes . However, since we
stored too many memories, the basins of attraction for each stored
memory diminishes in size and S becomes attracted to 
.
- 1
- F Crick and C Koch.
Towards a neurobiological theory of consciousness.
Seminars in the Neurosciences, 2:263-275, 1990.
- 2
- V G Hardcastle.
Psychology's binding problem and possible neurobiological
solutions.
Journal of Consciousness Studies, 1:66-90, 1994.
- 3
- E Harth.
The Creative Loop: How the Brain Makes a Mind.
Addison-Wesley, Reading, Massachusetts, 1994.
- 4
- D O Hebb.
The Organization of Behavior.
Wiley, New York, 1949.
- 5
- D O Hebb.
Essay on mind.
Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1980.
- 6
- D O Hebb.
The structure of thought.
In The nature of thought, P W Jusczyk and R M Klein,
editors, Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1980, pp.
19-35.
- 7
- J Hertz, A Kogh and R G Palmer.
Introduction to the Theory of Neural Computation,
Addison-Wesley, New York, 1991.
- 8
- J J Hopfield.
Neural networks and physical systems with emergent
collective computational abilities.
Proc. Nat. Acad. Sci., 79:2554-2558, 1982.
- 9
- W James.
Principles of Psychology.
Holt, New York, 1890; (Republished by Dover, New York, 1950).
- 10
- C R Legéndy.
On the scheme by which the human brain stores information.
Math. Biosci., 1:555-597, 1967.
- 11
- R J McEliece, E C Posner, E R Rodernich and S S
Venkatesh.
The Capacity of the Hopfield Associative Memory.
IEEE Transactions on Information Theory, 33:461-482, 1987.
- 12
- P M Milner.
The mind and Donald O. Hebb.
Scientific American, January 1993: 124-129.
- 13
- W S McCulloch and W H Pitts.
A logical calculus of the ideas immanent in nervous activity.
Bull. Math. Biophys., 5:115-133, 1943.
- 14
- G Palm.
Cell assemblies, coherence, and corticohippocampal interplay.
Hippocampus, 3:219-226, 1993.
- 15
- W Pitts and W S McCulloch.
How we know universals: The perception of auditory and visual
form.
Bull. Math. Biophys., 9:127-147, 1947.
- 16
- A Rapoport.
``Ignition'' phenomena in random nets.
Bull. Math. Biophys., 14:35-44, 1952.
- 17
- A C Scott.
Neurodynamics (a critical survey).
J. Math. Psychology, 15:1-45, 1977.
- 18
- A C Scott.
Neurophysics.
Wiley, New York, 1977.
- 19
- A C Scott.
The Stairway to the Mind.
Copernicus (Springer-Verlag), New York, 1995.
- 20
- C F Stevens.
Reworking an old brain.
Current Biology, 3:551-553, 1993.
- 21
- H White. The formation of cell assemblies. Bull. Math. Biophys., 23:43-53, 1961.
ACKNOWLEDGEMENTS
We thank Alain Goriely and Aaron King for many helpful discussions and are pleased to acknowledge support from the National Science Foundation under Grant No. DMS-9114503.