15-883 Homework #4
Computational Models of Neural Systems

A Simple Matrix Memory

In this exercise you will calculate some statistics for a matrix memory that maps input patterns into "simple representations" (Marr's term) in a higher-dimensional space. Read the 1994 paper by O'Reilly and McClelland before proceeding further.

The input space contains 50 units. Each input pattern has 6 of these units turned on.

The output space contains 100 units. Each output unit receives connections from 15 input units, selected at random. The output units' thresholds are set so that roughly 5 units turn on.

Matlab Pointers

The Matlab function nchoosek(N,k) returns the number of ways of selecting k elements out of a set of N. You will want to use it in this exercise. You also need to know how to define simple functions of your own. You can do it by creating a file in the current directory, using the Matlab editor or any other text editor. But you can also create simple functions right at the keyboard. Example: how many ways are there to flip a coin 20 times and get equal numbers of heads and tails? The expression below creates a function faircoin that takes a value N as input, and returns the value of the expression nchoosek(N,N/2). Computing faircoin(20) will solve our problem. 
>>  faircoin = @(N) nchoosek(N,N/2)

faircoin =
 
    @(N) nchoosek(N,N/2)


>>  faircoin(20)
184756

Questions

  1. How many possible input patterns are there for our little memory? Give a precise number.

  2. What is the sparseness of the input code, i.e., what percentage of units are active? What about the output code?

  3. Write a Matlab function to calculate the probability that an output unit receives exactly H hits from active input units.

  4. Calculate the probability distribution for values of H, and write it down in tabular form.

  5. What is the expected value of H? (a) Show how to calculate this from the probability distribution table above. (b) Show how to calculate this directly, using the formula on this page: http://mathworld.wolfram.com/HypergeometricDistribution.html

  6. What is the variance of the distribution for H? Write a Matlab expression to compute the result, and give the numerical value. To answer this question, read about the hypergeometric distribution here: http://mathworld.wolfram.com/HypergeometricDistribution.html

  7. What value should be used for the thresholds of output units in order to make it likely that the output pattern will contain 5 active units?

  8. Assume that input patterns A and B overlap by 3 bits. Consider an output unit that has 5 hits for pattern A. What is the likelihood that this unit has exactly 5 hits for pattern B? You must use the formula for Pb from O'Reilly and McClelland, and consider the different possible values for Hab, the number of bits that are hits for both pattern A and pattern B. Note: Hab cannot be greater than the overlap; it also can't be too small, since we know that the unit has 5 hits for pattern A, we're assuming it also has 5 hits for pattern B, and there are only 6 bits in each pattern. Once you define a function for Pb, you can write a one-line expression to calculate the answer.
Dave Touretzky