15-494/694 Cognitive Robotics Lab 8:
Convolutional Neural Networks

I. Software Update and Initial Setup

$ cd ~/cozmo-tools
$ git pull

II. Experiments with the MNIST Dataset and Linear Models

1. Make a lab8 directory.

2. Download the files mnist_data.zip, mnist1.py, mnist2.py, mnist3.py into your lab8 directory.

3. Unzip the mnist_data.zip file.

4. Skim the mnist1.py source code. This is a linear neural network with one layer of trainable weights.

5. Have a look at the PyTorch documentation, and specifically the documentation for torch.nn.Linear.

6. Run the model by typing "python3 -i mnist1.py". The "-i" switch tells python not to exit after running the program. Press Enter to see each output unit's weight matrix, or type control-C and Enter to abort that part.

7. Try typing the following expressions to Python:
   • model
   • params = list(model.parameters())
   • params
   • [p.size() for p in params]
   The first parameter is the 784x10 weight matrix; the second one is the 10 biases.

8. How long did each epoch of training take, on average? ________________

9. Modify the model to use the CPU instead of the GPU. (You just have to uncomment one line and comment out another.)

10. Run the model on the CPU. How long does each epoch take now? ________________
    Are you surprised? GPUs don't help for small models. A few thousand weights is small.

11. Skim the code for the mnist2 model. This model has a hidden layer with 20 units. Each hidden unit is fully connected to the input and output layers.

12. Run the mnist2 model on the GPU (the default). How long does each epoch of training take, on average? ________________

13. You can use the show_hidden_weights() and show_output_weights() functions to display the learned weights.

14. Modify the mnist2 code to run on the CPU. How long does each epoch take now? ________________

III. Experiments with the MNIST Dataset and a Convolutional Model

1. Skim the code for the mnist3 model.

2. Run the model on the GPU, not the CPU. You can ignore the "THCudaCheck FAIL" message. Look at some of the kernels the model learns.

3. How many parameters does this model have, where each parameter is a tensor? ________________

4. What are the parameters of this model? Describe them in English. ________________________________________________
   ________________________________________________________________

5. Note that two of the parameters are batch normalization values (means and variances) created by the BatchNorm2D layer. The rest are weights. (Biases are considered to be weights.) Looking at the sizes of the various weight and bias tensors, how many total weights does this model have? Show your calculation. ____________________________________

   A convolutional neural network is a "virtual" network where each kernel is replicated many times, but we don't actually build out all the units and connections as individual data structures, since they share the same weights. When running data through the network, though, we still have to do all the multiply and accumulate operations as if we had built out the network, so the number of "effective" weights is many times the number of weight parameters. How many effective weights are in the mnist3 model? Show your calculation. ________________________________________________

6. This model runs on the GPU. How long did each epoch of training take, on average? ________________

7. If you modify the model to run on the CPU, how long does an epoch take now? (You don't need to run the model to completion.) ________________

IV. Object Recognition with MobileNet

1. Run the MobileNet demo on Cozmo. Note: to install this demo you must download both MobileNet.fsm and the labels.py file found in the same directory.

2. Use your cellphone to call up a picture of a cat and show it to Cozmo.

3. Type "tm" to tell the program to proceed with recognition. Did it recognize the cat?

4. Try some dog breeds, and some other object classes such as airplanes or cars.

V. Homework Problem: Digit Recognition

1. In this problem you're going to have Cozmo recognize handwritten digits. You can assume that the digits are separated by whitespace; they do not overlap. They will be drawn with a fat magic marker on a white sheet of paper that fills the camera image so there is no background clutter.

2. You can use the torch.save call in in mnist3_train.py to save the trained weights. Likewise, you can use the torch.load call in mnist3_test.py to load the weights so you don't have to retrain from scratch every time you test your program.

3. Note that when testing rather than training a model, it is important to do model.eval() to turn off the gradient computation and switch the mode of the batch normalization layer to use training set statistics instead of batch statistics. This is done in mnist3_test.py and should be done in your code as well.

4. By combining code from mnist3_test.py and Lab8.fsm you can write a Cozmo behavior that captures camera images and does digit recognition.

5. Start by assuming the paper contains a single digit. You will need to resize the camera image to 28x28 in order to fit the neural network's input requirements. See this web page for help on resizing an image using the cv2.resize() method from OpenCV. Note that since the original image is 320x240, which is not square, you can't just resize it to 28x28 because that will introduce distortion.

6. Another issue is that all the data used to train the neural net was normalized: each digit was scaled to a uniform size and centered in the image. But if you're holding up a sheet of paper to Cozmo, the digit will vary in size based on distance, and may not be centered. Therefore, you will need to write some code to find the bounding box of the digit, allow for a bit of white space around it, and rescale the resulting region to 28x28 so it looks like the training data. You can assume that the input is well-formed, i.e., there is a single digit on a white background. But your code must work on real grayscale images from Cozmo's camera, so when finding the bounding box it cannot assume that the background pixels are perfectly white, or that there is no noise in the image. Furthermore, the network was trained on light digits against a dark background; you will have to invert your camera images to match.

7. Once you are recognizing a single digit successfully, modify your code to take the current camera image, segment out the individual digits, rescale each digit to 28x28, run it through the mnist3 model you trained previously, and have Cozmo speak the digit.

8. You can assume that there is one row of digits written on the sheet, so to segment out the digits you could do something simple like apply thresholding and then scan columns of the image, or something fancier like using cv2.connectedComponents.

9. UPDATED for 2024: Your program must make what it's doing visible to the person who runs it. That means (1) displaying the thresholded image, and (2) displaying each of the 28x28 extracted digit images that it is feeding to the neural network.

Hand In

Hand in a collection of files sufficient to run your digit recognition demo without further training. That includes your fsm file, your model definition, and your saved weights. Note: do not hand in a model definition that tries to load the training set; that step wastes time and is unnecessary when running on real camera images.