15-494/694 Cognitive Robotics Lab 7:
PyTorch and Neural Networks

I. Software Update, SDK Update, and Initial Setup

At the beginning of every lab you should update your copy of the cozmo-tools package. Do this:

$ cd ~/cozmo-tools
$ git pull

II. Experiments with the MNIST Dataset and Linear Models

1. Make a lab7 directory.

2. Download the files mnist1.py, mnist2.py, mnist3.py into your lab7 directory.

3. Read the mnist1.py source code and skim the source code. This is a linear neural network with one layer of trainable weights.

4. Run the model by typing "python3 -i mnist1.py". The "-i" switch tells python not to exit after running the program. Press Enter to see each output unit's weight matrix, or type control-C and Enter to abort that part.

5. Try typing the following expressions to Python:
   • model
   • params = list(model.parameters())
   • params
   • [p.size() for p in params]
   The first parameter is the 784x10 weight matrix; the second one is the 10 biases.

6. How long did each epoch of training take, on average? ________________

7. Modify the model to use the GPU instead of the CPU. (You just have to uncomment one line and comment out another.)

8. Run the model on the GPU. How long does each epoch take now? ________________
   Are you surprised? GPUs don't help for small models. A few thousand weights is small.

9. Skim the code for the mnist2 model. This model has a hidden layer with 20 units. Each hidden unit is fully connected to the input and output layers.

10. Run the mnist2 model. How long does each epoch of training take, on average? ________________

11. You can use the show_hidden_weights() and show_output_weights() functions to display the learned weights.

12. Modify the mnist2 code to run on the GPU. How long does each epoch take now? ________________

III. Experiments with the MNIST Dataset and a Convolutional Model

1. Skim the code for the mnist3 model.

2. Run the model. You can ignore the "THCudaCheck FAIL" message. Look at some of the kernels the model learns.

3. What are the parameters of this model? ________________________________________________
   How many weights does it have? ________________

4. How many parameters does this model have, where each parameter is a tensor? ________________
   How many total weights are in the model? (Show your calculation.) ________________________________________________

5. This model runs on the GPU. How long did each epoch of training take, on average? ________________

6. If you modify the model to run on the CPU, how long does an epoch take now? (You don't need to run the model to completion.) ________________

IV. Homework Problem: Partial Cube Detection

The file cube_cropped.zip contains images of all three light cubes at 10 different distances. There are images with the cube directly facing the camera, and images with the cube rotated by 45 degrees. The background has been cropped out and replaced with solid magenta (RGB of 255,0,255). Examine some of these images with "display" or "gimp", or you can browse them here. You can use cv2.imread() to read images into Python, and plt.imshow() to display them.

Your task is to develop a neural net that detects partially observable cubes given a 320x240 grayscale image as input. Your network should output a 1 if there is a cube fully or partially visible in the left half of the image; otherwise it should output a 0. We won't worry about the right half for now.

In order to train your network you will need a much larger training set than the 60 images we've provided to you. You will need to show it cubes shifted by various amounts. You will also need to show it cubes against various backgrounds so the network learns to discriminate cube features from non-cube features.

Do some experiments to see what cube detection might look like with a convolutional neural net. We'll discuss this further next week.

Hand In

Nothing yet. Stay tuned.