Using Function Optimization To Find Policies: Walking

This assignment explores using function optimization to do policy optimization.

Many examples of doing this are on Jack Wang's web page. A local example is Seungmoon Song. More interesting examples are on Michiel van de Panne's web page. If you want to use some other walking model, or do a quadruped, go ahead. OpenAIGym has models you can use: 2D Box, 2D Box harder, 2D Mujoco, and 3D Mujoco.

We are distributing a simple simulation in C that works on both Windows and Linux/Mac. Look for a readme.txt file in each directory.

The code including Linux ODE graphics (drawstuff) libraries.
Windows 32 bit ODE graphics (drawstuff) libraries

Graphics is provided in a separate animation program, and uses the graphics package from ODE. You can use the mrdplot/matlab plotting software provided, or some other plotting software like gnuplot.

Quick start for Linux:

Download and uncompress walk05.zip
cd walk05/walk
make - you will need to switch the drawstuff library to 32 bit if you are using 32bit Linux.
./simulate
The output d0xxxx is a data file that can be animated or plotted. To animate a data file run
./animate
We use mrdplot running in Matlab to plot data. This will be demo-ed in class.

Quick start for Windows:

You need to get the ODE graphics (drawstuff) libraries either by installing ODE or (for 32 bit) using the Windows 32 bit ODE graphics (drawstuff) libraries

Let us know what has to be done to get this code working on a MAC.

Part 1: Implement an optimization procedure to optimize simulation parameters to minimize the cost of a 20 second simulation. The parameters you can modify are listed in walk05/readme.txt

This part is similar to: Optimizing Walking Controllers. Optimizes parameters for Simbicon-like controller.

Part 2: Optimize for a robust gait that can handle large random white noise torso perturbations. What is the largest torso perturbation variance you can handle? What happens to performance on the unperturbed case?

This part is similar to: Optimizing Walking Controllers for Uncertain Inputs and Environments. Optimizes robustness for Simbicon-like controller.

What optimization software should I use? There is a version of CMA-ES for C. The Gnu Scientific Library provides some. A test program for the GSL Amoeba (Nelder-Mead) optimizer is available here. It includes instructions on how to install GSL on Windows. You can do this in Matlab and call the C-code-based simulation from Matlab. There are many other sources of optimization routines available free on the web.

We use SDFAST to generate the dynamics for the robot. If you need to, you can get a manual from www.sdfast.com. Hopefully, you should not need it. SDFAST generates C code which is in the directory sdfast/. sdfast/xxx.sd specifies the default kinematic and dynamic parameters of the robot. sdfast/xxx.h defines some useful constants for C

If you have problems using the graphics (which are only used by animate.c) try installing ODE.

What to turn in?

You can work in groups or alone. Generate a web page describing what you did (one per group). Include links to your source and any compiled code in either .zip, .tar, or .tar.gz format. Be sure to list the names of all the members of your group. Mail the URL of your web page to cga@cmu.xxx and snagaval@andrew.xxx. [You complete the address, we are trying to avoid spam.] The writeup is more important than the code. What did you do? Why did it work? What didn't work and why?

Questions

Here are some tips to get started under Windows (2015 info):

0. Install Visual Studio 2010 (or VS 2013).
1. Download and unzip walk05.zip.
2. Within folder walk05/, there should be 2 sub-folders walk05/useful/ and walk05/walk/. Create a third sub-folder walk05/lib/.
3. Download and unzip walk05-windows.zip into walk05/lib/. Now you should have folders walk05/lib/DebugDLL/, walk05/lib/DebugLib/, walk05/lib/ReleaseDLL/, walk05/lib/ReleaseLib/.
4. If necessary, enter folder walk05/walk/ and delete executables animate.exe and simulate.exe.
5. Open project simulate.vcproj and follow the steps to convert to a VS2010 project.
6. Open project animate.vcproj and follow the steps to convert to a VS2010 project. Open file animate.c. Change line 266 to give the correct path to textures on your machine. For example: fn.path_to_textures ="c:/users/sasanka/walk05/useful/drawstuff/textures";
7. Build and run simulate.vcproj. You should see it print out some messages in a command window as it simulates.
8. Build and run animate.vcproj. You should see an animation showing the robot walking. The header file drawstuff.h is in walk05/useful/drawstuff-windows/. The provided drawstuff libraries have been tested and work fine on a 64-bit machine if you use MSVC. There should be a copy of drawstuff.dll in the walk05/walk/ folder. As long as drawstuff.dll is in the same folder as animate.exe it should work. If you are running 32-bit Windows, please copy *.dll from walk05/lib/ReleaseDLL/ to walk05/walk/.
9. Now you can open up controller.c in simulate.vcproj and apply optimization techniques to complete the assignment. A cost function is provided. See readme.txt for more details. learn.c provides a simple model for running multiple simulations and improving the performance.

Here are some tips to get started on a MAC (OS X 10.10.2) running Xcode (2015 info):

It seems Xcode under OS X 10.10.2 may symlink gcc to clang, and clang builds C code by default in GNU C11 mode (same as GCC version 5 and newer).ï¿½ The sdfast code is not C99 compatible.
So try adding the following compile flag and rebuilding the sdfast stuff: -std=gnu89

Here are some tips to get started on a MAC (OS X) running VirtualBox Ubuntu (2015 info)

The following packages are missing in Ubuntu: -lX11, -lGL, -lGLU, -lstdc++.
```
$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ sudo apt-get install libx11-dev libglu-dev g++
```
If you still get an error for '-lGL', assuming you are running Ubuntu 14.04 LTS (x86-64), you may also need to install the appropriate one of the following packages to provide libGL.