Exploring Tekkotsu Programming on Mobile Robots:

Simulator Mode

Prev: Vision
Up: Contents
Next: Visual Routines

Contents: Purpose, Compiling the simulator, Running the simulator, Image and posture files, Image/posture sequences, Debugging with GDB, Injecting events, Setting breakpoints

Purpose of Simulator Mode

Running Tekkotsu in simulator mode allows you to run behaviors on your computer instead of on the robot, using pre-recorded sensor data. This provides several benefits:

Compiling for Simulator Mode

For Linux-based robots, you don't have to do anything special to use Tekkotsu in simulator mode. It's the same code, whether you run it on the robot or on your workstation. (For AIBOs, you must compile for the simulator by typing "make sim" in your project directory, and the result will be an executable file called tekkotsu-ERS7.)

The project/Environment.conf file defines both a target model (TGT_CREATE for the Create/ASUS robot, TGT_ERS7 for the AIBO, etc.) and a default target platform. For Linux-based robots like the Create and Chiara, the target platform is always PLATFORM_LOCAL. The target platform is PLATFORM_APERIOS for the AIBO, but the for the simulator it is PLATFORM_LOCAL, since the simulator runs on your local host. Typing "make sim" forces the target platform to PLATFORM_LOCAL for the current compilation, so that you don't have to edit the Environment.conf file.


Running in Simulator Mode

Once the make has completed successfully, you can run Tekkotsu in simulator mode from your project directory and talk to it via its command line interface. You can talk to the robot it is simulating by using the ControllerGUI.

Talking to the Simulator

  1. In your project directory, type "./tekkotsu-CREATE" (or "./tekkotsu-CHIARA" or whatever corresponds to the type of robot you are using) and hit return.

  2. Tekkotsu will print out a bunch of start-up information. When this ends. hit return again and you'll see the Tekkotsu command prompt, "HAL:Create>". (HAL stands for Hardware Abstraction Layer.) You don't need to know any commands to make basic use of the simulator, so we won't go into these commands now.

  3. Start up the ControllerGUI. For a hostname, give it "localhost", or the name or IP address of the computer on which you're running the simulator.

  4. In the ControllerGUI, go to Root Control > Mode Switch, and start the Hello World behavior. You will see a bunch of lines print out on the Tekkotsu console, showing output that was sent to cout, serr, and several other streams. On the AIBO you would have to telnet to port 59000 to see these messages, but with the simulator this is not necessary.

  5. The full Tekkotsu functionality is available to you through the ControllerGUI, but not everything works the same as on the robot. For example, you can go to Root Control > Status Reports > Battery Check, and a battery status report will be printed in the Tekkotsu console window, but all the values will be zero. The simulator doesn't bother simulating the robot's power limitations.

  6. The cleanest way to exit Tekkotsu is to type ^C (control-C). There are other ways to exit, such as by typing "quit" or "exit". or ^D (control-D), but depending on what Tekkotsu is doing at the time, these can cause a lot of error messages to be displayed.

  7. If the simulator freezes or has a messy crash, and you can't use ^D or ^C to exit, type the suspend character, ^Z, and then type "kill -9 %" to kill the job.

Note: sometimes when the simulator exits abnormally, some of its processes are left lying around. If that happens, the next time you run the simulator you may experience interference from these old processes, e.g., ControllerGUI may be unable to connect. Use the command "ps -a" to check which processes are running, and do "killall -9 tekkotsu-CREATE" to kill any lingering ones.


Supplying Image and Posture Files

You can supply the simulator with a series of camera images to use as input to your behavior. Currently these must be JPEG files in RGB format or PNG files in YUV format. Vision applications often want to know the camera pose at the time each image was taken, so the simulator also provides a way to load posture files.

Recording Image and Posture Data

  1. You'll want to record your images at full resolution, so begin by starting up the ControllerGUI and double clicking on the "Take Snapshots" script. (See the section on Collecting images in the Color Image Segmentation chapter for details on what this script does.)

  2. Start up the Raw Cam viewer, or kill it and restart it if it was already running when you ran the Take Snapshots script. Point the robot's camera at something interesting.

  3. Make a temporary directory to hold the image and posture files.

  4. In the Raw Cam viewer, Click on "Freeze Frame" and then "Save Image". Check the box that says "Save joint positions as well". Then specify a filename in the temporary directory you created, such as pic01.jpg. Be sure to store your data in RGB format if using JPEG files, or YUV format if using PNG files.

  5. Click "Unfreeze", and repeat the steps above to take as many additional pictures as you like.

  6. When you've finished collecting data, look in the temporary directory. You should see a series of .JPG or .PNG files, and a parallel series of .POS files. These posture files are human readable, but they use a more compact format than the ones created by the Posture Editor.

By default, logged data (both images and postures) are taken from the ~/project/logged directory. Create a ~?project/logged directory now if you don't already have one.

Using a Single Stored Image with the Simulator

  1. Move a single JPEG or PNG file and its associated posture file into the ~/project/logged directory. Make sure there is only one file of each type in the directory.

  2. Start up Tekkotsu in simulator mode by typing:
       ./tekkotsu-CREATE -c sim.plist
    

  3. Connect to the simulator with the ControllerGUI.

  4. Click on the Raw Cam or Seg Cam viewer and the image should pop up on your screen.

  5. In the ControllerGUI, go to Root Control > File Access > Posture Editor, and you will see the joint positions that were loaded from the posture file.

If you put multiple files in the project/logged directory, the simulator will loop through them, in alphabetical order, at 30 frames per second. Try it and see. There are ways to slow or stop this process, discussed below.


Image/Posture Sequences

The robot can record camera images at 30 frames/second. New sensor frames arrive at a similar but not necessarily identical rate. Thus, the simulator does not expect a one-to-one correspondence between image and posture files. It loads the files in the images and sensors streams independently. For each stream, the files are loaded in alphabetical order. If you want to test the robot's response to a moving stimulus, or to a stream of images taken as the head moves around, you can record such a sequence by not clicking on "Freeze Frame"; in that case the Save button is labeled "Save Image Sequence" instead of "Save Image".

But for some kinds of experimentation you will want to cycle through a handful of carefully selected images, and for each image, you will need to know the camera pose at the time the image was taken. In this case you want to force the image and sensor streams to stay in synch, and you want to be certain that the corresponding sensor data have been loaded just before each image becomes available for processing. To do this, you will need to adjust the frame rate of the sensor stream so that it matches the image stream, and you will need to set the start time of the sensor stream so that it just precedes the start time of the image stream. Then both streams will remain in synch.

Using Multiple Stored Images with the Simulator

  1. Move several of the image and posture files you recorded previously into the images/ subdirectory.

  2. Invoke the simulator with the following command line switches. freeze tells the simulator not to move to the next image until you give the advance command. Setting Vision.Verbose=true causes the simulator to announce each image file as it's loaded.
       ./tekkotsu-Create -c sim.plist freeze Vision.Verbose=true
    Note: you can also take these actions from the simulator's command line, using the commands freeze and set Vision.Verbose=true. However, if the simulator starts up in un-frozen mode, the image and posture files will already be out of sync.

  3. Start up the ControllerGUI and turn on the Raw Cam viewer. Type status to see which files the simulator will load next.

  4. Type advance in the simulator and you should see the first camera image. (Alternatively, go to Root Control > Vision Pipeline in the ControllerGUI and double click on "Advance Frame".) Each time you type advance or double click on "Advance Frame", the image and sensor information should advance again, and eventually loop back to the beginning of the sequence.

Explore more:

  1. Turn the "Advance Frame" command into a ControllerGUI script so you can invoke it from the Scripts menu.

  2. Type set to see the current settings of all simulator variables.

  3. Try set Speed=0.025 using the simulator's command line interface. What do you see in the Raw Cam window?

  4. Boot the robot and use the Walk Remote Control to drive the robot around. While doing so, record a 10-15 second image sequence using the RawCam's "Save Image Sequence" function. Be sure to check the box to record posture files as well. (You must specify a filename prefix, like "myseq", in the Save Image Sequence dialog box, and the ControllerGUI will generate files myseq000.jpg, myseq033.jpg, and myseq000.pos, myseq031.pos, etc.) Move these files into the ~/project/logged directory and play them back at half realtime by setting Speed=0.5.


Debugging with GDB

GDB is the Gnu debugger. There are several good GDB tutorials on the web. You can also learn about gdb by typing "gdb" to run the program, and then typing "help" at the (gdb) command prompt.

In this section we're going to write a program that can crash the robot. Debugging such programs is difficult because there is no way to examine the program's state after Tekkotsu crashes. But with gdb we can examine the execution stack and determine the cause of the crash.

#include <iostream>
#include <sstream>

#include "Behaviors/StateMachine.h"

using namespace std;

$nodeclass DstBehavior : StateNode {
  virtual void doStart() {
    cout << getName() << " is starting up." << endl;
    erouter->addListener(this,EventBase::textmsgEGID);
  }

  virtual void doEvent() {
    string* menu[4];
    menu[1] = new string("appetizer");
    menu[2] = new string("main course");
    menu[3] = new string("dessert");

    const TextMsgEvent &txtev = *dynamic_cast<const TextMsgEvent*>(&event);
    const string &userinput = txtev.getText();
    istringstream ins(userinput);
    int item;
    ins >> item;
    cout << "Now serving item " << item << "."
	 << "  Enjoy your " << *menu[item] << "!" << endl;
  }
}

REGISTER_BEHAVIOR(DstBehavior);

First we'll demonstrate the crash:

Crashing DstBehavior

  1. Compile Tekkotsu with the version of DstBehavior given above; you will have to add it to UserBehaviors.h.

  2. Run Tekkotsu, and start up the ControllerGUI.

  3. Activate the DstBehavior.

  4. In the ControllerGUI "Send Input" box, type the following:
        !msg 1
    
    This should produce correct output.
  5. Now send the following input:
        !msg barf
    
  6. Tekkotsu will crash and print out a bunch of information. Try running it again and supplying the following input:
        !msg 0
    

Now we will use gdb to determine the cause of the crash.

Diagnosing Crashes with GDB

  1. In order to use gdb with Tekkotsu, you simply run gdb and pass it the name of the Tekkotsu executable as an argument:
    cd ~/project
    gdb tekkotsu-CREATE
    
  2. When gdb starts up, you can set breakpoints, which we'll get to later. Then you must type "r" or "run", and it will start Tekkotsu.

  3. Use the ControllerGUI to activate DstBehavior.

  4. Send the input !msg 3 and verify that the correct output is produced on the Tekkotsu console.

  5. Send the input !msg barf. You should see error messages, beginning with "Program received signal SIGSEGV, Segmentation fault." This means the program tried to access a nonexistent memory location. The ControllerGUI will no longer respond. Tekkotsu has crashed. gdb will show its best guess as to the line in the program where the error occurred. (When statements are spread across multiple lines, this guess might be off by a line or two.)

  6. Now try out the following gdb commands:

    bt   backtrace: show the execution stack
    p item print the value of the variable named 'item'
    p userinput print the value of 'userinput'
    p menuprint the array 'menu'
    p menu[2]print element 2 of 'menu'
    p *menu[2]print the string pointed to by menu[2]
    i locinfo locals; show local variables in this stack frame
    llist: show source lines
    l -list backwards: go earlier in file

  7. Use the "help" command to view the documentation on each of the above commands, e.g, "help p" shows the documentation for the print command.

  8. Type "q" to quit gdb.


Injecting Events via ControllerGUI

Using the ControllerGUI's !post command, you can manually inject events into either the AIBO or the simulator. Not all event types are supported at present. Most importantly, timer events are not supported. But other simple event types such as button presses, statemachine events, and audio events are supported.

Here's a little demonstration program that responds to head button press events by speaking a digit:

#include "Behaviors/StateMachine.h"

$nodeclass DstBehavior : StateNode : counter(0) {
  int counter;

  virtual void doStart() {
    erouter->addListener(this,EventBase::buttonEGID,GreenButOffset,EventBase::activateETID);
  }

  virtual void doEvent() {
    advance_the_counter();
    char filename[30];
    sprintf(filename,"numbers/%d.wav", counter);
    sndman->PlayFile(filename);
  }

  void advance_the_counter() {
    if ( ++counter > 9 )
      counter = 0;
  }

}

We will use this sample program to illustrate event posting from ControllerGUI. The three arguments to the !post command are the event generator (buttonEGID), the source (GreenBut for the green button), and the event type (A for Activate, S for Status, or D for Deactivate).

Injecting Button Press Events

  1. Compile the demonstration program above, and start the simulator.

  2. Connect to the simulator with ControllerGUI, and activate DstBehavior.

  3. In the ControllerGUI's "Send Input" window, type the following:
       !post buttonEGID GreenBut A
    

  4. The simulator should display the message "Playing numbers/1.wav". (The simulator does not yet have sound support, so it won't actually play the audio files.)

  5. In the ControllerGUI's "Send Input" window, press the up-arrow key to recall the previous command, and hit Return.

  6. The simulator should display the message "Playing numbers/2.wav".

Explore more:

  1. The !post command can take a numeric source id instead of the symbolic value "GreenBut". Suppose you wanted to send an event indicating a yellow button press. What numeric value should you use for the source id? (Hint: look in CreateInfo.h for the Create, or the corresponding file for whatever robot you are using.)

  2. Modify the program to accept all types of button press events, and modify processEvent to print out the result of the event's getDescription() function. Then try sending different kinds of button press events to your program by varying both the source and the event type.


Setting a Breakpoint

Prev: Vision
Up: Contents
Next: Visual Routines