Interesting Roads Encountered During NHAA

During the trip, we encountered many different road types, despite the fact we were on the same interstate, I-70, for most of the trip. Some of the roads, like the two below were like you would expect from an interstate highway, nice pavement and good lane markings.

RALPH drove at night as well as during the day, using images that looked like the one on the left below. RALPH was also able to drive in situations where the road had been freshly paved but not painted yet. In this situation, the only good feature was the boundary between the road and the off road region.

We encountered rain in Kansas, which caused pretty severe specular reflections off the pavement, obscuring the lane markings, as can be seen in these two images. The tracks from previous vehicles were sufficient to allow RALPH to continue steering accurately.

Some of the most difficult segments of the trip were when we passed through cities. Here the traffic got heavy and the lane markings were often non-existent or very hard to see. Sometimes during these segments RALPH locked onto the vehicle ahead, and steered to follow it. For example, RALPH was able to followed several trucks around almost the entire Denver beltway.

West of the Rocky Mountains, there were some stretches of really poor roads. Often the lane markers were nearly invisible due to wear. Several times we also encountered long stretches of construction where the road was basically very fine, packed gravel. Needless to say there were no lane markings during these stretches. But RALPH was able to key off the variations in appearance of the packed gravel and the loose gravel around it and continue driving autonomously.

The freeways in California were very interesting. Instead of having painted lane markings to delineate lanes, they had reflectors that were nearly invisible during the day (see left image below). But RALPH was able to key off the discoloration from the oil spot down the center of the lane. In fact one of RALPH's longest uninterrupted segments (56 miles) was on pavement like this as we approached San Diego. We also drove on the I-15 HOV lane into San Diego, and like the rest of the California freeways, it had reflectors marking the lane boundaries. However it also had a strong boundary between the cement road surface and the asphalt shoulder, making driving pretty easy for RALPH.

Questions and Answers about the Road Images

Were you able to see what happened when Ralph drove into a sunset?

Yes. We experienced this effect most dramtically coming out of the mountains, near Grand Junction CO., at about 7:00 pm. Although the camera mount is equipped with a makeshift sun visor which shields some of the glare, the system is still more sensative in these conditions. Basically, the camera iris has problems adjusting properly. Glare off the road isn't too much of a problem though, because the parts of the road which give you the most glare are generally consistent - like the oil spot down the middle.

Did the pictures of urban driving show Ralph locking onto the truck ahead, or was it finding the lane edges? I couldn't tell from the red trapezoid.

These pictures showed RALPH locking onto the truck, doing what we call vision-based platooning. You can see this by looking at the lowest feature template, which is the third one below the preprocessed image in the lower left corner of the truck images. This shows the dark "feature" near the middle of the template, caused by the rear of the truck. (Which is just inside the red trapezoid in the high res image.) This is similar to what you see when there is an oil spot down the middle of the road. (The reason we know that RALPH is using the truck and not the lines is that if the truck changes lanes, RALPH will follow it - is this a feature or a bug? )

Did Ralph lock onto the trucks by happenstance, or did you train it to do so?

Ralph locks onto or adapts to whatever features are present. There is no training in the sense of ALVINN. If we pull behind a truck slowly, RALPH can adapt to following it. Opposite goes for if we slow down.

In all of the pictures, how does the red trapezoid relate to what Ralph is keying on?

The area inside the red trapezoid is resampled and preprocessed into the "birds-eye" image in the lower left corner. Basically, we use geometry to project the pixels so that the image looks as if it was taken by a camera looking straight down onto the road.

Does Ralph lock onto trucks only when it can't find other, more expected features, like road edges, lane markers, oil slicks?

RALPH locks on to truck if they represent a consistent feature of the "road." This is not necessarily the best thing to do if there are clear lane markings, but for city driving, where markings are poor or confusing, this behavior is beneficial.

Is there a danger of Ralph following a truck onto an exit ramp?

Yes, but I wouldn't call it a "danger." We do have the ability to supress certain parts of the image to bias RALPH to take or ignore exit ramps in normal conditions, but haven't tested this extensively following trucks. An interesting concept that we will probably investigate is moving the red trapezoid forward and backward in response to things like trucks which are in the viewing field. By bringing the viewing field in, you may be able to image just the road and not the truck, or by moving it out, you may be able lock onto another vehicle (cars work fine too) to get you through a tough environment. We already move the trapezoid in and out based on speed, we just need an obstacle detection sensor, like radar to test the other ideas.