The brown haze that settled over Pittsburgh and other U.S. cities last summer was merely an irritant to most residents, but for researchers at Carnegie Mellon University's Robotics Institute (RI), it was a reminder of why they are developing drones to help fight wildfires.

The smoke was generated by record-breaking wildfires in Canada. Over the course of the summer, the fires burned an area roughly the size of North Dakota and caused at least six deaths, including those of four firefighters.

Andrew Jong, who recently completed his master's degree in robotics in the School of Computer Science and is now pursuing his Ph.D., said autonomous drones promise to provide firefighters with situational awareness of these fires, which can range over thousands of acres and are subject to rapid, weather-related changes in direction and intensity. Seeing through dense smoke is one of many challenges firefighters face and poses a hurdle to using drones.

"The situation on the ground can evolve in a matter of minutes," Jong said. Better real-time information on wildfires could improve firefighting tactics, contribute to a deeper understanding of fire science, and save lives of firefighters and others caught in the fire's path, he added.

"As of now, the military-grade drones used by those fighting wildfires are high-altitude aircraft that fly far above the trees," Jong said. "They can't fly low because they can't see through smoke. We want to fly just above the trees or even below the canopy." Flying below the canopy could enable firefighters to remotely map access routes, escape routes and hot spots.

To avoid branches, power lines and other obstructions hidden in the smoke, the drones rely on multiple sensors — long-wave infrared and visible light cameras as well as radar — combined with artificial intelligence techniques to make sense of incomplete information.

Not only can this approach be used to navigate through smoke, but it can also provide "de-smoked" 3D reconstructions of homes or other structures, along with the vegetation surrounding them and even what the buildings are made of, Jong said. This would enable firefighters to triage structures, concentrating their limited resources on the buildings with the best chances of being saved. Now, firefighters must make such assessments in person and up close, putting their lives in danger.

Jong is part of a research team headed by the RI's Sebastian Scherer, Katia Sycara and Ioannis Gkioulekas that is developing the drones with support from the U.S. Department of Agriculture and the National Science Foundation.

The need for firefighting drones is great, said Josh Wilkins, a second-generation firefighter who retired after 27 years with California's San Bernardino County Fire Department and is collaborating with the CMU group.

"Typically, firefighters beginning an eight-to-12-hour shift are given a piece of paper that tells them where the fire was 12 hours ago," Wilkins said. Such dated information doesn't provide a good understanding of the battleground firefighters face, making it difficult to effectively fight fires. It can also put firefighters' lives at risk.

"I helped bury too many folks who died for lack of actionable information," Wilkins said.

One of the best-known examples of firefighter deaths emerged from the 2013 Yarnell Hill Fire in Arizona, where 19 members of the Granite Mountain Hotshots died when a sudden shift in weather patterns cut off their escape route.

The CMU team's work draws on the unique expertise of the principal investigators. Scherer, an associate research professor in robotics, previously directed a project that developed drones that could navigate smoke-filled corridors to fight fires onboard ships and headed CMU's DARPA Subterranean Challenge team that built robots that could navigate underground. Sycara, the Edward Fredkin Research Professor, contributes expertise in developing systems that enable robots to work cooperatively in teams, while Gkioulekas, an associate professor of robotics, specializes in novel imaging and sensor systems.

While it's important that the drones can fly at high speed below the canopy, it's also important that they fly intelligently, Jong said. That's why the researchers are working on informative path finding — determining which flight path will yield the most information.

Path finding can also be applied to search and rescue operations, he added. Likewise, many of the other technologies necessary for wildfire drones have broader applications in robotics.

Jong said the group has been working with the Pennsylvania Game Commission to evaluate their technologies during prescribed burns that the commission uses to manage game lands. They have tested thermal imaging cameras during these burns, successfully tracking vehicles and crew members near active fires.

The group also uses computer simulations, which mimic the thermal images of fires, to train the drones, as well as lighting campfires or using smoke generators to assess navigation abilities.

"Not only are we flying through the trees, but we also want to provide a 3D reconstruction — basically a digital twin — of what's going on," Jong said. "That reconstruction enables firefighters to explore the environment more quickly, identifying access routes, anchor points, natural fire breaks, hot spots and flare ups without putting themselves in danger."

Drones also can be used to supply real-time measurements of wind, humidity and general weather conditions.

Wilkins foresees using drones to collect information on forests prior to fires. Using drones to survey these areas, noting terrain and the presence of under-canopy vegetation, can help firefighters develop plans on how to fight fires, should they occur.

"It's good to know what's going to burn and how it's going to burn," he explained. But such forest inventories are time-consuming and difficult to perform without automation.

Wilkins said the firefighting community is often hesitant to adopt innovative technologies, and experience has shown that drones can create hazards in some situations. But he predicts that the capabilities of these new drones will eventually win them over.

"There's always resistance, particularly in the fire service," he said. "But once we show them the safety factors that have been designed into these systems and the good data that we can collect with the drones, I think we'll win them over."