Researchers at the Massachusetts Institute of Technology have developed a robot that uses radio waves capable of penetrating walls to sense occluded objects. This robot, named RF-Grasp, combines this powerful sensing capability with more traditional computer vision to locate and grasp items that might otherwise be invisible. This advancement could one day simplify e-commerce fulfillment in warehouses or help a machine pull a screwdriver from a cluttered toolbox.

The research will be presented in 5 at the IEEE International Conference on Robotics and Automation. The lead author of the paper is Tara Boroushaki, a research assistant in the Signal Kinetics Group at the MIT Media Lab. Her MIT co-authors include Associate Professor Fadel Adib, director of the Signal Kinetics Group, and Associate Professor Alberto Rodriguez, a 1957-year associate professor in the Department of Mechanical Engineering. Other co-authors include Research Engineer Lengshan Leng from Harvard University and PhD student Ian Clester from Georgia Institute of Technology. You can read the paper here (PDF).

As e-commerce continues to grow, warehouse work often remains the domain of humans rather than robots, despite sometimes hazardous working conditions. This is partly because robots struggle to locate and grasp objects in such crowded environments.

"Perception and picking are two bottlenecks in today's industry," said Rodriguez. With optical vision alone, robots cannot perceive the presence of items packed in boxes or hidden behind another object on a shelf—visible light waves certainly do not pass through walls.

But radio waves can.

For decades, radio frequency (RF) identification has been used to track everything from library books to pets. RFID systems have two main components: a reader and a tag. The tag is a miniature computer chip attached to the item to be tracked (or implanted, in the case of pets). The reader then emits an RF signal, which is modulated by the tag and reflected back to the reader.

The reflected signal provides information about the location and identity of the tagged item. This technology has gained popularity in retail supply chains—Japan aims to use RF tracking for nearly all retail purchases within a few years. Researchers realized that the proliferation of RF is a boon for robots, giving them another mode of perception.

"RF is a different way of sensing than vision," said Rodriguez. "It would be a mistake not to explore what RF can do."

RF Grasp uses a camera and an RF reader to find and grasp tagged objects, even when they are completely blocked from the camera's view. It consists of a robotic arm with an attached gripper. The camera sits on the robot's wrist. The RF reader stands independently of the robot and relays tracking information to the robot's control algorithm. Thus, the robot continuously collects RF tracking data and visual images of its surroundings. Integrating these two data streams into the robot's decision-making is one of the biggest challenges faced by the researchers.

"The robot must decide at each time step which stream is more important," said Boroushaki. "This is not just eye-hand coordination, but RF eye-hand coordination. Therefore, the problem becomes very complex."

The robot initiates the search process by performing a ping operation on the RF tag of the target object to determine its whereabouts. "First, use RF to focus visual attention," said Adib. "Then, you can use vision for fine manipulation." The sequence is similar to hearing a siren from behind, then turning to see the source of the siren clearly.

With its two complementary senses, RF Grasp zeros in on the target object. As the distance decreases and even as it begins to manipulate the item, vision, which provides better detail than RF, controls the robot's decisions.

RF Grasp demonstrated its efficiency in a series of tests. Compared to similar robots equipped only with cameras, RF Grasp was able to precisely locate and grasp its target object with approximately half the total motion. Furthermore, RF Grasp has the unique ability to "tidy" the environment—clearing packaging materials and other obstacles to approach the target. Rodriguez stated that this demonstrates the "unfair advantage" of RF Grasp over robots without penetrative RF sensing. "It has guidance that other systems simply do not have."

RF Grasp could one day perform tasks in packed e-commerce warehouses. Its RF sensing can even instantly verify an item's identity without manipulating the item, exposing its barcode, and then scanning it. "RF has the potential to improve certain limitations in the industry, especially in perception and localization," said Rodriguez.

Adib also envisioned potential home applications for the robot, such as finding the right hex key to assemble an IKEA chair. "Or you can imagine the robot finding lost items. Like a super Roomba, it can retrieve my keys no matter where I put them."