26 August

I had the honor of speaking at a working at MIT yesterday on the testing of robotics systems. My own talk was heavily based our experiences testing our robotic boat, robot plane and underwater vehicles. The field trials we do involve a range of challenges in terms of experiment planning, but also including logistics, safety and team management.

The workshop overall was motivated by a project called PATFrame that deals with understanding and optimizing test planning, especially for large military-sponsored robotics projects. Other participants from the US government and military spoke about the challenges of testing adaptive systems that both change over time, and also exhibit performance which depends on (variable) features of the environment. In short, it is essentially impossible to cover al possible conditions when a system is so versatile and complex.

As intelligent systems gradually approach the complexity and richness of humans, testing them definitively will become as tricky as testing people. Even when you think you know a person very well, they can exhibit surprising behavior and there is no way to preclude the same phenomena from robotics systems. This poses a challenge not only for testers, but for society at large.

Robot Attack
Robot Attack: did we test it enough

By Gregory Dudek at | Read (2) or Leave a comment |    
Re: Testing Robotics Systems

The time robots will truly behave in an interesting unexpected way we will have made a big leap in robotics (and AI!). If you are talking about bugs, I agree with you. Bugs in robotics (either software or hardware) are tricky, as most of them depends on some non-repeatable-outside-world input.

But if a bug (!?) will turn out to be an unexpected behaviour, something the designer will have to devote time to investigate... well, that's the topic or a whole research project.

And maybe a film! (Short circuit?)

Posted by: Lorenzo Riano at August 28,2010 07:37
Re: Testing Robotics Systems

Lorenzo, I think it is fair to say that today we are seeing robot systems in the field that exhibit both bugs as well as other unexpected behaviors that may or may not count as bugs. These other behaviors include phenomena known as "emergent behaviors" that result from the combination of two or more algorithms or behaviors to produce a new one that was not clearly anticipated. Even the Roomba robot exhibits simple forms or emergent behavior. In practice, with robotic systems in extensive use (for example thousands of explosive disposal (EOD) robots in the ground in Iraq) this is a real issue here an now.

Clearly predicting such emergent behavior is very hard and a research problem, but that does not eliminate the need to estimate and determine risk factors NOW. Even in cars this problem needs to be done every day despite the fact that some phenomena cannot actually be verified analytically.

Posted by: dudek at August 31,2010 20:16
Please send trackback to:/blog/201/tbping
There are no trackbacks.
Post your own response

Each comment is manually screened for the presence of appropriate and substantive content, due to a constant onslaught of comment-spam. This means there may be a delay before your comment appears.

(Some kind of name is required, will be visible)

Required, whatever you enter will be visible to other users.

(Optional, used for "mailto" link)

Your email address is not required, but if you insert it it will be displayed so people can contact you.

Answer this question correctly to demonstrate that you are not a dumb spambot.

The title for your comment.

Your comment goes here. All relevant comments are welcome, except for those that simply promote an irrelevant product or else are used to fraudulently inflate the link count to an irrelevant web page. They appear after moderation. Don't forget to also fill in the captcha below or your text will be rejected automatically!

You must answer this question to prove you are human
What kind of restaurant has waiters, napkins, and pizza (enter a number): 1:Seafood, 2:Drive-through, 3:Pizzeria, 4:Reform school, 5:Roadside hot-dog stand?

Answer this question correctly to demonstrate that you are not a dumb spambot.