Activities

Online Posts and Relevant Sites:  informal communication and links to resources

• EURON GEM Sig Website
• Reproducible Robotics Blog Post

Workshops and Meetings: create a sustainable culture of performance evaluation and benchmarking; provide forums for exchanging ideas and approaches.

• Technical Committee Town Hall Meeting, ICRA 2015, Stockholm, Sweden
• Proposals for experimental protocols for Robotics Research, RSS Workshop , June 27, 2013, Berlin, Germany
• Metrics of sensory motor integration in robots and animals, Workshop, October 12, 2012, Vilamoura, Algarve, Portugal, Half-day IROS2012 workshop
• Safety, Security, and Rescue Robotics Summer School, September 10-14, 2012 Alanya, Turkey. Students and early career researchers experience test method development process first-hand.
• From theory to practice of performance comparison and result replications in Robotics Research, Workshop, Australia - July 12, 2012, Sydney, NSW, Australia RSS 2012 workshop
• Conditions for Replicable Experiments and Performance Comparison in Robotics Research, Workshop, Minneapolis MN USA, Full-day ICRA 2012 workshop.
• Performance Metrics for Intelligent Systems (PerMIS'12) March 20-22, 2012, College Park, MD Three-day Workshop
• Towards Replicable Experiments in Robotics Research, San Francisco, CA USA, ICRA 2011 Full-day Workshop
• Performance Evaluation and Benchmarking for Intelligent Robots and Systems with Cognitive and Autonomy Capabilities , Taiwan, IROS 2010 Half-day Workshop
• Performance Metrics for Intelligent Systems (PerMIS'10) Three-day Workshop September 30, 2010 - Baltimore, MD
• Benchmarking Intelligent (Multi-)Robot Systems August 20, 2010 - Lisbon, Portugal, Full-Day Workshop
• Good experimental methodology in robotics and replicable robotics research June 28, 2010 - Zaragoza, Spain, RSS 2010 Half-day Workshop.
• The Role of Experiments in Robotics Research Anchorage, Alaska, USA, ICRA 2010 Full-day Workshop.
• Performance Evaluation and Benchmarking for Next Intelligent Robots and Systems St. Louis, MO, USA Full-Day IROS'10 Workshop
• Performance Metrics for Intelligent Systems (PerMIS'09) Workshop, September 21, 2009 – USA Three-day Workshop
• Good Experimental Methodology in Robotics Workshop June 28, 2009 - Seattle, Washington, USA, RSS 2009 Full-day Workshop

Major Publications (not counting Proceedings from above workshops): provide references for researchers seeking information on performance evaluation and benchmarking.

• IEEE Robotics and Automation Magazine Special Issue, "Replicable and Measurable Robotics Research", Bonsignorio, F., del Pobil, A., eds., Volume 22, Number 3, September 2015
• "Toward Replicable and Measurable Robotics Research [from the Guest Editors]", IEEE Robotics and Automation Magazine, Bonsignorio, F. and del Pobil, A., Volume 22, Number 3, pp 32-35, September 2015
• "Distributed and Adaptive Shared Control Systems: Methodology for the Replication of Experiments", IEEE Robotics and Automation Magazine, Stoelen, M. F., de Tejada, V. F, Huete, A. J., Balaguer, C., and Bonsignorio, F., Volume 22, Number 4, pp 137-146, December 2015
• International Journal of Intelligent Control and Systems (IJICS) Special Issue "Quantifying Performance of Intelligent Systems," Madhavan, R., Tunstel, E., Messina, E., eds., Volume 16, Number 2, June 2011.
• Quantifying Performance of Intelligent Systems," International Journal of Intelligent Control and Systems Special Issue; Volume 16, Number 2, June 2011
• Performance Evaluation and Benchmarking of Intelligent Systems Edited Book Volume, Raj Madhavan, Edward Tunstel and Elena Messina (eds.), Springer, ISBN: 978-1-4419-0491-1, September 2009.
• Autonomous Robots Journal, Special Issue (R. Madhavan, C. Scrapper, and A. Kleiner (Eds.)) November 1, 2009 - New York
• Performance Evaluation and Benchmarking of Robotic and Automation Systems [TC Spotlight] Madhavan, R.; del Pobil, A.P.; Messina, E.; Robotics & Automation Magazine, IEEE, Volume: 17 , Issue: 1; Publication Year: 2010 , Page(s): 120 – 122.

Competitions:

These put into action the benchmarks and performance evaluation methods. By organizing and promoting competitions, we put into practice what is discussed during workshops and in publications. Competitions validate the approaches proposed and expose members of the community (especially students) to the concepts. Some of the metrics and test methods are being advanced into draft standards (e.g. the 6DOF localization from the Perception Challenge formed the technical foundation for an ASTM standard being balloted under E57 Committee on 3D Imaging Systems).

• ICRA'12 Mobile Microrobotics Challenge (MMC) Minneapolis, USA
• ICRA'12 Solutions in Perception Challenge (SPC) Minneapolis, USA
• ICRA'12 Virtual Manufacturing Automation Challenge (VMAC) Minneapolis, USA
• ICRA'11Mobile Microrobotics Challenge (MMC) May 12, 2011 - Shanghai, China
• ICRA'11Solutions in Perception Challenge (SPC) May 11, 2011 - Shanghai, China
• ICRA'11 Modular and Reconfigurable Robot Challenge (MRRC) May 10, 2011 - Shanghai, China
• ICRA'11 Virtual Manufacturing Automation Challenge (VMAC) May 10, 2011 - Shanghai, China

List of outreach activities outside the RAS

The TC has coorganized events with Euron and workshops at RSS. Other initiatives have been carried out in collaboration with the RoboCup organization and in conjunction with the US Department of Homeland Security (DHS). The RoboCup Rescue International competitions rely on quantifiable measures of robot performance within draft and existing test methods for evaluating their ability to assist in disaster response. The best-performing robot teams are invited to participate in the NIST-DHS Response Robot Evaluation Exercises, typically held every 18 months at Disaster City, a major responder training facility in the United States.