This is a challenging but an exciting project from the Organisation for Economic Co-operation and Development (OECD) that involves a team of collaborators from the United Kingdom and Spain.
I worked alongside other research assistants with Anthony Cohn, Professor of Automated Reasoning at the University of Leeds, working in AI and robotics, and
Professor José Hernández-Orallo of Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València.
In this project, we are working on a framework to categorise what we refer to as “Evaluation Instruments (EIs)” (which subsume Benchmarks and Competitions). The framework consists of meta-informations and 18 “facets” that are used to evalaute Artificial intelligence (AI) agents and robots.
For more detail on this project, check Educational Research and Innovation , On evaluating artificial intelligence systems: Competitions and benchmarks
and/or Identifying artificial intelligence capabilities: What and how to test.
One of the EIs we evaluated is RoboCup@Home, a league that aims to develop service and assistive robot technology with high relevance for future personal domestic applications.
It is regarded as the largest international annual competition for autonomous service robots and it is also part of the RoboCup initiative. According to the organizers, "A set of benchmark tests is used to evaluate the robots’ abilities and performance in a realistic non-standardized home environment setting.
Focus lies on the following domains but is not limited to: Human-Robot-Interaction and Cooperation, Navigation and Mapping in dynamic environments,
Computer Vision and Object Recognition under natural light conditions, Object Manipulation, Adaptive Behaviors, Behavior Integration,
Ambient Intelligence, Standardization and System Integration." More and up to date details can be found on the competition website
