CommRob

This report describes the activities of Task T7.3 in workpackage 7 (WP7) related to the evaluation of the usability and user experience of the shopping trolley. Such a trolley should, through its design, invite to a shopping activity that is efficient but also enjoyable. Therefore evaluation methods are used that combine assessment of task performance with subjective measures, assessed through questionnaires and interviews. The results concerning task performance are partly reported in D6.3 (System tests). The report describes four different studies:

• An experimental study on the effect of motion cues on users’ behaviour.
• A study of user experience in shopping tasks with the robot trolley.
• A preliminary study on the Walking Aid.
• A preliminary study on Collaborative shopping.

A questionnaire survey of acceptance and satisfaction with the robot trolley was made with participants in the first two studies.

These studies have shown that the system can be used for its main task, i.e. shopping in a realistic setting, by untrained users. About a third of the participants in the first two studies were satisfied with the robot’s behaviour and virtually all of them thought the robot was interesting. Problems with robustness and recognition rate in interaction with the robot were encountered, and the participants rated the touch screen interaction highest of four integrated user interfaces. The use of Motion Cues as an interaction element is an interesting aspect of HRI that should be considered in future research. The walking aid function is operational, but in a too early prototype stage to allow for a more extensive study. The collaborative shopping uses robot behaviours that are already implemented and only time and logistical constraints prevent further studies of this functionality.

CommRob defines “Dialogue Design Support Features” as “advanced multimodal dialogue templates using the communication platform as a basis”. In developing Dialogue Design Support (DDS) we have chosen to focus on a toolset which allows modelers to create multi-modal dialogue templates in the form of communication models. Therefore, the modeling language and tools were generalized from tools that support just the modeling of GUI communication to tools supporting modeling of multimodal communication, which further can be used by “modelers”, too, i.e., for people with an expertise in the communication domain but not in working with code For that, concrete representations of the generated user interface (in multiple modalities) were devised, along with their correspondence to the dialogue model. Also an overall map of the possible interaction is available to the modeler in the form of a User Interface navigation map. Special attention is payed to example dialogues that can be generated from a certain discourse model, recording of dialogues and assembling dialogue instances into corpora for future reference and verification. At the end of the dialog design support process, the number of ideas developed exceeded the capacity for implementation and therefore many such ideas are left to future work, which are also presented in this report.

This document represents the deliverable D6.4 which provides a technical description of the robot trolleys to present the robotic platforms developed in the course of this project.

This document represents the deliverable D6.3 - Report on system test results. It sums up the results acquired by tests of the individual sub-system performed by the developing work package or in the integration or test weeks organized by WP6, tests of the individual subsystem once these were integrated on the demonstrator and three tests of the overall integrated system.

This document represents the deliverable D6.2 - Report on integration effort. It sums up the final state of work package 6 concerning the progress on the effort taken on integration. It describes the final state of integration on the two robots (while the description of the two robots themselves can be found in D6.4). Additionally an overview of the current state of the art in similar applications is provided.

This deliverable is actually the developed software of the communication platform, which is included electronically only, of course. This document explains its structure and sketches the content of the various modules.

This deliverable reports on the advancements software development of the communication platform and the application logic of the robot since Deliverable D5.3. It describes advancements in the interaction modelling approach based on discourses by introducing actions and notifications as well as improve- ments in the support of the interplay of automatically generated user interfaces and manually developed user interfaces. Furthermore, it reports on the usage of the communication platform in robot-robot com- munication scenarios and on implementation improvements of modality fusion and fission. So, this deliverable is actually a report on the improvements, extensions and modifications of the whole top layer of the robot’s software architecture done in the third period.

This deliverable lists one after another the functionalities that have been developed in this WP and which have already been enumerated in the deliverable D4.1@m8 titled “report on Human-Robot interaction specification”. The required hardware and software were presented while details concerning the input, description, output of components and their importance in the overall layered architecture were given. This deliverable gives an overview of these functionalities, together with their current status and associated results. The entire details regarding theory, evaluations, and comments can be found in the associated papers which can be sent to the reviewers on request.

Since interactive robots directly interact with people, finding natural and easy-to-use user interfaces is of fundamental importance. One challenge in the CommRob project is the development of a flexible multimodal interface based on speech and gesture means in order to control the trolley in noisy and cluttered scenes.

This document reports the first developments carried within the Task “Commanding and deictic gestures interpretation. It provides a snapshot of the work at month 30 concerning theoretical issues and early experiments in terms of: (i) speaker localization for speech understanding, and (ii) gesture recognition.

This deliverable is the final report on developments in work package three (WP3) “Advanced Robot Behaviour and Navigation”. This final report presents a compilation of technologies developed during the project. This includes the localisation concept, the hybrid hierarchical control architecture, the hierarchical data abstraction in the Local World Model and the Advanced and Basic Behaviour Repertoires as well as safety behaviours and the hardware abstraction.

Due to the fact that already three deliverables (D3.1 to D3.3) have been submitted this report will focus on new developments of the last year and briefly summarize earlier developments with corresponding references to the earlier deliverables.

The first of the major developments are user-centred behaviours, so this report is closely related to the deliverables D4.2.2 “Report on Haptic-based Detection” and D4.6 “Final Report on perception of humans” to which there will be several references. The second main objective of the last period have been adapting the robot's control to the dynamic environment: goal point can be blocked, corridors can be impassable and suitable parking positions at products have to be identified, to mention just some of the topics. Hence a group for Scene Analysis has been developed which analyses the local environment, the given task and the behaviour of the user.

The third major objective has been the development of multi robot behaviours to enable multi robot collision avoidance, queuing up at the cashier or to enable one user to handle several robots at the same time.

Building such an innovative robot like the trolley in this project is a difficult task at today’s state of the art, even though it will be “just” a prototype. In particular, a new communication platform has been integrated with a robot platform supporting advanced behaviours. This integration is even able to handle expressions of communication through behaviour.

For the given overall task, we adopted a Systems Engineering approach. After all, such a robot system consists of mechanical parts, electronics and software. Following such an approach, the requirements on the system were defined and specified. According to these requirements, a system architecture was devised.

Finally, the work on system engineering for these robots involved system integration as well. From the system’s perspective, this included checking the fulfilment of the requirements and the consistency of the built system with the designed system architecture.

This report describes work in Task T7.2 Developing dialogue design support features. It defines interac- tion design situations that are envisaged to occur at various stages in the development process of the robot trolley: research-oriented design of low-level behaviours and communication modelling, application de- velopment by system specialist close to end-user, and in-store reprogramming by non-technical staff. The design support that is needed in these different situations ranges from (1) guidelines for multimodal user interfaces; (2) use of multimodal corpus data in the design process; and (3) support for application development.

When comes to guidelines for robotic trolleys it is safe to say that they do not exist. Robotic supermarket trolleys are a novel kind of technology and therefore we have analysed and described guidelines for interface technologies that occurs in the envisages system, i.e., guidelines for touch screen interfaces and speech interfaces. As there are no guidelines for how to design communicative robot body movements we have revised the literature in the field in order to support future attempts to explore the possibility of creating initial guidelines for robotic body moves. We have also investigated how corpus data can be used in the design process in the various design situations. Here we propose the idea of a case-book that is formed by corpus excerpts, illustrations or visualisations based on the corpus data. It is necessary that the research ethics related to the use of personal data is respected. This is something which is supported with the notion of a case-book.

The application development support is aimed to create the possibility to work with short time devel- opment cycles. This means that tests with system of various fidelity levels should be possible, ranging from visualised models, hi-fi prototypes where parts of the functionality is simulated in the Wizard-of-Oz framework, to full-scale working prototypes, that can be modified to create various user experiences and allow for testing of the dialogue design support features.

The future work will be focused on the development of these features, the Case-book, the support for having guidelines available as help information and heuristic tests within the system. The application support will be tested by creating various prototypes using the modeling tools developed in close collab- oration with WP5. The prototypes will be evaluated with respect to user experience and tailorability in WP7 (T7.3).

This report presents the work on interaction design and communication platform assessment. Interaction design was conducted as a cooperative design endeavour, in a series of workshops with a number of stakeholders. We employed and refined prototyping and enactment techniques during the process. Based on agreed-upon prototypes in the later parts of the design process, we performed communication modelling for communication platform assessment.

This deliverable is actually the developed software of the communication platform, which is included electronically only, of course. This document explains its structure and sketches the content of the various modules.

This deliverable reports on the software development of the communication platform. In addition, it reports on the development of the related part of the world model repository and the application logic of the robot. So, this deliverable is actually a report on the development of the whole top layer of the robot’s software architecture.

Since interactive robots directly interact with people, finding natural and easy‐to‐use user interfaces is of fundamental importance. One challenge in the CommRob project is the development of a flexible multimodal interface based on speech and gesture means in order to control the trolley in noisy and cluttered scenes.

This document reports the first developments carried within the Task “Commanding and deictic gestures interpretation. It provides a snapshot of the work at month 24 concerning theoretical issues and early experiments in terms of: (i) speaker localization for speech understanding, and (ii) gesture recognition. It will be followed by a second deliverable at month 30, complementing some theory and focusing on more evaluations.

This deliverable describes the developed visual‐based coarse people tracking for coordinated human‐ robot motions. Tracking will be used to plan the robot’s path with respect to the user in a socially acceptable way. Recall that our main guiding line is to consider the outputs from heterogeneous and complementary on‐board sensors based on vision, radio‐frequency, possibly audition. Our primary motivation in this deliverable is to include these heterogeneous functions for detection/recognition of people (see the previous deliverables D4.3.1@m12 and D4.3.2@m18) in trackers i.e. spatiotemporal analysis to estimate the coarse human‐robot distance and keep the visual contact during any guidance missions in the supermarket.

This document represents the deliverable D3.3 - Report on the advanced and cooperative behaving. It sums up the final state of Workpackage 3 concerning the progress on the development of the behaviour-based control for the navigation. The main focus of this document lies on the advanced behaviour repertoire and the user-related behaving of the robot, but the complete navigation system is displayed as well to be able to refer to certain issues from the M21 report.

Building such an innovative robot like the trolley in this project is a difficult task at today’s state of the art, even though it will be “just” a prototype. In particular, a new communication platform is to be integrated with a robot platform supporting advanced behaviours. This integration is even supposed to be able to handle expressions of communication through behaviour.

For the given overall task, we adopted a Systems Engineering approach. After all, such a robot system consists of mechanical parts, electronics and software. Following such an approach, the requirements on the system were defined and specified, first from a user perspective and then from a system perspective. We also made sure that the requirements of both views are consistent with each other. According to these requirements, a system architecture was devised, first from a logical view. It integrates the communication part with the part supporting advanced behaviours. This logical view was finally mapped to a physical view of the robot trolley’s architecture.

Due to the difficulties of the given task and especially the innovative character of this robot system, the actual work was performed, however in iterations of the activities sketched above. This approach is carried over from software development and helps to cope also with evolving requirements. In particular, it helped in our system engineering endeavour to come up with a good architecture.

This deliverable describes the developed low‐level functions to person detection and recognition in the robot vicinity. It complements the first deliverable D4.3.1@m12 at month 12 with recent theoretical and experimental considerations instigated during the last six months. Recall that our main guiding line is to consider the outputs from heterogeneous and complementary on‐board sensors based on vision, radio‐frequency and audition. Speech recognition is involved at a semantic level. Further developments will be to include these basic functions in medium or high‐level robotic functionalities dedicated to spatiotemporal analysis (D4.4@m24) or to multimodal command interpretation (D4.5.1@m24). An overview of these functionalities can be found in D4.1@m12 and also D4.3.1@m12.

For the interaction based on physical contact, a haptic interface for intuitive use has been created. Depending on the applied forces, different user intentions have be derived. The first deliverable D4.2.1 focused mainly on the overall functionalities, the mechanical and electrical design of the haptic handle. This document describes the work that has been carried out on the definition of useful classes for commanding the robot using a haptic interface, the classification of the user intention and the evaluation of the results.

Building such an innovative robot like the trolley in this project is a difficult task at today’s state of the art, even though it will be “just” a prototype. In particular, a new communication platform is to be integrated with a robot platform supporting advanced behaviours. This integration is even supposed to be able to handle expressions of communication through behaviour.

For the given overall task, we adopted a Systems Engineering approach. After all, such a robot system consists of mechanical parts, electronics and software. Following such an approach, the requirements on the system were defined and specified, first from a user perspective and then from a system perspective. We also made sure that the requirements of both views are consistent with each other. According to these requirements, a system architecture was devised, first from a logical view. It integrates the communication part with the part supporting advanced behaviours. This logical view was finally mapped to a physical view of the robot trolley’s architecture.

Due to the difficulties of the given task and especially the innovative character of this robot system, the actual work was performed, however in iterations of the activities sketched above. This approach is carried over from software development and helps to cope also with evolving requirements. In particular, it helped in our system engineering endeavour to come up with a good architecture.

This document represents the deliverable D3.2 - Report on basic navigation functionality and local behaviours. It sums up the actual state of Workpackage 3 concerning the progress on the development of the behaviour-based control for the navigation. The main focus of this document lies on displaying the basic navigation system. This includes on the one hand the Global Navigation that is mainly implemented by a topological navigation and on the other hand the Local Navigation that is implemented by a behaviour network.

This report presents the work done during the first year of the workpackage WP7, Interaction Design and User-Centred Prototype Evaluation. It summarizes the work in Task 7.1 during the first 12 months, which has mainly aimed at exploring the setting where the robot trolley targeted in the project will be used, and initiating a dialogue with potential users and stakeholders. An extensive field study has been carried out in supermarkets, involving observations and interviews as well as study of relevant literature on the shopping experience. Focus groups have also been arranged to elicit user reactions to a robotic shopping trolley. Finally, we have initiated the development of a visualization tool which will be further used in the next phase of the project, in order to design and test prototypes suitable for the trolley’s multimodal interfaces. The report ends with a list of implications of the study for the interaction design of the robot trolley.

The main purpose of this deliverable is to define the detailed design and layout for a demonstration robot. The set-up of the demonstration robot needs to be defined by its mechanical parts and electronics.

The partners have specified these components and the overall set-up of the robot according to the special demands and requirements as defined in their tasks. Therefore the demonstration robot will be able to fulfil the given goals. Special attention has been given to ergonomics and the real supermarket environment of a trolley.

The design specification constitutes the base for the later integration work of the modules as provided by the partners.

Having a communication model is important but certainly not sufficient for enabling a robot to communicate with humans (and other peer robots). In particular, our communication model is modeindependent (which also means that modellers do not have to include information about modalities), but there is something needed to operationalize these models and to enact them as multimodal dialogues.

We provide a high-level specification of the requirements on a communication platform to be developed in software, which is supposed to do exactly this. While this software will be able to handle discourses in general, additional software is needed for the specific “application logic” of this particular robot, so that it can fulfil the tasks given especially via communication at runtime. So, we provide a high-level specification of the requirements on this software as well.

When robots will be introduced more and more into everyday human environments, and when they are supposed to cooperate closely with humans, they will have to communicate well with humans (and with other peer robots). For enabling this, we postulate that the robot will have to possess certain communication abilities like humans.

Therefore, we devised a high-level approach for communication that is inspired from human communication theories. We provide an overview of related theories and specify the core of our current communication approach in terms of a communication definition language.

This deliverable describes the developed low-level functions to person detection and recognition in the robot vicinity. As the trolley aims at evolving in heavily cluttered and highly dynamic human-centered environments, our line of investigation is to consider the outputs from heterogeneous and complementary on-board sensors based on vision, sound and radio-frequency identification, as well as with speech recognition in order to interpret verbal commands. These functions constitute the basis of all the human trackers and human activity interpretation systems to be developed in the workpackage. This document provides a snapshot at month 12, mainly concerning theoretical issues and early experiments. It will be followed by a second deliverable at month 18, complementing some theory and focusing on advanced evaluations.

For the interaction based on physical contact, a haptic interface for intuitive use must be created. Depending on the applied forces, different user intentions can be derived. Based on the gathered sensor information, a decision making algorithm for the actual intention and the computation of the desired behaving parameters will be realized. Based on this information the behaviour-based robot control is activated. This document (D4.2.1) focuses mainly on the overall functionalities and the design of the haptic handle.

The aim of CommRob’s WP4 is the development and implementation of functionalities enabling two main tasks: 1) coordinated User/Trolley motions thanks to physical (haptic) or non-physical (vision) contact, and 2) human-friendly User/Trolley multimodal communication through speech and/or vision. This document constitutes the associated deliverable planned at month 12. It specifies the functionalities dedicated to multi-modal User/Trolley (U/T) interaction in order to meet the requirements defined in WP2, to be materialized in the key-experiments detailed in the Technical Annex. After their specification, these functionalities will be developed and evaluated by one or several partners involved in the WP.

This deliverable is structured as follows: The functionalities of the human perception functions and the Human-Robot Interactions (HRI) are first outlined in Section 2. The required hardware and software is then presented in Sections 3 and 4 respectively while details concerning the input, description, output of components and their importance in the overall layered architecture, are given in Section 5.

This document represents the deliverable D3.1 “Report on Behaviour and Navigation Specification and Architecture”. It sums up the actual state of Workpackage 3 concerning the progress on the development of the “Behaviour-Based Robot Control Architecture” for movements, navigation and interaction with humans and other robots, as demanded in the Technical Annex. The main focus lies on the concept of the control architecture itself and the elaboration of a prototype of the behaviour network containing the basic behaviours to start with.

Project Website Online