Designing an Artificial Attention System for Social Robots
The following video summarises the overall outcomes of the project. Besides the outcomes for the solutions already detailed below, it also presents the two top-level components that regulate top-down modulation of attention (aka the "top-down controller"):
The full technical description together with overall results was presented in IROS (see Publications). The final technical report documenting the full software framework is presented below.
Multisensory 3D Saliency for Artificial Attention Systems
As one of the main outcomes of this project, the CASIR team developed a short-term 3D memory for artificial attention systems, loosely inspired in perceptual processes believed to be implemented in the human brain. Our solution supports the implementation of multisensory perception and stimulus-driven processes of attention.
For this purpose, it provides:
Visual and auditory signals are processed to extract features that are then filtered by a proto-object segmentation module that employs colour and depth as discriminatory traits. We consider as features, apart from the commonly used colour and intensity contrast, colour bias, the presence of faces, scene dynamics and also loud auditory sources. Combining conspicuity maps derived from these features we obtain a 2D saliency map, which is then processed using the probability of occupancy in the scene to construct the final 3D saliency map as an additional layer of the Bayesian Volumetric Map (BVM) inference grid [Ferreira et al. 2013].
The inherent properties of our solution were presented in REACTS 2015 (see Publications), and a summary video showcasing these properties is presented next.
Apart from the generic properties of the 3D saliency framework, there are also two important sets of features that are provided by this solution:
Some of these properties are shown in the video presented below.
[Ferreira2013] J. F. Ferreira, J. Lobo, P. Bessière, M. Castelo-Branco, J. Dias - "A Bayesian Framework for Active Artificial Perception" - IEEE Transactions on Cybernetics (Systems Man and Cybernetics, part B), vol.43, no.2, pp.699-711, April 2013
Demonstration of current implementation
The CASIR team has developed a demo showcasing the most recent stable features of the CASIR-IMPEP attentional framework, namely in how it replicates human-like overt attention processes by using a sophisticated probabilistic multisensory perceptual framework, while providing a means to validate current implementations of low-level supporting algorithms. The Integrated Multimodal Perception Platform (IMPEP v2.0), an active robotic head designed and developed at the Faculty of Science and Technology of the University of Coimbra (FCT-UC), is to be used as the prototypical robotic agent throughout the project. The mounting hardware and motors were designed by the Perception on Purpose (POP - EC project number FP6-IST-2004-027268) team of the Institute of Systems and Robotics/FCT-UC, and the sensor systems mounted at the Mobile Robotics Laboratory of the same institute, within the scope of the Bayesian Approach to Cognitive Systems project (BACS - EC project number FP6-IST-027140).
The current version of this demonstrator showcases basic active perception capabilities using OpenCV functionalities, which provide a set of feature images and 3D point clouds that are used by the CASIR attentional framework to promote gaze shifts (i.e. active head reorientations) towards the most behaviourally relevant features in the image. The demonstrator is to be run in this fashion in an intentional infinite loop.
A clip showing a demonstration of a few of the system's capabilities, together with a Technical Report detailing the system, are presented below.