A. L. Ronzhin, A. A. Karpov

The architecture and laboratory prototype of the automatic information enquiry system MIDAS (Multimodal Interactive-Dialogue Automaton for Self-Service) with a multimodal user interface are presented in the paper, as well as researches and analysis of interaction between users and developed device are described. For distant speech recognition of Russian commands, a microphone array is applied in the multimodal kiosk. It allows us to localize a source of useful speech signal and to eliminate influence of external acoustical noises to the accuracy of speech recognition. Miniature video-cameras and corresponding processing methods for optical flow provide both detection and tracking of position of a user inside of the kiosk-s working zone. Information system proposes multimodal user interface for getting information about employees and departments of SPIIRAS as well as map-based information on the streets of St. Petersburg. The main hard-and-software modules of the kiosk are: (1) video processing using a technology of computer vision in order to detect position of human-s body, face and some facial organs; (2) speaker-independent system of automatic recognition of continuous Russian speech that uses a microphone array to eliminate acoustical noises and to localize a source of useful voice signal at distant speech recording; (3) a module for audio-visual Russian speech synthesis to be applied for realization of a virtual character - avatar; (4) an interactive graphical user interface based on a touchscreen; (5) a model of dialogue and a dialogue manager that include a database of an applied domain and a system for dialogue strategy control. Fusion of user-friendly computer interfaces with speech technologies and usage of virtual talking characters allow us to create effective and natural interfaces, where comfort of a human being plays a main role. By means of the multimodal information kiosk the questions of natural and ergonomic communications between a user and a machine are studied taking into consideration diverse ways of communication. Collected experimental data were analyzed in order to investigate cognitive and behavioral characteristics of a user and to optimize the multimodal interfaces proposed to clients. The experiments on quantitative evaluation of users - operation speed with diverse contact-based and hands-free means of information input were made, attracting potential users and using methodology proposed by the ISO 9241-9 standard based on Fitt-s laws. Moreover the estimation of accuracy of spatial localization of a sound source by the developed microphone array and several methods for spatial-spectral analysis of speech signal was made. Organized cognitive experiments on communication between users and the multimodal automaton have shown willingness of the most of potential users to natural multimodal or speech interaction with an automatic system and their refusal to operate by the common communication way, but unnatural for inter-human conversations, based on a touchscreen and mechanical manipulators