19^th International CODATA Conference
Category: Infoscience

Multimodal Interface for Data Retrieval during Conversation

Roman Zenka (zenkar1@fel.cvut.cz) and Mr. Pavel Slavik (slavik@fel.cvut.cz)
Czech Technical University in Prague, Czechoslovakia

We propose a tool for aiding the scientific conversation by providing the participants an easy access to discussed data. Our tool utilizes sketch and speech recognition capabilities of a TabletPC, which allows it to identify the data being discussed and to visualize them in proper context. The entire progress of the conversation is being recorded for future offline viewing and archiving.

Combination of two natural means of communication – speech and sketching – gives our tool several advantages compared to conventional means of human-computer interaction.

The richness of speech allows fast and comfortable specification of required data, without the burden of navigating through conventional GUI, such as menus or directory trees. The command for retrieving the information can be even given within the conversation itself (i.e. as a part of a longer sentence) so the speaker is not interrupted by handling the computer.

The sketching significantly raises the robustness of speech recognition. Compared to speech it also allows fast specification of 2D positions and relationships. The permanently visible sketches also complement the transient character of speech, providing a visual feedback about the progress of the conversation.

Finally, by simultaneous sketching and talking, the scientists automatically create a graphical representation of the recorded conversation. By clicking on the sketches it is possible to quickly navigate to the part of conversation that is related to them. This way the results can be distributed and watched offline by other members of the research team.

An important advantage of our approach is also its social factor – our tool requires nearly no direct users' attention, which can be spared for other participants. Users can speak fluently – without delays caused by complicated interaction with the computer – and even maintain direct eye-contact most of the time, while presenting their data in high quality.

Our approach has been tested on a sample application for illustrating the ongoing conversation in real time. The user tests have shown that this tool can be valuable especially for managing larger meetings, where it allows faster and more fluent communication, saving valuable time of the participants.