Bart Dohmen Industrial Design | Graphic Design | Photographybd[at]bartdohmen[dot]nl

Multi Modal Interaction

Multimodal interaction looks at the synergetic properties of interaction when multiple modalities are taken into account. This module placed this theme in the context of a design team. An idea generation session is often chaotic, sketched ideas are scattered all over the place and ‘descriptions’ or certain aspects of ideas sometimes get lost in the heat of creating ideas. Moreover the mapping (or clustering) of ideas is mostly done in 2D space whereas 3D space could provide added analysis and mapping possibilities. Traditional brainstorm interactions involve modalities like speech, body language and gestures mainly supported by visual cues on post-its, papers and flip-overs. More complex idea analysis requires application of more modalities or different involvement of these modalities, better enabling the design team to grasp and organize the data.

The first part of the module consisted of going through some basic theory and literature. After a small group exercise the second part was about new concept development for multi-modal interaction. To have the most benefit of a pragmatic and hands on approach we choose for one specific idea from a team member to start working with right away and from there work with several iterations to learn by doing.

Result We designed a tool which gives design teams the opportunity to cluster ideas into a 3D physical architecture of a brainstorm session and map this to a digital world. The goal was to improve communication in a design team in a multi modal manner. The concept for now only was a physical representation used for acting out in a movie to demonstrate the intention of the design.

The system layout consists of a physical interface (spheres and branches), a projector (projecting downwards onto the physical architecture and position, gesture and speech recognition software. In total the system helps the user to create a mental map of how ideas and themes are inter-related, and how they relate to specific characteristics by means of physical representations. Instead of being limited to the 2D surface for analysis now a third dimension is available for better analysis purposes. The users can define how this 3D space is arranged in order to make a clear arrangement of ID and tag clouds. The physical aspect of the system enables to fix clusters in this 3D environment and provide easy access by touch and speech cues to store and retrieve information and tags. Seeing and manipulating the objects while discussing the content makes the discussion and collaboration more effective. The physical representation lowers the cognitive load for the participants as more complex relations can be clearly communicated.

Please, do share!

    Comments are closed.