Research Projects

MelodicBrush: A Cross-modal Link Between Ancient and Digital Art Forms

MelodicBrushMelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user’s calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.  [more]

Mobile DJ: A Tangible, Mobile Platform for Active and Collaborative Music Listening

MobileDJMobile DJ is a tangible, mobile platform for active music listening, designed to augment internet-based social interaction with the element of active music listening. A tangible interface facilitates users to manipulate musical effects, such as incorporating chords or “scratching” the record. A communication and interaction server further enables multiple users to connect over the Internet and collaborate and interact through their music. User tests indicate that the device is successful at allowing user immersion into the active listening experience, and that users enjoy the added sensory input as well as the novel way of interacting with the music and each other.  [more]

i*Chameleon: A Scalable and Extensible Framework for Multimodal Interation

iChameleoni*Chameleon is a multimodal interaction framework that enables programmers to readily prototype and test new interactive devices or interaction modes. It allows users to customize their own desktop environment for interaction beyond the usual KVM devices, which would be particularly useful for users with difficulty using the keyboard and mouse, or for systems deployed in specialized environments. This is made possible with the engineering of an interaction framework that distills the complexity of control processing to a set of semantically-rich modal controls that are discoverable, composable and adaptable. The framework can also be used for developing new applications with multimodal interactions, for example, distributed applications in collaborative environments or robot control.  [more]

i*CATch: A Scalable, Plug-n-Play Wearable Computing Framework for Novices and Children

iCATchi*CATch is wearable computing framework developed specially for children and novices to the field. The i*CATch framework is based upon a bus-based architecture, and is more scalable than the current alternatives. It consists of a set of plug-and-play components, a construction platform with a standardized interface, and an easy-to-use hybrid text-graphical integrated development.  [more]

TeeBoard: An Education-Friendly Construction Platform for Wearable Computing

MelodicBrushWearable computing and e-textiles has a lot of potential as an educational computing topic. They allow students to exercise their creativity and imagination while learning about concepts in computing and technology. However, there are still numerous difficulties involved in deploying existing technology in an educational environment. We present the TeeBoard, a construction platform for e-textiles and wearable computing that is designed to be robust, reliable, easy to construct and to program. It has also passed initial tests in a practical workshop for high school students.  [more]