MelodicBrush: A Cross-modal Link Between Ancient and Digital Art Forms


MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user’s calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.




Figure 1.Setup of MelodicBrush’s writing component (in centimeters). The user “Writes” on the horizontal display surface; the writing mechanics are captured using a depth camera; the brush strokes are generated in real-time.


Figure 2. Rendering of a brush stroke. The blue dots indicate the smoothed touch points on the brushstroke trajectory; the purple dots are resampled points. The size and the orientation of the green polygons are dependent on the orientation of the brush. Filling in the polygons with textures renders a stroke with smooth boundaries.


Figure 3. Music generated by the system. The written character is 王(Top). Direct mapping from calligraphic strokes to notes generate repetitive music reminiscent of religious chants (middle and bottom). Computer-assisted music generation in slow writing (middle) and fast writing (bottom). Directly generated notes are in the black, colored notes are generated by the statistical model.


Figure 4. Effect of smoothing. Writing on paper (bottom) and rendered character on display surface (top).


Figure 8. Screenshot of the writing surface. In addition to calligraphic strokes, the musical score is displayed as a running track across, the bottom of the display. Directly-generated notes are shown in red, computer-assisted generated notes are in black.