This TUI, presented as a tabletop music instrument controlled by pucks placed on the surface, revolutionizes the way we synthesize and perform sounds. The pucks used in the interface are easy to maneuver — they can be spun, flipped onto the other side, removed, or replaced. Depending on, for example, the orientation and relative position on the surface, the interface provides audio feedback and beautiful visual representations of the synthesized sounds.

Use Scenarios

It was designed for a diverse demographic of users, including visitors at a museum installation, casual users, and professionals in concert. Since there are no privileged points of view or points of control, it provides for a highly social and collaborate user experience. Unlike the TUI video editor or TERN, the possibilities and combinations on the reacTable are infinite, and the user’s continuous actions will result in realtime response.

Interaction Techniques

Each object represents a modular synthesizer component and has a specific function. Objects generate, modify, or control sound. A set of rules connects objects depending on type, affinity, and proximity to their neighbors.

The visual feedback projected around an object conveys information about the behavior of the object, as well as draws the connections between the object and its neighbors. The visual feedback can also convey the real sound waveforms. Users interact with the interface given these visual and audio cues.

Implementation

Under the tabletop surface, the TUI uses computer vision (camera) and a projector. The camera loads video into reacTIVision, which analyzes the markers on the pucks and their placement on the tabletop surface. The algorithms spit out a TUIO into the connection manager, which then transmits the connection info into an audio synthesizer and visual synthesizer. The visual synthesizer determines the video to be displayed through the projector. Meanwhile, the audio synthesizer not only plays the audio but also produces for the visual synthesizer the waveform data to be displayed.

Above the surface, markers and pucks tagged with fiducials provide the information needed for the detection algorithms written for reacTIVision. These markers and pucks are ideal for this interface because they allow for infinite combinations of patterns for information representation, and they are cheap to product. The latter reason is a particularly compelling for use scenarios such as a children’s museum, where markers and pucks can easily be damaged or missing.

Interface-related Skills and Knowledge

There are no skills required of the user, not even any related to the application domain, in order to perform on the reacTable. A user could certainly benefit from having knowledge in the physics of audio, since each synthesized sound is a function of a certain sound wave. The interface relies on a specific syntax comprised of 6 functional groups, including:
- Audio generators
- Audio filters
- Controllers
- Control filters
- Audio mixers
- Global objects
The pucks from each of the 6 groups have vary in their physical shapes and associated audio and visual feedback. They are designed to affect one another so that the final synthesized sounds can be very nuanced and complex. For example, the sound resulting from an audio generator can be further manipulated by N neighboring controllers. Thus, the reacTable is a venue for a variety of complex relationships between each type of puck. In order for an audio mixer to perform its function, it must be associated with 2 audio generators each with only 1 audio output. The audio mixer itself takes 2 audios inputs and returns 1 audio output.

Even without knowledge of these complex relationships, however, a casual user can still interact with the surface and enjoy an experimental experience.

To conclude, the reacTable’s continuous and multidimensional interaction combines “precision with freedom” and “expression and creativity with entertainment.” Most importantly, it successfully avoids the files, folders, and hyperlinks metaphors still dominant in HCI.