Problem Definition

My problem is to create a theory about a user interface that can be used to create the control system for a live interactive performance. There are many examples created explicitly for live performance some for specific media, others as "show control" programs. The main unsolved issues are ease of use, cost, sharing, and integration. Although systems are available, there is no way to build upon what others have done. The key, if performance makers don't want to repeat rebuilding things that have been done already is to have a way to share what has been done. This requires a general theory about the pieces that can be shared and a common language around talking about those pieces.

Current cultural knowledge is stored in commercial projects, individual programmers and artists. Reused pieces are often disparate non-cooperating pieces that must be cajoled into cooperating with each other. Other times, the experiences and memories of users are the reusable elements that have to be recreated or merged with other technologies.

Performance Beans

In order to create and reuse elements of a live performance, what is needed is a mechanism like java beans where people can create "beans" that can be reused by a community of users. Each performance bean would have particular attributes that are required in order to used by others.

Several types of "beans" would be defined that fulfill different parts of the performance programming paradigm. Each one would be specific to a particular media types or set of media types.

Sequencing - putting events in particular orders.
Control - providing an interface to a capability that the user can manipulate.
Digitizer - bringing in data from sensors.
Filters - modifying data from sensors to condition it.
Sensing - taking digitized information and adding to the system's knowledge base.
Driver - giving the system a capability.
Rendering - taking a signal and manifesting it.
Modifier - taking a signal and modifying it for the purposes of re-rendering.
Translator - taking a signal and modifying it for the purpose of adapting its meaning to be understood by another component of the system.
Communication - allows modules to send messages and to be distributed.


The problem is on of efficiency and real-time.

The "Beans" would run inside a dataflow environment similar to max or Isadora.

Connections

Thinking about the way elements of a performance need to be connected up in order to work. Elements here are the independent "aspects" that need to communicate data to each other to run. For instance one aspect might be a cd player that can be controlled by a particular protocol, another might be a process that renders video from a file.

For interactivity there is another kind of data sharing going on between decision making elements. Information accessed is knowledge that is used to make decisions about how to act. This is a similar but different problem.

Perhaps there is a way to have the objects themselves connect up based on agent architecture types of connections.

Perhaps classes of objects produce data, others consume, some control, others do some other task. This is mainly a recognition that some types of communication and sharing between objects occurs implicitly. For instance, if a process needs to know the time, it doesn't care where it gets it. So data can be requested and answered through management by the scheduler/os.

Other kinds of data is explicitly chained/pipelined together in a flow network.

"Implied ordering" and "implied iteration" are useful concepts here. Implied ordering would allow people to implicitly know what tasks are used where and implied iteration takes away the need to create indexing loops into lists.

On the coding end of things its useful to separate operations on data from indexing through the data explicitly. If indexing is separated from operations, then a non-programmer can add behaviors without having to know details about the structure of the object with which they are working. An example of this is "eyes" where an image operator can be created to modify an image by just writing the code that happens at every pixel and not having to worry about how the image data is structured.

Robert Rowe:Interactive Music Systems

Robert Rowe
Interactive Music Systems

The main contribution to my thesis is contained in chapter 5 on machine listening. This chapter is interesting because it outlines a taxonomy of sensing feature agents used with a machine listening program called Cyper. The domain over which the system operates is strictly with MIDI data, even so, it provides an interesting list of ways to interpret sensed information and an insight into the difficulties involved in extracting meaning from sensed data.

Interesting concepts:

Automatic thresholds
Focus
Decay
Manual assignment of thresholds

Summary:

Notes and Events are the main structure that the system analyzes. These structures contain information about the pitch, duration, and timing of actions.

Focus and Decay- This is the ability to change the context of what is evaluated by the system. The focus can be wide if values are changing over a large area of space and time or narrow if detection of the change requires analyzing a small spectrum of possibilities.

Decay- "the adjustment of focal scale over time"

Agents:

Register agent: "classifies the pitch range within which an event is placed". In cyper this is represented by two bits. Register as it is used here is for judgments like "its a high pitch range" or "its in the middle", or its "a low pitch".

An interesting application of register and focus is described that helps the system make judgments about register. In my vocabulary I would describe this as an automatic threshold technique. In Robert Rowe's book he determines the register by keeping track of the maximum and minimum pitch over a period of time. This determines the range over which register can be judged. If the range is less than two octaves register is classified into two ranges: high and low. If more it is classified into four.

Robert describes decay as a necessary complement to focus. Decay adjusts the range over which register is judged by contracting the range slowly over time if new data is not moving the endpoints outward from the min and max values. This contraction decay's timing and rate is determined empirically.

Dynamics Here, dynamics is the relative loudness of the composition events. Because of a wide variation in the way instrument interpret MIDI data, Each instrument must be thresholded differently. Robert uses MIDI data so he is limited to the information carried by MIDI events. This is an example of where an automatic thresholding algorithm can't be applied because of ambiguity. The ambiguity comes in the computer not knowing if the score is being played softly or the instrument has a small range of response.

Density: Vertical and Horizontal. horizontal is speed and vertical is "the number and spacing of events played simultaneously".

In vertical density focus and decay are ignored due to the low resolution of the measure.

There are more:

Attack Speed
Duration
Chord identification
Beat analysis and tracking: pulse (regular recurrence of undifferentiated events), meter (differentiates regularly recurring events), and rhythm (pattern of strong and weak beats)
Key Identification
Meter Detection

Higher levels:
Phrase finding
Regularity

The point is that evaluation and analysis of even very simple clear reliable data such as MIDI is complicated. But, even so, can a general method for analyzing sensor data be established that works across data types and meanings? Is there a cannon of techniques that underlies the analysis of sensed data?

Introduction

This blog is an annotated bibliography of information related to the design of computer systems that support the creative process of making live performances. The key concepts here are that it is information that supports a theory about how a computer system can be designed that makes creating interactive performance easier. This can be thought of as an authoring system for live performance.

Much work has been done both in industry and academia around the creation and control of media for and in live performance. Yet, in the area of interactive media in performance there is very little. Ideas around authoring in interactive media tend to be thought of in the context of website design, games, or CD/DVD experiences.

What is missing is a theory about how to create a computer system that supports the authoring of live performances for interactive media.