I’ve always been interested in the relationship between gameplay and musical performance. Theres a remarkable structural similarity between certain game systems/mechanics and compositional ones. There is also a risk/reward/challenge aspect that is core to both practices. Anyway, for a short talk I took part in for the Leeds Evolution Festival I wrote a quick augmented chess/draughts app.
The software (Openframeworks) I wrote keeps track of a matrix of grid points that are mapped to the grid cells of a chess board. The application then samples the color values of each point (via my hacked eyetoy cam) and reports back their position and rgb attribute. In this example 2 ‘watcher’ values can be set by simply clicking the image to define a target color. These ‘watcher’ triggers are called whenever the pixel sample point is within a specific deviation of the target value. The pixel sample points move in a traditional sequencer sweep across the board.
Of course 8×8 is perfect for a basic score/drum machine sequencer (Connect 4 is my other challenge, but it has a more awkward time signature). The height of watcher A is mapped to a different sample in a standard drum machine sample bank. The height of watcher B is mapped to the pitch of a short bass-synth sound. Obviously in the video I am just composing with the peices to demonstrate the system. But you can of course play a proper game of draughts and the software sonifies the play-field in a way that you can hear who is winning & losing. There are versions of the setup that use less rhythmic soundsets and create a more choral/textural result. It could even issue specific audio warnings when certain gameplay situations arrive or segue into different compositional style based on win/loss ratios etc. Theres a lot of potential for this system and I’m going to explore it further, perhaps as far as producing a less DIY setup so It can be used in a more robust way.
I originally wrote the app in MAX/MSP/Jitter and then ported to PD/GEM. But although both these systems are great for audio they don’t perform as well with video (not without convoluted patching). Openframeworks seemed the way to go (again!).