1. Outline
2. References
3. Techniques
4. Development
5. Conclusion

1. Outline
Synesthesia is an audiovisual environment where users can compose music and visuals in a reflexive generative system. The possibilities offered by digital technology to synchronize the generation of audio and video processes are rarely explored in either sequencing software or games. In this project users would engage with a sequencing system that is also part of an abstract puzzle game. As the player progresses the environment evolves and reacts to the developing audio track which will likewise be effected by changes in the game world. This reflexive process between audio and video is guided and manipulated by the player allowing them to explore a vast range of different sonic environments.

2. References
Some video games/art works use audio-to-video synchronisation techniques. Vib Ribbon (PS2) and Rez (PS2,DC) present abstract visons of such relationships whereas ‘Bemani’ games provide a more mainstream approach. It is rare that such projects venture far into the realm of creating a generative and holistic a/v system. Media artist Toshio Iwai has created art works that allow users to ‘play’ with sound and visuals but most ‘interacive’ av art only provides a very brief and shallow experience of this kind.

3. Techniques
a) Realtime sequencing
Using a pool of audio samplesselected specifically for a particular environment the player will be involved in the process of sequencing rhythms and melodies in realtime. The progress of the player through a level will relate to their interaction with these systems. This sequencing will be represented in a visual form integral to the game landscape.
b) Digital Signal Processing (DSP)
Realtime DSP effects and synthesis could be used to ensure that the audio generated is always unique. Audio-Art applications such as MAX/MSP, Nato and Jitter have shown how realtime processing can create a myriad of experimental and atmospheric effects. Sounds could be filtered according to player energy levels or modulated to represent certain game states.
c) Image Synthesis
Using audio to generate imagery is a common technique used in PC sound players (I-Tunes, WinAmp, Windows Media Player e.t.c.). A combination of FFT analysis and image quantising would allow the visuals of the game to relate more instrinsincly to the evolving audio track. This would result in a visual environment that would reflect the players progress and technique.

4. Development
Synesthesia could be devloped in a number of ways and has the facility to be a scalable system. Obviously the most visually lush incarnations would be based on high end technology such as consoles and PCs. However handheld systems and mobile platforms also offer a good opportunity to create such an a/v sequencing game. There is also the ambition of creating networked versions of the software where players ‘jam’ with each other either in competition or collaboratively.

5. Conclusion
Synestesia is an ambitious project that aims to create an interactive environment true to its name. Melodies and images interweave in a virtual space where the user plays to conduct and direct the flow of audio-visual data.