Synthpaint - Devlog 02 - Origins and First prototype

Hi there!

In the previous entry, I explained my motivations for designing a new synth instrument. Today, I will talk a bit about my attempt at developing a first elemental prototype.

But first, some background:

Balldrop and how this idea came to be

A year ago I developed an instrument using Unity and Max/MSP for a sound programming class at my University. I called it Balldrop4Life (you can download the source code on my Github). It was inspired by Balldrop, an experimental instrument made in the Godot engine by developer Bauxite. The concept is simple: 2D balls drop from the top of the screen, and the user can draw lines that serve as platforms, with which these balls will collide. The collision will produce a musical note or percussive sound.

My objective was to take this idea and remake it in a way in which I could use the game to trigger instruments and effects from an Ableton Live session, and also be able to manipulate how some of the game parameters behave from that same Live session. Following this series of tutorials by developer samyam I learned how to draw lines with mouse interaction in Unity. The balls were simple: just make them physical objects with gravity and collision detection. For Unity-Ableton Live communication, I used a free OSC plugin called extOSC, which responded to the collision system inside Unity and sent OSC messages to patches I programed in Max4Live. I am no programmer, so my code was full of annoying bugs, but I was able to finish a version that was usable enough (thanks in no small part to the help of my very talented friend pr00thmatic, who answered all my C# related questions).

I am a sucker for chaotic systems and random and noisy improvisation.

While is true that I (and some friends I showed it privately) had fun playing with the system as it was, I found the drawing part quite bland and uneventful. Since the balls only react to the collider in each point of the line renderer, the “drawing” could well be replaced by a simpler interaction, like just clicking the canvas to position collider points on the screen. The “drawing” was a fun but ultimately superficial coating that did not alter the resultant soundscape in any meaningful way. Soon I started wondering if I could do something different to change this.

Other pre-existing sound/drawing hybrids

With the experience of Balldrop4Live in mind, I began to explore ideas about how to translate the drawing into sound in a meaningful way. This concept is far from revolutionary. As I explained in my previous blog entry, Greek composer Iannis Xenakis and the team at CEMAMu developed UPIC back in the 70s, which was a tool that allowed composers to make music by drawing lines on a hardware-based canvas, which translated the lines drawn onto its surface into sounds, either sampled or synthesized. Some software-based recreations of UPIC have been released. Here’s UPISketch:

UPISketch, a software based re-imagination of Xenakis’ UPIC

Another approach is the one used by tools like Metasynth’s Image Synth and Image Filter, which translates the drawings that the user makes on its interface into spectral information that can be then played back. In both UPIC-like and Metasynth-like applications the canvas is treated as a timeline that is read from left to right, while the pitch information is interpreted from the vertical axis, with low pitches or frequencies at the bottom and higher pitches/frequencies at the top of the canvas/timeline.

A loop-based Metasynth drawing improvisation.

The performative problem

While interacting with UPISketch, Metasynth, and other similar tools I always experienced creation and execution as two different processes. Some of these applications have a degree of real-time drawing/playback capabilities, but in general, they act less as musical instruments and more as playback engines in which you need to draw first in order to listen later. Perhaps because of my background as a guitarist, I find a lot of joy in the immediate feedback of instrumental manipulation and improvisation, and I would personally prefer to play with something that feels less conceptual and more tactile and reactive. I also believe that the separation between processes misses a chance to make these systems more interesting from a performative side, where the user (a.k.a. performer), visuals, and audio are not dependent nor independent, but interdependent entities that inform and influence each other. 

First prototype

Of course, none of these ideas matter if I don’t put them first into practice. With the code of Balldrop4Live as template, I started a new Unity project with the goal of developing the simplest possible iteration of the idea: drawing a line = playing a sound.
Unity’s sound synthesis capabilities are arcane at best. Advanced users have experimented with Unity’s low-level audio framework DSPGraph to synthesize sound with varying levels of success (here’s a node-based granular synth!). Unfortunately, for non-programmers like me, the scarce documentation and almost nonexistent examples of use make this technology impenetrable, so I decided to look elsewhere. In the beginning, I was tempted to use Wwise's built-in synth called SynthOne, which is a basic two-oscillator synth. After some research, I learned about LibPDIntegration, a wrapper that lets you integrate Pure Data into Unity, and decided that it was the best tool I could use at the moment. Although I haven’t used Pure Data before, I figured that my previous experience with Max/MSP might come in handy. Also, Pure Data has the potential to be much more scalable and flexible than Wwise’s SynthOne.

Following Yann Seznec’s tutorials, I was able to have a functional synth that reacted to game engine calls in no time. As demonstrated by the other projects I showcased above, the idea of bottom = low frequency, top = high frequency has been made into a convention, which to me makes sense and feels intuitive, so I decided to go in that same route. As for the horizontal axis, I decided to use it to represent the stereo field with sound being drawn at the left of the canvas moving towards the left speaker, and vice-versa. The result was this extremely basic but fun to use “drawing” synth:

My personal thesis was true (at least to myself): generating audio while drawing is way more fun than drawing first and listening later!

Timeline considerations

While this first prototype was successful at representing the core foundations of the tool (which by the way has already been presented at this same stage time and time again), it also raised many morphological questions, both in relation to shape and sound. Nevertheless, the most urgent topic for me was without a doubt the representation of the main domain of sound: time. As previous examples illustrated, other designers opted for discarding the idea of a canvas in the traditional sense entirely in order to represent time as a scrolling timeline. This is no surprise. Time is the domain that differentiates the most what we can recognize as traditional static art from sounding art. Ignoring the many complex psychological aspects that influence perception and focusing only on the materiality of these two mediums, we can probably conclude that an image is mainly perceived in space. Sound on the other hand can only be perceived by the human ear as long as its envelope develops through time. I suspect this is the reason why a graphical artist usually creates on a static canvas that is akin to the bounds of the human visual field, while composers and sound designers use sequential systems like music scores or software timelines that can theoretically be extended linearly to the infinite. A tool that is truly a hybrid between both domains should not opt entirely for one or the other way of perceiving the working space, because in doing so it would automatically subordinate one to the other.

Canvas: Graphic Art creation is static. Occurs in a bounded space akin to the bounds of our visual field.

Timeline: Music creation is sequential. The required workspace needs to represent time, so it scrolls linearly to infinity.

So, how can we achieve synergy between these two contradictory ways of interacting with the empty time/space? That’s the question I tried to answer with my second prototype, which I will attempt to explain in the following entries.

Until then, stay safe!