Dazed and Confused, full interview with Stephen Malinowski (and photo outtakes)

D&C: Had you worked with Björk before?

S.M.: No. Until recently, I had only done collaborations with my friends, like these, which I did with guitarist James Edwards, who I've known since we were teenagers ...

... and this, done by Quaver, a group my friend Marie Dalby is in ...

As a result of my work getting known through YouTube in the last few years, I've done a few small projects with composers and performers I met that way ...

(James Edwards, Steffen Wick, Vincent Lo, Anderson and Roe, Etienne Abelin, Del Sol String Quartet, Quaver Viol Consort)

D&C: How did you and Björk meet/hear of each other?

S.M.: She saw my videos on YouTube. My first contact was with her manager, Derek, who asked me to collaborate on Biophilia. My first contact with her personally was at a dinner party in San Francisco in March, when her team was in California to meet with people at Apple Computer.

D&C: Exactly how long were you working on the Biophilia project?

S.M.: Derek first emailed me around the end of January 2011, and I've been working on it on and off since then.

D&C: What kind of processes did it entail?

S.M.: At first, the plan was for me to produce videos like the ones I have on YouTube, but when I delivered the first draft of the first video to Björk's team, they realized that the file size was too big, so we switched gears, and instead of pre-rendering the animations, I developed an iPad app to generate them on the fly in real-time.

D&C: What did you learn?

S.M.: I don't know where to begin, but I'm not sure whether any of the things I learned are the kind of things you're asking about. I learned a lot about doing business with Björk's team, and about how to write an iPad app, and about how my health and temper suffer when I work sixteen hours a day for weeks on end without a break ...

D&C: Did you work at home?

S.M.: Yes.

D&C: There is undoubtedly something quite beautiful about watching the music come to life like this - did you think that the first time you made it happen?

S.M.: I'm not sure which "first time" you're referring to (but the answer is probably "yes" regardless).

D&C: It seems like it has been a quest that is bearing great fruit now - is this fair to say?

S.M.: It's been bearing great fruit all along. The main thing that's changed between 1988 and now is that more people are seeing my work. I'm hoping that in the future, I will get back to developing the animations more. I was expecting to do some of that as part of the Biophilia project, but the small amount I did when I first started work on the first song (in January and February) didn't end up making the cut, and after that I was too busy making an iPad app to do much creative work. The "flexing bars" effect used in Biophilia's opening song is the only new thing I had time to think up.

D&C: I have been reading the timeline on Musamin and was intrigued when reading about your instant embracing of the iPad - were you waiting for something like it to come along?

S.M.: For many music-related ideas I've wanted to explore over the years, the state of user interface technology has been a limiting factor. The iPad was the first affordable device with a multi-touch display, and there are many "wait until the technology is ready" projects that it made feasible. There are lots of kinds of user interface devices which, if they appeared, would let me do things that are on hold.

D&C: I've been looking at [the photos you sent] and was interested in the pictures of you playing piano surrounded by technology - in one picture you are playing with an iPad and two iPhones to your left and the lyrics "Crystals grow like plants": could you explain what is going on exactly?

S.M.: That's my setup for testing my part of the Biophilia app. I already had an iPad (because I wanted one) and an iPhone 4 (I got one as an employee perk), so for the Biophilia project I got an iPhone 3GS so that I could test my app on the slowest iOS device we were going to support.

D&C: What is next for you project-wise?

S.M.: The answer depends on the time-scale you're talking about ...

Immediate (next few days): When Biophilia landed on me (sort of like Dorothy's house), I was in the middle of the animation of the last movement of Beethoven's 9th symphony, using a recording by the ensemble Philharmonia Baroque; I'd like to wrap that up (a lot of my YouTube viewers are waiting for it). I've been working on the first movement of Bach's Italiano Concerto, and I'd like to do a video of that, showing my hands from two angles (as I did in this video). I'd also like to do the laundry and vacuum ("hoover" in your neck of the woods) and deal with everything else I've neglected during the last few months.

Short term (next few weeks): I'd like to release some of my own animations using the iOS player I developed for Biophilia.

Middle term (next few months): I've been collaborating with violinist/conductor/entrepreneur Etienne Abelin to develop a live-performance version of the Music Animation Machine, one that would allow musicians to use my animations in concert situations. The people at the Weill Music Institute are planning to include my animations as part of their instructional materials for their Link Up program next year, and as a complement to that, they'd like to use my live-performance system at Link Up's culminating event (May 2012), so they may help fund this development.

Longer term (next after what I've mentioned so far), I would like to make an iPad app that teaches you how to sight-read rhythm notation.

Longest term (when I know enough to be able to tackle it), are my more ambitious projects, the first of which is to make a video game that teaches you how to sightread contrapuntal keyboard music.

D&C: Are all the stuffed bears your own?

S.M.: No, they belong to my friend Liza; taxidermy was once one of her hobbies (lately she's more of a writer). My photographer friend Anna Detrick (she did about half the photos I sent you) and I dropped by Liza's home hoping to do something based on the wonderful picture of Björk with a bear, but the geometry didn't work out. The rest of the photos were taken at my home (and all the props other than bears are from my collection).

D&C: Have you always been interested in the way music can be presented visually? Is it something you were interested in at an early age?

S.M.: I've always liked making things that are interesting to look at, but the connection to music didn't happen until I was in my twenties.

D&C: Was it the case that you were frustrated by traditional notation and wanted to change it?

S.M.: Oh, no --- not at all. Traditional music notation works very well for what it was designed for. When I was younger, I was frustrated that I wasn't fluent in reading it, but that had nothing to do with the nature of the notation. New notations have been designed to do other things (e.g. figured bass for Baroque-era players to improvise from, lead sheets for jazz musicians to improvise from, tablatures for string players to read complex fingering and hand positions from, etc.), and I've developed some alternative notations myself (like a notation for flamenco heelwork), but I think attempts to "modernize" traditional music notation are, for the most part, misguided. Whether or not my animations of music are "notation" is a matter of definition, but they weren't intended as a replacement for traditional notation in the traditional context.

D&C: Regarding the 1974 LSD-induced hallucination that triggered the impulse to come up with the music animation machine: Can you remember what happened specifically? Was it the case that you could visualise what you had in mind immediately due to the hallucination? Were you taking LSD to induce this kind of Eureka moment?

S.M.: I began by putting Henryk Szeryng's recording of Bach's unaccompanied sonatas and partitas for violin on my Teac reel-to-reel tape recorder. I got out the score to follow along. The first movement I remember listening to was the Siciliano from the G minor sonata (you can watch a homebrew documentary I made in 1996 to get a little more detail about the first part of this story ... caution: low production values, poor color saturation, skip the long organ part from 3:11 to 11:47).

The hallucinations I had while looking at the score are hard to describe because what was different from normal perception was not that the notation looked any different, but that I looked at it differently. The result was that the notation seemed animated, as if it were dancing along with itself. The graceful shapes of the note forms and the graceful gestures of the music became a single thing. The progression from note to note seemed like footsteps. I could go into more detail, but you get the idea.

Over the years, I've returned to my memory of that episode many times and pondered the question of how to share the experience with others (aside from training them in music notation and becoming a drug dealer), and have come up short. The difficulty is that my sense of the notation being alive didn't come from what I was actually seeing, but from what I knew. When you look at a photograph of someone in action --- a ball player swinging a bat, a ballet dancer leaping, whatever --- you know (from a wealth of experience) what action is in progress, even though what you're looking at is a still image. In a similar way, although I was looking at a group of notes in a static musical score, I knew what kind of motion was in progress, what action they were a part of, and it was that knowledge that changed from moment to moment, and which gave rise to the dance I was watching.

The next thing I remember (after the Siciliano) is the Chaconne from the D minor sonata. This movement starts slowly, but as it proceeds through the variations, the notes go faster and faster. What was amazing to me was that my eye's saccades became synchronized to the rate of the notes, with the result that the "now note" (the note that was currently being played) appeared to be a single note head moving only vertically --- not horizontally. It reminded me of watching a fishing bob riding up and down on the surface of an ocean of surrounding notes ...

Then, the pattern changed, and instead of being relatively contiguous, the pattern of notes started jumping ...

I was amazed to find that my eyes were still able to track the motion, jumping wildly. At that point, I put down the score, stopped the recording, took off the headphones, and said to a friend who was with me, "I afraid that what I'm doing may be damaging my eyes." He said he thought that was unlikely, so I went back to listening/watching.

At some point, I put on the last movement of Bach's 4th Brandenburg Concerto, and was disappointed that the score-watching experience was nothing like it had been for the unaccompanied violin music. Score-following requires you to not only direct your attention to the correct horizontal position, but to switch from staff to staff (instrument to instrument) continuously (and quickly), and integrate information from many disparate locations. I wasn't experienced enough at score reading to do this easily and automatically, so it was just frustrating.

A few days later, I had the idea of making a new kind of score that would be easier to follow while under the influence of LSD. It was this that resulted in the bar-graph scores (you can see one of the scrolls from that phase of the project in the photographs I sent).

D&C: Lastly, you describe how your work at Audience, specifically your work on voice-processing chips, helped you learn about "neurobiology, psychoacoustics, and cognition". Can you explain what you learned in more detail?

S.M.: To go into this thoroughly would require more effort than I have the energy for at the moment, and would be more than the scope of your article could accommodate, so let me just give you a few short examples to give you the flavor.

I knew about the fast music-synchronized visual note-tracking I'd had with the Chaconne and similar effects I'd witnessed/caused in other situations experientially and viscerally, but I could not explain them scientifically. Studying the neuroscience of the visual system, I learned about saccades, and when I learned about the mechanisms behind synaesthesia (leakage of domain-specific cortical activity into functionally unrelated but structurally nearby regions) and the actions of hallucinogenic drugs, it started making sense.

One of the first projects I worked on at Audience was an algorithm for polyphonic pitch detection. I read research papers about human discrimination of multiple pitched sound sources, perception of harmonics, etc., and developed an algorithm based on what I learned. This made me more aware of what is involved when you listen to music with more than one pitch happening at a time, and the biological roots underlying the relationships between timbre, harmony, musical scales and tuning systems, orchestration, etc.

A big part of auditory perception is "grouping" --- how we know which vibrations at which frequencies are associated with which sound source (me talking versus the radio on in the background versus the street noise outside versus the baby crying, etc.). To make a DSP (digital signal processing) algorithm that separates speech (you talking) from everything else (all the sounds in the room you're in), you need to do this, and Audience's approach was to use what was known about human perception as a guide. I designed a piece of an algorithm that was roughly analogous to "attention" in hearing.

The short summary would be: people have been studying the nature of sound, the nature of auditory perception, and the structure and function of the human auditory system for a while, and some of what they've discovered is related to what we, as composers, performers, and listeners, are interested in. In my animations, I'm trying to help listeners organize their musical experience so that things which might not be noticed or not make sense under normal circumstance (due to lack of experience or inattention) are obvious and have the effect the composer or performer intended. Knowing how music perception works is a useful piece of the puzzle, and helps me make decisions about how best to present something.