Top

Pictures With Sound

• Abstract acrylics by Tanya Khachiyan.

Background

Recently, I added an experimental audio feature to the Dogulean (dog-you-LEE-an) Planetarium (here or there). That feature uses a simple color to sound translation on colors of stars, planets, and moons.

I decided to try that translation technique on actual art in order to see what sort of sounds it would produce and, maybe, if it would help color-impaired people appreciate art that depends strongly on color, as Ms Khachiyan's work does.

I would like to try the technique on a wider variety of pictures, but it is clear that different styles of art will need the technique to be extended in different ways.

For Ms. Khachiyan's work, I have made some relatively modest extensions and modifications to the method used for the Planetarium. For Ms. Khachiyan's nonrepresentational, vividly colored work, the resulting atonal, unstructured "music" actually seems appropriate. I also feel that it helps me follow the details of the paintings.

Technical Background

• Hue – the "color of the color", given as an angle from 0 to 360: Red-Yellow-Green-Cyan-Blue-Magenta-Red.
• Saturation – the intensity of the color. The value 1 means "colorful color", 0 means "no color": white-grey-black.
• A measure of brightness or luminance.

The obvious way to use this representation is:

• Hue determines the pitch of the sound. There are three options when deciding how to do this:
  • What range of frequencies should be produced?
  • What hue should be mapped to the low and high notes? (Hues form a circle that has to be cut and stretched out into a straight line.)
  • Which direction around the circle of hues is "up" in pitch? For example, is the pitch produced by red lower or higher than the one produced by orange?
• Brightness or luminance determines how loud the sound should be.
• Saturation means — what? Low saturation plus high luminance means "white", so in the Planetarium I used white noise (actually pink or red noise) to represent low saturation. But this makes a hiss that I have never really liked, so for these pictures we generate a series of random chords in the same range of pitches as is used for the hue. This makes a pleasant babbling brook sort of sound. (That, by the way, tells us what makes a babbling brook sound the way it does.) Its character can be changed greatly just by changing the frequency with which the chords change. Choosing the frequency of the babble can do a lot to make the "noise" fit in with a picture.

As things stand, this approach has a couple of obvious drawbacks:

Black should not always be silence – it might have emotional significance, or it might be an outline that should make a noticeable noise when it is crossed.

Sometimes white does not mean brightness. Rather it may mean background that hasn't been painted and should therefore be silent. And maybe a well-lit, low saturation color should be quiet, not "noisy". Or maybe pink (low saturation) should be similar to red (fully saturated) and just as loud, but somehow softer.

There, then, are two things to work on. It is clear that translation of color to sound needs to be tailored to the character of the picture.

Ideally, the sound should be integrated with the art, as with musical theater of various kinds, so that it is somehow meaningful. On a trivial level, one could could fairly easily make an image that plays a tune when you slowly drag the mouse back and forth across it. On a less trivial level, if color has an emotional value in a painting, one would like it to produce a sound with a similar emotional value. But that is a distant ideal.