Pictures With Sound


Recently, I added an experimental audio feature to the Dogulean (dog-you-LEE-an) Planetarium (here or there). That feature uses a simple color to sound translation on colors of stars, planets, and moons.

I decided to try that translation technique on actual art in order to see what sort of sounds it would produce and, maybe, if it would help color-impaired people appreciate art that depends strongly on color, as Ms Khachiyan's work does.

I would like to try the technique on a wider variety of pictures, but it is clear that different styles of art will need the technique to be extended in different ways.

For Ms. Khachiyan's work, I have made some relatively modest extensions and modifications to the method used for the Planetarium. For Ms. Khachiyan's nonrepresentational, vividly colored work, the resulting atonal, unstructured "music" actually seems appropriate. I also feel that it helps me follow the details of the paintings.

Technical Background

The technique used is first to decompose the color into three values: There are several such systems. I have used HSI – hue, saturation, intensity – supposed to be best for recognizing colors and shapes as opposed to specifying them.

The obvious way to use this representation is:

As things stand, this approach has a couple of obvious drawbacks:

Black should not always be silence – it might have emotional significance, or it might be an outline that should make a noticeable noise when it is crossed.

Sometimes white does not mean brightness. Rather it may mean background that hasn't been painted and should therefore be silent. And maybe a well-lit, low saturation color should be quiet, not "noisy". Or maybe pink (low saturation) should be similar to red (fully saturated) and just as loud, but somehow softer.

There, then, are two things to work on. It is clear that translation of color to sound needs to be tailored to the character of the picture.

Ideally, the sound should be integrated with the art, as with musical theater of various kinds, so that it is somehow meaningful. On a trivial level, one could could fairly easily make an image that plays a tune when you slowly drag the mouse back and forth across it. On a less trivial level, if color has an emotional value in a painting, one would like it to produce a sound with a similar emotional value. But that is a distant ideal.