Format 3 - An Interactive Sound Experience For A Museum

Creating interactive spaces with hidden sensors, that trigger sound or projection mapping, can create a truly immersive experience. These spaces blur the line between the physical and digital worlds.

Format 3 is just that. It's an interactive project by FOO/SKOU that makes use of conductive paint and the Bare Conductive Touch Board, amongst others. We caught up with the team behind Format 3 to find out more.

FORMAT 3 from FOO / SKOU on Vimeo.

Hi Martha and Louise! We recently came across the project you designed for the FORMAT which uses immersive technologies. Can you tell us more about the concept?

Format 3 is a graphic spatial and interactive musical score. It's an interactive sound experience, using a custom-made iPhone app and touch reactive sculptures, commissioned by the MAD museum for the three floors of their stairwell. The graphic score is historically a form of musical notation that utilises visual symbols to represent music.

We explore how a graphic notation can unfold in physical space, in the digital age. We have developed the project as a dialogue between image and sound. Format 3 is about meditating on what role technology plays in our relationship to the physical world.

How did the collaboration with FORMAT happen? What is your background?

We are both Danish-born and NYC-based artists. Our collaboration started organically, from collaborating on our thesis projects in 2014. Our formal education is in Art/Design (Martha attended the Royal Academy of Fine Art, School of Design and Louise studied Visual Art and Creative Technology at The Interactive Telecommunications Program (ITP) at NYU ), and at the same time we both play music, so working with sound as a material for artistic exploration was a natural convergence of the two worlds.

We noticed that you used the Bare Conductive Touch Board to be able to trigger sound. Can you give us more details about the technical part?

The Bare Conductive boards are great! We used the Touch Board in the installation "Earth, Water, Fire" to play mp3s when visitors touched the shapes.

For the panels "Balance" "Citybirds" and "Duet", which are a bit more complicated pieces, we used Ableton Live to play the tracks, so this circuit involved a Teensy microcontroller and the MPR121 breakout board (also used in the Bare Conductive board) to sense the capacitive touch elements.

The Teensy is able to act as a MIDI controller in Ableton, which is really convenient for more complex pieces. We were also able to connect all three pieces to one computer running the software.

How did you record the sounds? Did you use any particular programme?

The alphabet consists of 81 sounding symbols, inspired by the elements earth, water and fire - represented by the square, the circle and the triangle and the three categories: Grain (concrete sound earth, water, fire) Cyma (vibrations from the human body) Sono (data of animal sound)

The Format 3 app reads the graphic symbols and translates them into loops of sound. All of the sounds in the alphabet come from the 9 basic sound and image sources that have been manipulated through different processes to create a series of audio-visual effects, by literally cutting up and putting it back together.

With the sounds, we did this in a DAW. Specifically, in this case, we used Ableton and a bunch of plugins. We have named those effects after mental states such as focus, scatter and order. We think of the project as a meditation over the connections between the spiritual world and the physical world, and the tools and technology with which we engage. The 9 basic sounds are a mix of recordings and found sounds. They are: Rumble of the earth, low-frequency chant, a wolfs howl, water, a heartbeat, a dolphin’s clicks, fire, high-frequency chant, and a bird.

What was the most challenging part and what did you learn from dealing with it?

Pitching a large-scale installation to a museum and being very specific in dealing with measurements and budgets, before knowing exactly what you are creating. But the end result turned out to look surprisingly close to the early sketches/prototypes - so we learned that it paid off to go with our gut feeling.

You also used conductive paint. What other materials did you use and why?

Format 3 is a series of works in various media: prints on paper, canvas, a tapestry + iPhone app + sound. And wood, conductive paint, electronics + sound. We combine analogue and digital worlds.

Besides the pieces that work with the app, we have created a series of wooden sculptures that function as instruments or little compositions in themselves, in which you can dive deeper into concepts from the alphabet. The wood sculptures all have sensors - made with conductive paint and electronics that audiences can activate the sound by touching the sculptures

''Duet'' (the circle board) explores the human voice and relationship between pitches - harmony and is actually the two of us singing together. ''City birds'' (the triangular board) comment on how birds change their songs in the city, because of noise pollution and ''Balance'' (the square board) is a meditation over the balance between frozen and melted water on the planet.

How do you find the use of technology in art and design? What fascinates you?

We work with ideas of music, sound, and space that have existed for many artists, musicians, scientists, innovators and thinkers before us— for example, Newton combined tones to colour. Throughout history, there have been so many experiments of visualising, notating, understanding and experiencing music in new ways, and those experiments have been part of pushing our understanding of what music and sound is.

Our project brings to this tradition the novelty of our own emerging technology— for example, the iPhone. Or the possibility to make physical things trigger sound, with conductive paint and electronics. And through developing our own language— our own alphabet— we are creating a reflection of our reality, right now. 'Format 3' is about meditating on what role technology plays in our relationship to the physical world, because our current society has become about the digital world, social media, etc.

We think that there are many aspects of sound that have yet to be discovered, and that is what draws us to further explore this medium’s potential. What will the future sound like when machines no longer make mechanical sounds? What other uses could sound have beyond music and entertainment? Scientists are exploring sound levitation, for example, as a way to move materials around without touching other materials—that idea seems like magic to us!

What was the best moment of watching the audience interacting with the installation?

There have been so many great moments, it’s hard to choose one. It is really people of all ages that seem to enjoy the installation. You can think of the entire installation as a musical composition, in physical space, that is performed in those moments when our audiences interact, so without the audience, there would be no music.

Format 3 was commissioned by the Museum of Art (MAD) for the exhibition 'Shaping Space with Sound' running from September-February. The app is made in collaboration with programmer, Aaron Sutula, and the touch reactive sculptures are in collaboration with creative technologist, Mark Kleebak.

Images & Video: Fooskou

We love it when you share your projects! Post your project on Instagram, YouTube, or Twitter, and make sure to tag @bareconductive or use #bareconductive. You can also send your videos and photos to info@bareconductive.com so we can post them on our site for the world to see.