Synesthesia in VR is the world’s first interactive synesthetic experience. Essentially, through AR and VR technology, people now have the opportunity to see through the eyes of a synesthete.
Don’t know what that means? Imagine seeing music as shapes, or days of the week as colors. German agency Denkwerk was able to recreate this experience: which is why Synesthesia in VR landed a 2018 Webby Award in Best Visual Design – Function.
We spoke with the Denkwerk team about what they hope people gain from this experience, and more. Here are their answers below.
“ We have tried to draw a balanced picture with all the wonderful advantages but also the few disadvantages that a synaesthete has in his everyday life.”
What inspired the idea to create the world’s first interactive synesthetic experience?
We already knew the synaesthesia phenomenon before, but only in broad outline. We actively established contact to the synesthesia society and were very quickly fascinated by synaesthetic worlds and the dazzling and deep synesthetic personalities. I think it was the desire to look into this world and experience it one to one that drove us the most. Only by the technological development of the last years and the developments of WebJL and AR/VR were we able for the first time to look through the eyes of a synaesthete. And of course this was also one of the driving factors, to be a pioneer and to be the first to do and see something amazing.
It’s a beautiful project that combines virtual reality, sound, and color to recreate how synesthetes interact with the world. How did you ensure this project accurately reflected how those you interviewed experience the world?
The exact translation of personal synaesthetic phenomena has taken us the most technical and human effort. We started the process with intensive interviews and surveys. In the surveys we were able to use already established simple colour tables for synesthetes to determine specifically how colours are seen in tones, letters, numbers, and weekdays. Afterwards, a 3D artist visualized in several more intensive one-to-one cessions specifically selected synaesthetic perceptions. It helped us the most if the synaesthete could draw, otherwise it became very quickly very complex to capture something visual only through speech. In many steps the 3D artist approached the exact post-binding of the phenomena in 4 dimensions. Interestingly, in the end only women were willing to go this very personal way with us.
Synesthesia in VR
How do you hope and imagine this project is used to positively influence children who have not yet been “diagnosed” as synesthetes?
We have already received a lot of positive feedback from parents of synaesthetic children. Dr. Zedler, a worldwide expert in this field, has also told us he uses our visualizations and tools in conversations with parents and children and recommends them to his colleagues in the field.
Parents can now show the phenomenon directly to many simple teachers and doctors and let them experience the world of their synaesthetic child.
What do you hope users take away from this experience, regarding synesthetes and our understanding of difference?
It was important to us not to present the synesthetes as victims but to show the enrichment of their world… Of course, the phenomenon also has disadvantages, especially in childhood and puberty, when it is particularly difficult to be different. We have tried to draw a balanced picture with all the wonderful advantages but also the few disadvantages that a synaesthete has in his everyday life. What we most want is for someone to pick up the ball and build synaesthetes a tool with which they can easily recreate their visual experiences even in 4D. Our budget was far from sufficient for such a complex task.
What does your Webby win mean to you?
We see the Webby Award as the holy grail of the digital world. This is one of the few awards curated by the world’s digital experts. The moral compass of individuals and agencies in the course of mechanization with AI, XR and IoT is becoming increasingly important to preserve our humanity. So we are especially proud to have been honored for a project that moved us so much and demanded everything from us both humanely and technically.
