I have just come to the end of two of the most exciting (and longest) weeks of my career.
I am writing this in the production office of our New York show Sleep No More. Running for a little over a year, the production is an immersive retelling of Shakespeare’s Macbeth, inspired by Hitchcock and set in a 1930s film noir world.
Participants are masked to explore a transformed six storey night club, with only their senses and instincts to lead them. Our Nesta Digital R&D fund project, developed in collaboration with MIT Media Lab, has been embedded within this project and during the week of May 14-19 we ran its five-day test phase.
We have been blessed with a great digital partner – MIT (Massachusetts Institute of Technology) is a leading research-based university with a strong emphasis on scientific and technological education and research. It is devoted to research projects at the convergence of design, multimedia and technology.
Our lead staff contact is professor Tod Machover, leader of the Opera of the Future group. Along with his project manager Simone Ovsey they have assembled a team of talented students who specialise in live performance and physical technologies, networking and online interfaces.
The relationship between the two organisations blossomed naturally after Punchdrunk staged Sleep No More in Boston. Many of the students had seen the show as stewards or audience members and we were invited to talk at the lab.
From there we helped them celebrate their 25th birthday by building a secret speakeasy in the basement, theming it narratively around Marvin Minsky and his pioneering work in the field of Artificial Intelligence. So when we began thinking about the Digital R&D project it seemed like a perfect progression for us both.
The project’s aim was to connect a live Sleep No More audience member to an online companion. We wanted to see if we could create an online experience which lived up to the visceral intensity of the live show and facilitate a shared experience which takes place in both the performance space and a remote user’s location.
We have tried to achieve this using a combination of the following: a specially built mask (one that houses dynamic communication and location based technology), pre-recorded and live fed audio and video (to both real and online participants), portal interactions (which used physical props allowing real time one and two way communication between participants), live video feeds from hidden cameras, live performance and bespoke graphics and props.
The project saw 13 pairs experience the project over five nights. Each night’s experience was followed by evaluation sessions with online and real world participants – these sessions allowed us to use feedback to keep adapting and changing the experience throughout the week.
Even in our eyes the project was ambitious, not least in terms of time scale. Working across three separate locations and two time zones has not been without its challenges either – we have had only snatched face-to-face time, with most work done via Skype and email.
The physical installation of the project saw us run over 8,000 feet of CAT-5 cable around the site, linking a 100MB internet connection to our control hub within the building with individual runs that broke out to 24 access points. This allowed us to create a network across the building to live stream sound and audio content to both live and online participants.
Using a combination of 10 RFID readers and 50 Bluetooth devices we installed a system that allowed us to track our participants’ progress through the space, ascertaining their exact location at any point.
We also installed 10 physical portals, which allowed participants to communicate – these included a poltergeist-like book which online participants could cue to flip a real book off a shelf when their participant was nearby and a typewriter that allowed online participants to type direct messages to their real-world companion.
Alongside these elements, we developed a complete online storycode system, which combines text-based adventure with imagery, pre-recorded video and both automated and operator based interaction. The Media Lab story team have written around 5,000 lines of narrative which translates the world of Sleep No More into an online experience.
The real challenge for Media Lab has been creating our so-called ‘cauldron’ system, which allows all of the different elements to talk to one another, creating cohesive and meaningful experiences for individuals both in their own environments and in the shared space where they come together. The team estimate around 100,000 lines have code have been written to make the project work.
Now that the dust is beginning to settle, we have the job of consolidating and evaluating the project. To say it was a glowing success would be inaccurate – we were treading a fine line between game and experience, in an already delicately balanced performance.
Practically speaking we had too little time, with our Beta testing rolling all too closely into the actual live test period. The technology was beginning to do what it should by the end of the week and we found ourselves beginning to make exciting discoveries just as we had to pack up.
Was it a game? Could the experience be solved? What was my mission? Did you feel connected? Did we need to make things more linear and visible? These were all questions that we have only just begun to interrogate.
We asked ourselves: how could we create a project that begins to give our shows the reach and access of a venture like NT Live? The aim was never to copy this model – our work is about individual experiences and personal connections. And we were equally adamant it wasn’t about plonking a camera on a real world participant and instructing them, drone-like, to move around the building.
Instead, the challenge has been to recreate the infinite possibilities for journeys and experiences happening simultaneously across a Punchdrunk production.
The system we have created has allowed us to stream multiple experiences at once. It’s been about creating an extra level of detail and intrigue inside an already rich experience, allowing online users to connect to this and simultaneously go on an individual journey in the comfort of their own home.
We have learned valuable lessons this week and know that these open up rich territory in what we’re coining ‘remote and real world interconnected theatrical immersion’. We’re sure that we are well placed and committed to explore future iterations of this model and that this is an exciting development for artists and technologists exploring these modes of interactions.
Peter Higgin is enrichment director at Punchdrunk.
Source – Culture professionals network: Culture professionals blog | guardian.co.uk