I designed the Spirit Evidence Existence Kit (SEEK) to be self-consciously analog, DIY, and quite chunky, in both its technology and aesthetics; I wanted each piece to be accessible in terms of cost and understanding. For that reason I designed the laser-cut case in Tinkercad, an in-browser 3D design tool aimed at schools, and used the Micro:bit as the device's primary controller. The Micro:bit had been given to every primary school in the UK so it would be familiar to teachers, and was designed to be played with and reprogrammed. My only deviation was the use of Decawave's Ultrawideband (UWB) development board to do indoor location sensing. Experiments using bluetooth, radio, and infrared lasers weren't reliable of flexible enough.

I decided not have a screen from an early stage: I wanted to foreground the physical space, and not have it fighting for attention with a device. I also wanted to create a clear separation between the experience and the games and apps participants used every day on smartphones.

I originally designed the single device to have four different functions:

• GMeter: Finding where the ghost had been by displaying a simple 1-10 number as the device got closer to AR hotspots.
• Ectoscope: A trail finder that worked like a dowsing rod, allowing the participant to follow in the ghost's footsteps.
• Spirit sign: A decoding function that allowed you to 'draw' symbols left by the ghost by tilting the device.

(The ectoplasm was glow in the dark slime we had to convince building managers to let us put on the wall)

I had guessed that this would stop the investigation bits from becoming repetitive, and prevent a single super-enthusiastic member of a group from doing everything. This turned out to be a reasonable assumption which the audience proved wrong (see below.)

The first show used a chunkier prototype combination of Micro:Bit and Raspberry Pi in a 3D printed shell. It turned out to be unnecessarily complicated, and uncomfortable for the ten year-old audience to wear for the duration of the show.

I redesigned the device into the smaller, lighter version pictured above. The SEEKs' function changed as well as its form factor, as I had the chance to observe audiences engage with it in different venues and formats.

SEEK C++ / Typescript library

I wrote the software library that the students used to program their Micro:bits was written in Typescript and C++. The earlier V1 detector that used a Raspberry Pi was written in Python 3.

This was an interesting challenge as the original Micro:Bit had only 16K of memory (about ten tweets!) and a small amount of storage. So I couldn't have a lot of assets or files, and instead decided to write most things inside the code itself.

Below is a sample, a function would query the UWB board to find any nearby anchors, as well as the device's current location. This would be used by other functions to show proximity to hotspots and trails.

(The full main.cpp file is on the SEEK V2 Core Library Github.)

I would then call these C++ libraries in Typescript which was much friendlier to use on the Micro:Bit. Here the Gmeter SEEK (see above) gets its location, displays a little animation (the three showImage lines) and then shows a number showing how close we are to the nearest spooky hotspot.

The whole file main.ts is available on the SEEK Application Library repository

I could then expose these typescript functions as blocks in MakeCode, allowing even low level functions written in C++ to be used easily in visual programming.

Audience analysis and lessons learned

The project relied on an array of traditional evidence gathering methods:

• Observation: Observation by trained facilitators, academic partners, and performers. My role in the production was deliberately vague and boring: 'Junior Technician.' The original idea was that I could unobtrusively fix anything that was broken, but it also allowed me to observer how participants engaged with both the narrative and the devices I'd built, especially the SEEK detector.
• Surveys: These were light touch and used by both academics and teachers in school.
• Focus groups and post-show interviews: These recorded interviews allowed us to assess the impact of our work on the audience, and what had been memorable for good or ill.
• Workshops: I ran two in-world coding lessons, ostensibly to prepare students for field work. This allowed me to observe their engagement with the Micro:bits and their attitude to technology in general.

The key insight from this audience feedback for my work as a creative technologist was that there were too many things to do, and it was holding back collaboration. In my eagerness to ensure there were enough things to do for everyone, I had overcomplicated the field work portions of the production. In trying to design how far I could push the interactions in the show I had fallen into a more solitary model of individual achievement, and lost the collaborative play that should have been at the heart of the experience. Observing the audience play showed me the error of my ways.

This insight led to me dropping the spirit sign part of the ghost detector and focusing on a few simple challenges. This coincided with the third production at the garden museum: designed to be a museum late, it was much shorter than the production at York Theatre Royal, 45 minutes compared to an hour and a half. This leanness allowed participants to all work together rather than on separate bits of evidence, and created a feedback loop increasing the immersion for all of them.

Media, publications and resources

What allows this to work is the UWB sensor grid Actors in unreal are placed in the virtual room exactly matching where they are in the digital twin of the physical room. So the component above can use the world space locations of these virtual sensors as a kind of 'Rosetta Stone' to translate the real, physical sensor readings relative to their world positions.

Room is Sad Permalink to this heading#

Creator

Room is Sad appeared at the 2023 London Design Biennale, created in association with Charisma AI and King's Culture. The exhibition was designed by Alphabetical Design. It was seen by over a thousand people at the Biennale and during its remount at Bush House, King's College London.

For a full discussion of the project and its background, see a post I wrote on the King's Digital Lab blog about its development.

Room is Sad is an immersive experience that uses AI and the Internet of Things (IoT) to tell the story of a smart room that isn’t feeling quite itself. It's feuding with the desk upon which its laptop brain sits, and it's worried that it's best friend -- the plastic plant on the shelf -- isn't talking to it. The participant wanders into this tangle of inanimate object angst. They can be an impartial observer, a peacemaker, or deliberately cause trouble. It’s up to them.

Room is Sad was designed as a humorous, accessible way to engage both an academic and general audience on questions of identity, privacy, and intelligence.

The Technology Permalink to this heading#

• Charisma AI. A branching storytelling engine that uses Natural Language Processing (NLP) to understand users and serve responses. This is how the Room talks and understands players.
• Unreal Engine 5. Serves as the director of the experience, tying everything together with a beautifully terrible interface straight from 1994. It does the UI and passes messages to Charisma AI through itsSDK and to devices in the room via MQTT with plugin for MQTT communication.
• Internet of Things (IoT). Smart bulbs run by a smart home platform called Home Assistant. UWB radios give me a precise location for the player and some props (the plant) using sensors. This gives the room a physical sense, and can react based on the participant's movements and those of objects in the room.
• AI Guest Stars First, I used Stable Diffusion to visually represent its mood and subconscious (below) changing based on the interactions with the audience. OpenAI's Chat GPT-4, served as its therapist, but I didn't use it to write any of Room itself.

Room uses the same plugin library as BridgeXR to translate uwb sensors into unreal, in this case to allow Unreal Engine to inform the AI in Charisma about the movements of people and objects in the room. In both projects, I affect actors with UWB and IoT calls from MQTT by attaching an MQTTActorComponent that listens to delegates from the MQTTManagerComponent attached to the game mode, which is subscribed to the topics.

The result is that the Room AI can be informed by Unreal (via MQTT) when things change in the room, affecting its mood accordingly.

In the clip below, the Room asks the human to bring the plastic plant (its best friend) closer to it, and registers when the person has done so. (You can see the mood portrait change.)

Glow: Illuminating Innovation

Creative Technologist

A multi-sited London exhibition, showcasing groundbreaking artworks by leading women artists working with cutting-edge technologies. From early AI explorations in the 1980s to Virtual Reality (VR) in the 1990s, to virtual world creation in the 2000s, and portable VR in the 2010s, GLoW: Illuminating Innovation chronicled the often overlooked impact of women on the evolution of technology.

I was a supporting technologist for this project, working with four artists who were commmissioned to create new work. My work (with colleague Neil Jakeman at King's Digital Lab) focused on giving advice on immersive XR technologies, from real time engines like Unreal and Unity to WebXR frameworks, as well as scripting in Python. The real challenge was evaluating these technologies from a creative perspective, in conversation with the artists' ir work as it evolved, trying to find the right mix of tools that would be effective, sustainable, and live up to their creative vision.

More on the artists and the project at the GLOW website

Stem Cell City (beta)

The majority of my development work in GLOW was for the artist Yarli Allison's work Stem Cell City. Yarli's work uses video, 2D art (and some VR experiments) as well as physical sculpture.

Yarli didn't have a lot of experience using the technologies available in GLOW, so the initial stages of our collaboration were several technical experiments, beginning in Nottingham University's Virutal and Immersive Production Studio, that would giver her a chance to play with various immersive technologides and better understand which would fit best with her artistic vision. I worked with YArli during these experiments, to try and find some combination of technologies that would fulfill our criteria:

• Be accessible to Yarli (technically) so they could use them in their practice and understand how to get the most out of the technologies with our support.
• Be sustainable in the longer term, so Yarli could use and remix what was made without our help after the project ended.
• Fit the format of the GLOW exhibition, in terms of space and hardware available.
• Be accessible to our expected audience, casual foot traffic on the Strand in London.

Our audience was expected to be pretty broad and casual, in the sense that the exhibition wanted to appeal to those just passing by on the street as much as people who came explicitly to see the works. So asking the audience to download and app or go through a long onboarding experience would have put up a significant barrier to engagement.

Although we did some experiments in Unity, we settled pretty quickly on doing something in WebXR. As it's in-browser there would be no downloads and it would be compatible without almost any modern smart device. Yarli gave me a series of 2D assets she's been building as well as some 3D models she'd made that were processed by my colleageu Neil Jakeman to make them more web-friendly. From these I built a virtual experience according to her designs, a multi-room clinic that was part of a feminist utopia, using A-Frame and Three.js.

The assets made use of a lot of animated gifs: for TV displays, characters, and interactions. These don't work very well inside the A-Frame context, so as part of the development I wrote several custom A-Frame components, including the one below which essentially takes the single frames of the gis and applies them as textures to an object like a flipbook.

Other components I made, including interactions components to interact with posters, radios, and even a sleeping patient (to play a cell division minigame) are in the project's repository.

Before/After

Writer, Technical Artist

This short film was the final assignment of the Virtual Production certificate course I took at the National Film and Television School. We built an environment to be used as backdrop on the NFTS volume and then filmed a very short (2-page) script with actors. It gave us a chance to not only do the technical work, but have actual experience of working on a VP set under a heavy time pressure - we have just under four hours to setup, rehearse, and film everything.

I collaborated with some brilliant fellow students with a range of skills - director, cameraman, VFX artist, and designer. We created the concept together, and then I wrote the screenplay. Telling a story in essentially one location in 2 minutes was a real challenge, and I also wanted to leverage some of the creative potential of having a realtime background by making a story that allowed it to radically change. I also helped to create the environment, making sure it was stress-tested and efficient so we didn't have any nasty surprises.

As part of the course I also created a short film in Unreal, a previs is based on a screenplay, The Invasion, I co-wrote with Alex Walker.