The social life of mixed reality – Yoyo Munk

How can mixed reality technologies create emergent social experiences? Scientist-artist Yoyo Munk (US) joined SLV's Paula Bray and CEO Paul Duldig to discuss blending virtual geometries with shared physical space in their recent KAGAMI and Medusa projects. Yoyo asks how technological imperfections can evoke meaning, and explains their journey from scientist to artist by finding the signal in the noise. Together, they wonder how mixed reality might interact with library collections, and the potential for digital visualisations to not only reveal the dynamic evolution of knowledge archives, but to shape future collecting practices.

This conversation was recorded a few months before Yoyo joined SLV LAB as its first Creative Technologist‑in‑Residence. His resulting project, In Memory | Of Being, was a mixed reality installation built from the Library’s digitised holdings: thousands of unidentified portraits, fragile glass‑plate negatives, vintage stereographs and oral history recordings.

Transcript

0:00:02 - 0:01:17
Dilan Gunawardana
The best word to describe what you experience in the mixed reality works of Yoyo Munk is “encounter”. Think of walking amongst softly undulating virtual columns that wrap themselves around you as you move through a gallery space, or imagine coming face to face with creative luminaries in digital avatar form and observing them from all angles in a communal experience.Virtual works like these and several others, are the products of a scientist-turned-artist who began by researching the biomechanics of gliding canopy ants, before turning to large scale art installations that explore how we perceive how we move and how we make meaning of what we experience together.I'm Dilan Gunawardana, Digital Engagement Producer at State Library Victoria, and you're listening to an SLV Lab podcast, part of its Conversation series, where we discuss emerging technology, digital experimentation and library futures with artists, technologists and workers in the cultural sector.0:01:17 - 0:01:37
Yoyo Munk
I like in my own work to not have control over everything that is happening in the room. The behaviors of the pieces that I'm presenting are inherently kind of a collaboration between whatever ideas I put into it, and how an audience responds to it within the space self.0:01:37 - 0:03:15
Dilan Gunawardana
In 2025, Yoyo Munk not to be confused with the famous American cellist Yo-Yo Ma, became SLV Lab’s first creative technologist in residence. The brief was open ended, however, Yoyo challenged themselves not to create new material, but to work entirely with the library's existing digital collection. Images and recordings. Fragments of history already held within the institution. The result was In Memory | Of Being a mixed reality installation, developed over eight weeks in a gallery overlooking the La Trobe Reading Room.You can read more about this work on our website. Before this residency officially began, there was a conversation recorded earlier in 2025. Yoyo joined State Library Victoria's Chief Digital Officer Paula Bray and then CEO Paul Duldig for a wide ranging discussion about mixed reality as a medium, its technical challenges and social possibilities, and how it might be used to help our visitors encounter our vast collections in new and interesting ways.The following conversation was recorded on the unceded lands of the Wurundjeri Woi-wurrung people of the Kulin nation. We acknowledge the traditional lands of all the Aboriginal clans and their cultural practices and knowledge systems. We'd also like to acknowledge and pay our respects to any First Nation listeners joining us for this conversation.0:03:15 - 0:03:20
Paula Bray
Yoyo, can you just give us a little background into your practice?0:03:20 - 0:05:01
Yoyo Munk
Of course. So my background is as a scientist. I studied as a physicist originally and then transitioned into working in biology for my graduate work. I focused mostly on insect flight aerodynamics and navigational behaviours, and neuroscience for my academic work, and then transitioned into working with augmented reality after leaving my postdoctoral work. For a while, I spent several years at the startup Magic Leap in the United States, where I founded and led the Quantified Experience Group there, which was doing experiments on human vision and visual responses to augmented reality displays.And then for the last six years, I've been working to create content for mixed reality exhibitions, primarily in collaboration with the content collective Tin Drum. We produced a number of pieces that have been publicly exhibited. The first piece that was shown was 'The Life', which was a volumetric piece featuring the performance artist Marina Abramović which showed at the Serpentine Gallery in London in 2019.In 2021, we showed a piece where I was the primary artist and creator of it called 'Medusa', and that showed as the headlining show of the London Design Festival and featured in the Victoria and Albert Gallery in London. Then most recently in 2023, Tin Drum produced a very ambitious volumetric piece called 'Kagami', which has toured the world and most recently showed as part of the Asia TOPA festival, most recently here in Melbourne.0:05:01 - 0:05:16
Paula Bray
Yeah, it was fantastic. Both Paul and I have experienced Kagami and really enjoyed it. Can you tell us a little bit about working in mixed reality as opposed to VR, when creating works like you did for Kagami and Medusa?0:05:16 - 0:08:01
Yoyo Munk
Mixed reality means different things to a lot of folks. I guess when I talk about mixed reality, I'm usually thinking about a headset technology that features an optically transparent display. So when you wear one of these devices, your vision of the real world is unobstructed, at least in the sense that the light that is being reflected off of objects in the real world is still going into your eye without going through a camera in the meantime.So this is distinct from camera pass-through style devices, where it's a VR helmet featuring a live feed of a camera from the outside, and the devices themselves are capable of showing virtual geometry overlaid with the physical world which you are inhabiting. So mixed reality in that sense, the idea that you are taking three dimensional virtual geometry and trying to integrate it into a real world space, is vastly more difficult than doing VR.One of the benefits of VR is that you essentially have complete control over the sensory apparatus of the human that's involved, right? You are replacing the visual input to the eyes. If you have a nice big pair of headphones, you're controlling the sound. In the mixed reality sense, you're having to make a piece that where you've got virtual geometry that is trying to integrate into that real world space, and there are immense challenges that come with trying to make that geometry fit in a way that feels cohesive and it's not jarring and creating an uncomfortable sense of friction between the virtual geometry and thereal world that people still very much feel a part of. There are some big advantages of it, I think, and one of the most important things from my perspective is the the opportunity for the sociality of the experiences that you create through this method. It is certainly possible to have VR experiences where there are many people wearing VR helmets in a room, and you have some kind of avatar representation of other people and where they are in that space.But like, this is a very tiny microcosm of all of the information that is encoded with the non-verbal communication between humans and crowds and the idea that you still have this sense that you are inhabiting an exhibition space and have that sense of nonverbal cohesion with the audience that you are a participant in, I think it's hard to overestimate in terms of its importance in creating an experience that that feels impactful.0:08:02 - 0:08:33
Paul Duldig
Can we talk then about Kagami? So both Paula and I saw Kagami and it was a really interesting experience. I think what you're saying certainly came through in Kagami, where, as a mixed reality experience, it wasn't really using much of the room. It could have been anywhere. But the fact that you could see other people experiencing the same thing you're experiencing and walking around Ryuichi [Sakamoto] and sitting down and just looking and so on.So it was more of a social version of mixed reality, perhaps.0:08:33 - 0:09:54
Yoyo Munk
Yeah. What I like most about an experience like Kagami is, I think if you're if you're looking for the kind of detail that you might be able to find in an exquisitely filmed performance or the kind of sound that you would get from a studio recording and a piano in that room, this is not going to... it is not intended to surpass or replace any of these forms of communication.There are many, many valid ways of interacting with the work of any particular artist. But where I feel like this can be most successful is when you are in a room with a small number of other people, and some of the artifice kind of melts away at some point, and you feel that you are in a room with Ryuichi playing a piano for a small audience of people.Like, I love standing around on the outside and just seeing the people who get up out of their chairs and they go and they just sit down cross-legged on the floor because they get to sit right next to the piano bench. And in the one sense, it's not real. I mean, the image there is virtual, but he feeling of being there is difficult to achieve through other methodologies.0:09:54 - 0:09:59
Paul Duldig
It might be worth just to give us a quick reminder of what Kagami is and then perhaps do the same for Medusa.0:10:00 - 0:11:42
Yoyo Munk
Yeah. So Kagami is a performance, a piano performance by Ryuichi Sakamoto. It was volumetrically captured in a studio in Japan in late-2020. And the way the experience unfolds is you have a circular arrangement of chairs, and an audience is invited into that space. And so all of the participants in this experience start out seeing Ryuichi at a piano playing in the centre of the room, and it proceeds linearly through a selected curation of songs – it has a sequence that runs through, and it goes through the same way each time in terms of how the songs play out.But the feel of any given performance is very different in terms of how the audience of that day interacts with that space and how they choose to inhabit that room and interact with one another, or have that sort of like ephemeral connection between the strangers that you have in live performance. And so I think it's something that feels like it has a more inherent sense of individual agency within that audience.That is somewhere in between the idea of like being at a live concert, having that sense of energy of a performer in the room that you feel like you are there, you are seeing something that's happening right now, versus the idea of sort of sitting by yourself and listening to music on headphones, or watching a film in the comfort of your own home.This operates within sort of a space in between.0:11:42 - 0:11:44
Paul Duldig
It could have been VR as well, couldn't it?0:11:44 - 0:12:11
Yoyo Munk
I mean, certainly from a technical perspective, there's nothing that would prevent this from existing inside of a VR space. I think one of the things that you would lose in that case is for ability to do a large audience. We would have people bumping into one another quite a bit more inside of that room.So like the fact that these devices are transparent is helpful in terms of being able to just ensure some degree of audience safety.0:12:11 - 0:12:20
Paula Bray
I really enjoyed that you could see people in layers in between the visuals and watching the hands of the piano play.0:12:20 - 0:14:29
Yoyo Munk
I like that aspect of this piece quite a bit. I mean, you had asked a little bit about Kagami and Medusa. So in Kagami we operate in a fairly dark room and, as you say, the actual environment of the room itself is background. It's not something that is intended to... there is a mixed reality component to the idea that you see your fellow audience members.But the room itself, aside from the theatrical lighting, that kind of backs up some of the the virtual effects that happens in that space. The focus visually is really on that virtual aspect, on Ryuichi playing in the centre of the room. For Medusa, this is a fairly different kind of idea. So, this is not a volumetrically captured pace.This is more of a generative, evolving experiment within the idea of trying to do architecture and structure within a virtual space, taking the idea of an environment that has a sense of volume and a sense of of occupancy within a room, but completely removing any physical aspect to it; this is a purely virtual sculpture/piece of architecture floating in that room.And there the idea was very much to... the lights are bright in the room there, and the room itself that contains the sculpture is a very important and integral part of how the piece works. So it's a very minimalist piece. It is composed of just shy of 10,000 hanging columns that move through the room and subdivide space, and change the way that they form themselves into an emergent structure based on how the structure is inhabited by the audience in real time.But at all points, the focus, as your vision is drawn up through that structure, there is the palpable background of the environment in which it is housed, and it's designed to be so that the architecture of the room that it inhabits is an equal player in the total architectural experience of the piece.0:14:29 - 0:15:24
Paula Bray
One of the things for me at Kagami was the introduction. You walk, you know, you have an onboarding experience before you go into the room, which is quite beautiful. It's got scent, it's got amazing visuals and you stay in there for a little bit of time. But then when I was experiencing the next phase in the room, Todd [Eckert] actually came in and talked a little bit about the experience for the audience and described it as not perfect.You need to forgive a little bit for some of the experience in terms of the... you're working at the cusp of advanced technologies. Can you talk a little bit through how that sort of onboarding with using advanced technologies in those experiences, whether that's important or that visitors appreciate that sort of introduction?0:15:24 - 0:18:46
Yoyo Munk
Yeah, I think that it is certainly like it can be jarring just to throw a headset on someone and just be like, all right, off you go. The idea of preparing the ground and having a space where you are transitioning into whatever experience is about to happen, I mean, this is not something that's unique to headset based experiences, right?This is like that idea of having some degree of expectation setting, of having this sort of sense of welcoming, especially when you know that what you're asking people to do next is something that they are probably they don't really have a reference point for. You're looking to essentially project this idea that what you're about to see is going to be different from things that you may have seen before.But this isn't a threatening space. Nothing bad is going to happen here. It's good to just have a sense of calm going into it. I mean, I think that with regards to speaking at the outset and talking about sort of perfection or lack thereof, I mean, I think it can go both ways.I mean, you want to avoid seeming like you're just being self-deprecating for the sake of it, right? And I mean, personally, I think that especially within that space of volumetric capture, the imperfections for me often lend a sense of poignancy that enhances my experience of it. Generally, the idea with any volumetric capture piece is that you are assembling a geometry, a three dimensional geometry of whatever happened inside of that capture space, and you're assembling that from the images that are taken from an array of cameras around that room.And this is an inherently algorithmic process. There's nothing in this pipeline where something is sitting there saying it knows what a human is supposed to look like. And so it really all depends on what the cameras are able to see at any given point in time, and what their best guess is of being able to reconstruct the geometry of what they think was in the room at that time.And so you get glitches with it. There's some noise that is inherent to this process, which is related to just the technical limitations of what you can do with cameras of a certain resolution, with the processing time that's available. But I think that there's actually something really interesting about that glitchyness to me, because it has a character that reminds me more of film grain than it does.For example, a lack of skill on the part of an artist who might otherwise be hand sculpting these details, right? You know, it's not something that feels like someone did a bad job, but there's a noise to it that reminds you of the artifice that's inherent to the capture. And I think, especially when you're thinking about volumetric capture of human performance, especially in the case where there will be no further opportunities to capture that individual performer, there's a sense that this is something that is of its time.This represents the desire to try and capture something while the opportunity existed. And if you wait for things to be perfect, you'll never end up with anything.0:18:46 - 0:19:01
Paul Duldig
And also, when you think about from when I was looking at it, the difficulty, the glitch, perhaps between telling the difference between someone's fingers and the keys is actually quite a beautiful metaphor for a pianist in any way. Right? Because they, they're at one with the instrument.0:19:02 - 0:20:08
Yoyo Munk
Yeah. I mean, getting the hands right for Kagami, that was a lot of work. And, ultimately, capturing hands in photogrammetry is already a reasonably difficult thing just because hands are complicated, there's a lot of geometry involved. It's really easy for one finger to get in the way of the other, and then suddenly you're not really sure what the shape of it's supposed to be.And I think that watching the hands move on the keyboard there, in many cases it's pretty close. And in some cases you'll just sort of see something drift a little further away than you would if it was actually a human playing the piano at the time. And I think that's a reminder that this is a imperfect capture, but that doesn't reduce the significance of its potential meaning and its mattering.You know, the idea that photorealism, or the idea of simulacrum as being an inherently required component for the attachment of human emotion on the part of a participant to an experience that they're having, that's obviously false.0:20:09 - 0:20:13
Paul Duldig
So can I ask, how does this scientist become an artist?0:20:14 - 0:22:33
Yoyo Munk
Through a convoluted, meandering path filled with luck and happenstance and mostly just kind of falling into it backwards. Coming from science, especially like working inside of biological spaces where you're often dealing with datasets that are inherently very high dimensional, there's a lot of noise inside of your measurements, and you spend a lot of time really thinking about how to take this noisy dataset and extract and narrative from it.It's very challenging to to try and think of like hypotheses that are really easily tested because like the space of problems that exist or areas of interest within biological sciences that are really, truly amenable to being able to ask a simple yes no hypothesis question is necessarily a subset of all the problem spaces that exist, and that doesn't make those other problem spaces any less compelling or or interesting to explore.It just means that they're less tractable from within a explicit hypothesis testing framework. And data visualisation is a very important tool in terms of being able to examine these high dimensional datasets and try to look at things from a bunch of different ways and sort of trying to see how we can find patterns within them.And this idea of playing around with signal and noise and extracting emergent properties from datasets that are inherently complex, I mean, these still feed through into the kinds of things that interest made within my artistic practice. I like in my own work to not have control over everything that is happening in the room. The behaviours of the pieces that I'm presenting are inherently kind of a collaboration between whatever ideas I put into it, and how an audience responds to it within the space itself.And that idea of having datasets emerging from and basically collecting data, representing that data through visualisation and trying to distill those down into a narrative, there’s not such a huge difference.0:22:33 - 0:22:36
Paul Duldig
I think that is an artistic process, essentially.0:22:36 - 0:23:11
Paula Bray
So libraries have incredible datasets. And, you know, as part of Digital, it's a practice that you can really use those datasets to look at, you know, a bird's eye view of a collection, or you can look at different lenses of a collection. And we've taken you on a tour of the library, State Library of Victoriathis morning. We've had a little chat about collection and participation with audiences. Can you give us a little insight into how you could apply your science art practice at a library?0:23:11 - 0:23:14
Yoyo Munk
I mean, it's a big question. I mean, you know, it is a...0:23:15 - 0:23:18
Paula Bray
You've got two minutes. [laughs]0:23:18 - 0:26:41
Yoyo Munk
Obviously State Library Victoria is among the most impressive examples of this in the world, but in any library of considerable size, I think that there's a sense of humility that comes from seeing this collection of immense amounts of human effort and, and immense amounts of knowledge painstakingly gathered over time and collected in these rooms where the sheer enormity of everything that has gone into not only understanding the contents of what's been kept here, but also the immense effort that goes into distributing that and keeping it and making it accessible so that that information can continue to live on andthat people can find it again. You could spend many, many lifetimes trying to absorb the information that exists here. And this is just the stuff that we've kept, right? I mean, for all of the knowledge that is out there, when we think about what we have kept inside of a library, this is a living document of the collective attention and memory of how our culture has moved over time, the things that we have chosen to archive and the things that we have decided are important at any given point in time.Is this moving, evolving target. One of the things that I like to play most with, in terms of my own artistic practice, is playing around with spatial and temporal scales. We are all human beings and there are time scales and spatial scales that are comfortable in terms of what we can realistically deal with on an individual basis.And I think that one of the reasons why libraries can feel like have that initial sense of overwhelm is that you are coming into an experiential contact with a hierarchical level of human knowledge and behaviour that is well beyond your individual capacity to absorb, right? This is a collection of knowledge that would be physically impossible for you to internalise as an individual.And that's not the point of it. And so when I think about what can be done inside of a digital space, I think about the fact that for all that we know that libraries are a living, moving archive, that constantly... these are dynamic collections of knowledge, there is a popular conception of use [unintelligible] and the stacks are this very kind of still space, there's a real sort of sense of permanence and the idea that everything is organised and that things move very slowly here.And so one of the things I think would be interesting to play with in terms of how we use could use digital technologies to play with the dataset of a library of knowledge would be to try and think about ways of representing how those collections and archives have moved between areas of interest over time, and think about ways that we can create sculptures from that data that impress a sense of the dynamism that exists within these collections, but simply isn't observable on the timescale with which we're used to observing things as individual humans.0:26:41 - 0:26:42
Paula Bray
Beautifully said.0:26:43 - 0:27:49
Paul Duldig
Beautifully said. And also makes me think that one of the things that really weighs on our minds – certainly weighs on my mind – is this is a collection of active collecting over 170 years, but obviously the knowledge that's being collected goes back thousands of years, many thousands of years in some cases. And so we are custodians for just such a short time of this incredible archive, and it's very hard to be able to take a third or fourth person view of that, because you're in the middle of it and you think, well, in years to come, somebody will look back and say, either what on earth were they doing?Or that was pretty good. And so but but how do you conceptualise that in real time? And, you know, there's potentially ways of doing that through being more intentional about what you're collecting. Practices are what your biases are, how that's playing out, how that might map or not map against what people are actually looking for. I mean, it should be a tool of beyond an art piece to actually really understand what on earth we're doing and whether actually doing a good job or not.0:27:49 - 0:29:07
Yoyo Munk
I mean, the last part about like, what people are looking for, I mean, I think is also a really interesting component of it because libraries exist. This is almost like a conversational exchange of knowledge, right? If I go into a natural history museum, assuming that I'm going through the public exhibition, this is a curated space where someone who has gone through this painstaking process of putting artefacts out for exhibition.But oftentimes when I go to a library, I have an idea of what I'm looking for, but I have this conversation with the librarian, with the search system like this isn't actually sociable process in terms of how I'm introduced to the information that I'm looking for. And one of the nice things about this is that oftentimes what I think I'm looking for is not actually what I want, right?But that idea that the way in which we are collectively interfacing with libraries, the kinds of information that we are seeking from it, this is the active process through which culture is expressing what it means to be important in any given point in time. This is a reflection of the values that we hold collectively.And given time. We'll see. We see how those values are shaped over the course of history.