A Simple Tool for Remote Real-Time Dance Interaction in Virtual Spaces, Or “Dancing in the Metaverse”
Dan Strutt*
Abstract
This article is a reflection upon practice-led research, undertaken during the pandemic, that addressed emergent forms of interactive, immersive, and virtual performance. Through experimental workshops we developed a network framework for the streaming of motion captured dance data, such that dancers could explore embodied co-presence within virtual spaces. I allow the dancers’ voices to directly address this research, describing their own feelings about the potential for remote dance creation. They describe how these practices sit adjacent to, and in many ways are antagonistic to, professional disciplined modes of dance work. We then ask, can remote motion-captured dance be elevated to being a truly poetic mode of creative performance, or is it merely a tool for a kind of loosely directed dance play?
Keywords: motion capture, virtual, telematic, streaming, dance, pandemic
During the COVID-19 pandemic, we have seen an acceleration and proliferation of performance practices that use digital cameras, social platforms and motion capture technologies to capture, document, stream and share content online and in shared virtual environments. Though in many ways a forced shift to online practices due to closed theatres and shuttered studios, these practices responded not only to the pandemic but also to the existence of a panoply of emerging capture technologies (beyond standard video conferencing platforms) that were poised and ready to be mainstreamed, or at least embraced, by performance practitioners.
While the hacking of the Microsoft Kinect over ten years ago first established this kind of accessible and affordable mode of performance capture (and its digital representation), we are currently seeing many more convergent technologies being used in interesting ways; from inertial motion sensors (IMUs), volumetric capture, webXR and games engines, to Virtual Reality (VR) “metaverse” type spaces such as Mozilla Hubs, VRChat and RecRoom. The real perceivable shift over the last two years, then, is how many artists and academics are now beginning to see the true potential that lies within novel syntheses of existing technologies for emergent forms of interactive, immersive, and virtual performance.
As a consequence, over the last two years we have witnessed the creation of significant and highly visible new digital and virtual work by established companies (such as the Royal Shakespeare Company) alongside an abundance of smaller creative companies, research centres and mainstream film and media festivals engaging with virtual production practices to suggest innovative applications of tech. Digital performance is now coming to be seen as less of a niche practice, or poor substitute for theatrical liveness, and it feels as if we are enjoying a Zeitgeist moment for virtual performance work.
This short article is a report of and reflection upon some practice-led research-in-progress, undertaken during the pandemic, that addresses these issues. The project is a collaboration between academics and postgraduates from Goldsmiths, University of London and LASALLE College of the Arts in Singapore, with dancers from Akram Khan Company and Alexander Whitley Dance Company, and with motion-capture industry leaders Noitom, Target3D and Studio Aszyk. Through a series of experimental workshops held between various lockdowns, and with very limited physical co-presence, we have developed, tested and honed a network framework and software tool for the streaming of motion capture data from multiple locations into virtual environments, such that dancers can explore virtual co-presence and dance communication by remotely connecting to avatar representations of themselves in digital spaces.
As humanities research, it has always been a priority of ours to critically assess the affordances and limitations of this kind of human-technology interface in performance. This critical stance led us first to reflect upon and position our own work within a genealogy of telematic and distributed performance (from Doug Engelbart’s The Mother of All Demos in 1968 to the ongoing live telematic video work of Paul Sermon, such as Telematic Quarantine from 2020)(Strutt et al.) and to theoretical approaches in the aesthetics of dance and its relative virtuality (for instance, from Suzanne Langer and Marc Boucher) (Strutt and Cisneros). With these processes and concepts in mind we have hosted a set of conversations, livestream Q&As and interviews with collaborating dancers to discuss how, from the performer’s perspective, these tools pose themselves as extension of, or limitation to, their habitual practice.[1]
Throughout this two-year process, extending from before the current pandemic through to the end of 2021, our research design strategy and development has been intimately entwined with both formal and informal professional dancer feedback. This article provides an opportunity to transcribe these feedbacks, bringing them into a single document and discourse, and explicitly connecting a set of theoretical preoccupations with the expressed feelings and material experience of the practitioners.
What we have found is that there is clear potential for dancers to use these kinds of tools both expressively and as a meaningfully felt form of choreographic communication and interaction. Our dance participants told us that there can be a simple and natural mode of attuned engagement with the avatar—leading, at best, to “out of body” experiences, however fleeting they may be. As one dancer remarks, avatar bodies as an extension of one’s own are quickly adapted to, such that “different kind of qualities and rigidities or softness of the figures can be embodied quite naturally” (Whitley). However, they also noted that this attuned embodiment with the avatar in virtual spaces comes with specific observed limitations and obstacles to be overcome. Amongst the noted stumbling blocks, that in a way break suture, are: the unpredictability of visual effects, an often fragmented body image, problems with over-ambitious embodiment, and the need to simplify expression into elemental movement practices.[2]
For these reasons, discussed below in more detail, for professional dancers there is a described need to relinquish the type of habitual control into which they have been highly-trained—leading to a relation of complex embodiment with an avatar form which is of one’s own body but does not always simply yield to our control. We then discovered, ironically perhaps, that it was actually in focusing on the other virtual dancing body within the communicative moment, and breaking the fixation on the sense of control of their own avatar, that dancers could actually feel like they “become” the figure on screen. Thus, the very act of live-streaming dance data in real time allows the dancer to transcend the sometimes awkward “puppeteering” type embodiment of digital dance and to fully immerse themselves in the virtual scene.
In what follows, I will briefly explain the research design and how it has evolved through its iterations and evolutions. I will then give a brief overview of some of the theoretical questions that have framed our learning process. Finally, I will allow the dancers’ voices to directly address both of these aspects, organising their feedback into thematic sections about the experiences of embodiment and connection. This written article is supplemented and supported by audiovisual documentation of the conversations and live-streams from which much of the dancers’ input has been transcribed, along with demonstrations of our “Goldsmiths Mocap Streamer” framework at work.[3]
Project Design
July 2020: This project first developed in collaboration with Goldsmith’s partner institution LASALLE in Singapore, with developer Andreas Schlegel and dancer Melissa Quek (fig. 1). Three creative technologists (Neal Coghlan, Clemence Debaig and Friendred Peng) used TouchDesigner’s visual programming language to generate interactive graphic visualisation of the motion capture data, spawning separately at each end of the stream and projected onto large screens for both dancers. The visualisations produced here were largely 2D and focussed on elemental and bold expressions of movement, velocity and of the connection between dancers’ joint positions. The time difference from London to Singapore, and, at an early moment of the pandemic, the rather strict restrictions on the use of institutional space, made this test session productive but time limited.
With Arts and Humanities Research Council (AHRC) funding received towards the end of 2020 within the “Ideas that address Covid19” research funding stream, we were able to build a completely new and more robust framework for streaming mocap data with U.K.-based developer Paper Plane Software. Here, we created a multiuser streamer software as a docker image hosted on a cloud server. This mocap streamer software tool (now dubbed Goldsmiths Mocap Streamer) acts as an aggregator hub, receiving multiple streams of BVH (Biovision Hierarchy) data from senders and then forwarding that data to any number of receivers within a node-like structure. The user interface allowed us to drag and connect these user nodes, with directional arrows to designate sender or receiver roles.
April/May 2021: For an experimental workshop at the end of a period of lockdown in the U.K., we stepped back into the University campus, working from the Sonic Immersive Media Lab (SIML) at Goldsmiths with three dancers—a single dancer in the room with us and two connected remotely from their own homes. For a live-streamed showcase event on May 7, we inset the video stream of the dancers over the Unity generated image to make it clear how the system is working (fig. 2). For this, our first showcase performance event, our creative technologists crafted five scenarios that were framed by theoretical concepts of virtuality, narrative and illusionism, and with a set of interaction designs that guided several semi-improvisational movement exercises.
November 2021: As the first AHRC research funding period drew to a close, we worked towards a final iteration of the research design, in which we offered a hybrid in-person and online performance called Dancing into the Metaverse. This showcase offered immersive multi-screen projection in the live performance space, alongside the same YouTube video-streaming format we had used previously. Working with one dancer in New York and two in London (fig. 3), we used an even simpler version of the streaming tool, removing the manual node-based web user-interface and simply allocating sender and receiver roles to all those connected. The content of this performance, delivered on site at Goldsmiths for the U.K. Being Human festival,[6] directly evolved from the previous iteration, with fewer, longer scenes, and with a focus on heightened interaction design. The showcase was followed the next day by a set of workshops with both dancers and technologists, where each attendee had the chance to suit up and dance with others participants some 3,500 miles away across the Atlantic.
Alongside these showcase performances, we also developed an application of the streamer software for use with virtual reality. In collaboration with Alexander Whitley Dance Company, we presented a short live performance in VR at VRHAM! Virtual Reality Arts Festival in Hamburg (in June 2021). This same performance was then delivered at the BFI London Film Festival within their virtual arts programme The Expanse, as proof-of-concept of our ability to bring both dancers and audiences from anywhere in the world into a tangible shared virtual performance space (fig. 4).
Going forward into 2022, we are looking forward to the next phases of our research. The first of these is a second AHRC funded project to build a truly international network for real-time mocap dance streaming practices through a global virtual artist residency programme. The second project is designed towards working with Machine Learning tools to enhance accessibility for non-normative and disabled bodies towards dance performance in virtual spaces. Finally, we continue to develop collaborative works in VR with project partners INVR Spaces (the makers of the BFI London film festival “Expanse”) and Alexander Whitley Dance Company.[7]
Theoretical Questions
Our team first approached this research area by framing it within a history of “telematic,” “networked” or “distributed” performance (expressions coined and explored by Roy Ascott, Alvaro Barbosa and Johannes Birringer respectively); descriptions of the various technological and interface configurations by which geographically distant artists are brought into virtual proximity to produce performative events.[8] In this work, we framed the history of telematic dance performance through both the inherent difficulties of telecommunication—delay, latency, noise and glitch—whilst also struggling against a type of technical gimmickery without aesthetic integrity (where the simple trick of the tele-presence of two remote bodies is seen to be impressive enough in itself).
We addressed this issue by focussing not on the capacities of technologies to produce, or re-produce, sensations of liveness but, rather, on how they might extend the dancer’s body, augment corporeal agency and complicate senses of embodiment between physical and virtual phenomenological spaces. We looked to sensations of virtual touch, proximity and intimacy not as substitutes for the physical correlates of the same but as realms of embodied experience to be explored in terms of engaged attention, meaningful connection, affective attunement and emotional communication. Applying this in practice-led sessions we returned again and again to the grounded and pragmatic insight of our main dance partner Mavin Khoo and his red-line which stated that technology should work for the dancer to produce feelings of genuine connection through narrative and affective frameworks, and not the other way around.
Having the technology work for the dancer, however, is not as simple as it might seem. The technological frameworks, sometimes clunky patchwork assemblages of multiple softwares, hardwares, drivers and interfaces, along with a development pipeline that usually places asset design long before choreographic creation, directly contravenes many basic tenets of the disciplinary practices of dance creation, and indeed of dance perception and cognition. Where the dancer is customarily in control of the expression of meaning through the boundaries of their own bodies, here control—and thus authorship—is partly yielded to the creation of an avatar that works for them but is not them. In this sense, they are working with a somewhat autonomous intermediary, often leading to unpredictable outputs. In a recent article in Body and Society, Kriss Ravetto-Biagioli aptly refers to
the interplay between humans and machines as a complex set of iterations—an indeterminate play of modulation and differentiation producing unexpected affects and relations—rather than discrete and manipulatable events.
4
The telematic dance performance is thus not only geographically “distributed” but also distributed through different layers of software and hardware, material and imagined spaces and objects, awareness of hypothetical or invisible audiences, and through different expressions of the body which range from the figurative (humanoid) to the abstract. This is inherently experimental, challenging at first, and it requires both a suspension of disbelief and a playful disposition.
These dance works offer a mode of bodily proprioception that cannot be said to conform to habitual modes of perception whereby the spectator or user recognizes and empathizes with figures in movement. They offer us a different mode of play within the digital domain, one that does not reify the figure of the dancer, rather it challenges those bodily regimes that strive for some model (figure or index) of bodily accuracy. (Ravetto-Biagioli)
26
Of course, these challenges can be frustrating to the performer, at first an alienating or cold experience. It can feel quite distant to the actual work of a dance professional and becomes more of a kind of “choreographic installation” (as described by Rubidge), or a series of rather mechanical exercises, rather than a flowing and natural attunement with one’s own dancing body and with those of the other dancers.
However, as I recently explored in a co-authored piece in Theatre and Performance Design with theorist and practitioner Rosa Cisneros, through an iterative process of applied and practice-led research, breakthrough moments of affective attunement with the virtual space and body can truly be achieved in ways that are both productive and revealing. In this article, we proposed that recent motion capture technologies and real-time graphics game engines have the ability to truly capture the nuances and vitalities of bodily gesture in ways that permit a naturalistic and flowing attunement with avatar expressions of self and others. This kinaesthetic attunement with a body in motion even extends, quite easily, beyond actual bodies to other quasi-humanoid forms, and, in fact, much farther—to graphic and geometric forms, animals, objects and instruments that offer mere abstraction of human-like motion. Dance movement recognition, though cognitively grounded in corporeal kinaesthetic and proprioceptive experience, can be felt through all manner of expressions of “movement vectors” in a palpable and deeply affective way (this is not a recent observation but one that has extended through the multimedia dance work of artists such as Merce Cunningham and Wayne MacGregor for many years) (see for instance Manning as well as Rutherford).
Ravetto-Biagioli rightly points out the ethical problems with this facility of identification with “captured” virtual bodies, describing a possible “identity tourism” or masquerade borne out of the observation that even if we can evidence empathy and attunement with an avatar figure, it does not mean that we always should (4). She seems to suggest that with motion capture, particularly in game formats, we often assume the avatar (human) form of some other real person’s captured body and movement. The implication is that motion captured bodies may be being inhabited, appropriated and used in ways that are not, and cannot be, consented to, and this bears true when we consider current lawsuits against the producers of the game Fortnite, in which well-known dance moves have been appropriated and commodified as in-game purchasable “emoves.”[9] Ravetto-Biagioli points instead to examples of motion capture art practices that in different ways mutate or disfigure, multiply or de-temporalise the captured motion of bodies, such that we experience the kind of critical estrangement that troubles any simple identification with the figure of the other. In these ways, for Ravetto-Biagioli, we cannot simply and easily “jump-into” the body of another; rather, we must critically negotiate with our level of immersion and affective attunement.
However, even a simple and natural mode of empathetic engagement with an avatar body takes a new tangential direction when we consider the influence of recent real-time interactive digital technologies. These systems are by necessity open-ended, non-linear and virtualising, rather than involved in the capture, documentation and archiving of motion (in such ways that might allow the kind of deepfake-like body-theft that Ravetto-Biagioli describes). They require data inputs to calculate changes, interactive triggers lead to dynamic dialogues between dancer and avatar, between avatar and avatar and between avatars, spaces and objects, in a constant state of becoming. While this does not necessarily totally override the ethical issues of masquerade and mimicry, what it does is direct the experience away from that interplay between our own virtual and actual bodies—what Ravetto-Biagioli calls “self-fashioning” (26)—and the focus becomes an open-ended and unpredictable conversational relationship between two or more people, or indeed between a human and an artificially intelligent system. We become caught in an intimate feedback loop into which we are playfully immersed in an act of co-creation, rather than simply consuming and using a “playable” image.
Ravetto-Biagioli actually considers one such example, Bill T. Jones’ work “Dancing with a Machine,” noting how “This project tries to make these bodily mappings an ‘extension’ of Jones body—an extension in time where Jones can reach some future audience or community of dancers by exceeding the limits of his actual body” (26). In this work, the dancers of Jones’ company perform a duet with a machine-learning system that tracks the motion of the dancers and uses that data to guide the words spoken by the dancers around their body. What Jones offers here is an explicitly open-ended process of becoming, rather than a system of fixed capture. This is evidenced by the fact that Jones’ work is not simply (only) a fixed and recorded performance but, rather, an open-source and interactive online “experiment” into which anyone can step via a web browser.[10] The AI system as real-time collaborative partner is, in a way, the work of art—not the visual documentation that accompanies it.
While Jones’ work is in essence a duet between a single dancer and a machine, our own mocap streaming practice is more human to human, with the machine systems acting as a (non-neutral) intermediary or collaborator. Our software Goldsmiths Mocap Streamer captures but does not hold motion data. Instead, it forms a constant stream from one person in one location to another in a different location and then uses that streamed data to transform, extend and augment that bodily movement to, in real-time, render expressions of presence, touch, weight and emotion through physical transformation of light, tone and form. It then asks for reciprocation; what extensions can my body throw out into space that interact with the forms of your body? We are drawn into an intimate, immersive attunement with the other body, not voyeuristically observing, nor inhabiting the body of the other, but instead a bonafide embodied communicative act with another dancing body. This does not seem to be a use-case that Ravetto-Biagioli considers in her ethical critical analysis of “playable” bodies.
Performer Feedback
In this section, it is my intention to prioritise the voices of the dancers who, as part of the research process, have become “entangled” with our own human-machine framework through various workshops and showcases, and who, through this, have influenced our research design. Though these words were collected at different times and through different iterations of the research design, I have tried to bring the dancers’ perspectives and feelings about their experiences into conversation with each other by grouping them into overarching themes of embodiment and connection.
The dancers speaking here are: Melissa Quek, head of the School of Dance and Theatre at LASALLE College of the Arts in Singapore; Mavin Khoo of Akram Khan Company; Alexander Whitley and Tia May Hockey of Alexander Whitley Dance Company. These artists come from very different backgrounds, and it was very interesting to get a diversity of responses that responded to each other, framing our research in constructively critical ways. Mavin here summarises what in fact came to feel like our guiding mantra:
Khoo: I personally come from a world that, to be frank, has initially been quite sceptical about anything that is based on technology because the work that we do is very much based on the lived body and the interpretation of the dancer in that moment. So, one of the things that I have been, I think in a good way, consistently questioning is, . . . how does it sit within the poetic nature of the kind of work that our company does? Is there a possibility for it to become a poetic collaborator?
Complex Embodiment
The first and perhaps most obvious theme that has come through our conversations is that of the relative sense of embodiment when one is dancing through an avatar representation of oneself. As noted above, this can be described through notions of kinaesthetic or affective attunement, where the avatar becomes a mirror of, or extension of, the dancer’s own body. There are clear obstacles to overcome to feel fully embodied within the avatar form and virtual space. As Melissa reflected well after our first workshop in July 2020, there was an initial feeling of distance that must be overcome, where the avatar body feels more like a puppet that is being controlled rather than a true extension of one’s own body:
Quek: Because your body’s not your body in a way. Like I’m not dancing my own body. I’m dancing the image of my body that’s being projected, so it feels a bit like being a puppeteer and you’re making the image of yourself dance with the image of someone else.
In a later iteration of the design in May 2021, Tia felt more connected to the avatar than Melissa, and embodied within it, but sometimes in a way that made her body feel incomplete, as if it were only a part of her that was extracted.
Tia: My experience is that the performer’s body does not feel as though it is extended by the technology but, rather, like a part of my body has been fragmented and warped. This fragmentation and warping effect seems [sic] to extract a specific part of my expressivity and reduce it to its simplest function
She went on to explain that to fully explore this fragmented sense of embodiment required a kind of hesitation and slowness.
Hockey: There’s also something in the subtlety of the movement that requires a little bit of hesitation in order to be accurate. Like I have to be a little slower, a little more sensitive to really understand what’s happening in that virtual space.
We can understand that there is a transformed relationship to the habitual expressivity of the dancer’s body that requires a fair amount of extra cognitive analysis and negotiation. It is not as natural or intuitive as holistically jumping-into or inhabiting the avatar body but, rather, a subtler process of simplifying the movement, learning what the avatar body can realistically express and then honing a sensitivity towards that.
Khoo: It’s still such an early phase of us, the dancer, really negotiating and trying to understand the existence of this. So, I think it has been extremely important actually to stay away from being too ambitious in terms of what we insert on dance terms, and to really go back to the basics and the simplicity of action gesture and the simplest movement first and foremost . . . to really go back to just a simple action of movement without engaging with our natural tendency to be virtuosic in a specific physical way.
In the scenarios that we showcased in May 2021, we thus tried to explore a variety of simple interactive exercises around embodiment, presence and touch, in scenarios that ranged from figurative to abstract, and from purely playable, open-ended interactions to clearer and more linear suggestions of character or narrative. Tia expressed a clear preference for the less figurative and narrative situations and expressed the feeling of being more confined and “fragmented” when having to assume a specific character or action.
Hockey: The more recognisably human avatars felt like a replication or mirror, the more abstract effects were closer to the feeling of being extended or influencing different approaches to improvised movement, and the more narrative scenarios felt fragmented and very focused in a particular intention and physicality.
This was also reflected in the opinion of Melissa, who seems to suggest that a level of abstraction in design allowed a more playful and free expressivity.
Quek: Other [scenarios] were fun and interesting to play with exactly because they were abstract. I felt like I was painting with my body in the space. With each of the sketches I would move differently because of what the sketch emphasises or allows to happen.
Alexander extends this sentiment and spoke specifically about the scene “bubblegum” (fig. 2) in which the avatars took a more abstract and “blobby” form.
Whitley: Avatar figures change how you move, yes, definitely, in really interesting ways. With each of these scenes—the different figures—it takes you a while to work them out. I mean there’s a few things going on; there’s the avatar figure itself—with these blobby figures in the scene that the three of us did together—that obviously have added appendages and things that aren’t naturally part of your body, so there’s some interesting working out of how it relates to you, which naturally changes how you’re thinking through your body. . . .
For Alexander, though, this was not a difficult or fraught process of analytic negotiation, but one that came quite naturally or “automatically.”
Whitley: I guess the different kind of qualities and rigidities or softness of the figures can be embodied quite naturally. . . . When you recognize enough of yourself in those figures you automatically start to take on the characteristics or qualities of those avatar figures.
Experience of Connection
Our research priority was to bring remote dancers into shared virtual and interactive spaces through a mocap data streaming software, and thus the immediate sense of embodiment within one’s own avatar was always entwined with, or secondary to, the goal of creating a sense of co-presence and of virtual proximity, touch and tactility. Mavin described the challenges he faced in trying to feel connected in this way, distracted by his own avatar image.
Khoo: I have to confess it was hard for me to focus in with the fact that I was dancing with someone because I was so much in a habit still of looking at myself to see what would happen. A genuine investment into dancing a duet with someone else was for me quite challenging.
However, Melissa felt that certain graphic expressions of connection (here describing the simple lines drawn between data points seen in the image of her in fig. 1) allowed her to break this focus on self and become more invested in the interaction.
Quek: One of the things that I thought about was how some of the sketches allowed me to focus better on the other dancer. Some created a good visual representation of the spatial relationship which made it easier to focus on my interaction with the other body—they quite literally draw the link, and that was helpful.
We discussed, with Mavin in particular, that too much abstraction actually limited a feeling of connection. While interesting to explore in terms of movement as a “painting with the body,” these purely abstract representations of velocity and joint position without a concretely “humanoid” figure visible made it very difficult to understand the presence or intention of other dancers. We learned that for dance communicative acts there needs to be both a recognisable body visually represented and a strategy of graphic interaction design that tangibly connects those bodies in a three-dimensional space.
Tia then explained how a set of shared semi-structured exercises as “shared intentions,” grounded within the interaction design of the specific graphic effects, and with some practice, could allow for a deeper investment in the virtual body/space such that she might “let the reality disappear.”
Hockey: I think it is easy for this experience to be cold and alienating, and, so far, it has often felt this way. However, with shared intentions, heightened awareness, conscious effort and sensitivity I believe a connection can be formed and enhanced. There were definitely some out-of-body experiences and moments where I felt the potential of the virtual connection with the other performers. It required me to fully engage with my imagination and to let the reality around me disappear in order to let my attention be fully present in the virtual space.
Interestingly, she compared this to a kind of play with an (non-human) animal.
Hockey: When I had a shared intention to play in improvisation with another performer, I felt moments of connection with our virtual characters and through to my being. It’s kind of like the connection you make with an animal and you both know that you’re watching each other, there’s a level of sensitivity required by both parties to listen, anticipate, predict and respond. It was hard to maintain this sensitivity, but I did experience flashes of it.
As Alexander explained, the framework in fact demands this kind of undetermined or improvisational interaction, guided by sensitivity, exploration and anticipation, otherwise there is no benefit to it being offered live and in real-time.
Whitley: Choreographing something would defeat a lot of the purpose of this and what makes it really special as a tool is its liveness and the fact that we can be interacting and responsive to each other in the moment. You know if you were just performing something choreographed you may as well do that independently and put it together into a games engine.
Overall Impressions
What our team have learned from the practice-led research sessions is that there is a clear potential for remote dance creation and telematic performance within virtual spaces. However, the practices that we have developed and experimented with sit adjacent to, and in many ways are antagonistic to, professional choreographic and disciplined modes of dance work. The obvious question then perhaps is: as a platform for remote virtual interaction, can it ever be elevated to being a truly poetic mode of creative performance, or is it merely a tool for a kind of loosely directed dance play?
Responding to this, Alexander suggested that in many ways a task-based play approach is not unlike how one would intuitively develop and create choreography in the dance studio.
Whitley: I think the fact that we could be as responsive to each other and work around the kinds of tasks that we would normally work with in a studio setting shows the success of this tool—that it does really enable us to interact in three-dimensional space with other people in a way that feels quite natural and intuitive to us as dance practitioners. That’s really the strength and benefit of this tool as far as I see it.
Tia explained that these types of practices could also inform new pathways for the dancer in ways that might inform and alter their habitual modes of expression.
Hockey: I’d say that the motion capture technology has an ability to encourage or influence the performer’s body through a combination of its limitations and freedoms which can inform or alter the dancer’s expressivity. The performer may discover new sensations, pathways or approaches through this tool, however, this has some clear boundaries and limitations.
The boundaries that Tia described have been evident throughout the research and can be summarised as the need to relinquish some level of “authorial” control over one’s own body and to always negotiate and carefully attune with the machine “intermediary.” Melissa, however, noted that with improving real-time technologies and more flexible, intuitive and responsive software and hardware (including the more immersive AR and VR interfaces), the scope for open collaborative communication and dialogue is shifting towards the greater control and empowerment of the dance artist.
Quek: If the dancer better understands the parameters of what the [digital] artist is doing, then their movements are more directly designing the sketches that result and the visualisation can better support what the dancer is doing. For instance, they can discuss what movement is being explored, what the data points are, how they relate to what is being visualised and what the effect of manipulating those variables could be. This could really improve the quality of this kind of collaboration.
This point was corroborated by Mavin:
Khoo: From what we have seen already there is a potential to actually engage with this tool from a very perspective where the dancers have the potential to really create with a sense of ownership.
While there are clearly many unanswered questions lingering about the real scope of this as a professional tool for creation and expression of dance, emergent and evolving technological assemblages permit a more natural and intuitive, poetic collaborative relationships between artists, machines and the creative technologists that control them. Alexander, looking to the future, described how it seems almost imperative that we will have to consider, and continue to develop, such collaborative practices as a realistic and viable alternative to current practices.
Whitley: I think that the ability to connect remotely potentially addresses some issue around the roles and responsibilities of companies touring all over the world and the climate impact of that. I think it’s worth pointing out that there’s really significant answers that this kind of technology can present to some of those challenges.
Conclusion
It can be concluded that dancers find that these tools are interesting and productive to use and that their use of them can merge in interesting ways with existing practices. It is not yet clear, however, that this can ever substitute for dance as a fully embodied practice, or indeed that work created or rehearsed in this way can ever directly translate to physical or theatrical performance in an interchangeable way. This is because, we have discovered, movement choice is different in each sphere, with a complex or fragmented sense of embodiment with digital avatars, with a perceived need for more minimal or elemental movement expression and with practices more adapted to semi-structured playful or exploratory improvisational exercises rather than tightly choreographed work. Does this mean then that it must remain partitioned off from mainstream dance as a niche digital practice and mode of performance (a “choreographic installation” for Rubidge)? Or can we evolve an understanding of how it might be used to develop alternate modes of dance interaction and communication that can, in time, inform and influence the core practices of dance? Can it become a worthwhile poetic collaborator in the creation of both digital and analogue work?
The dancers we have worked with are divided on this issue, based on their own personal experience of the system, and with understandable hesitation given the experimental and often makeshift nature of our research workshops throughout the pandemic. There is, though, little doubt that all of our dancers do seem to appreciate that this is a future-orientated practice in a process of refinement, which will be seen more and more within the industry going forward. What is required then in the next phase of research is not only the technical development and refinement of the interface, but, perhaps more importantly, a finessing of the modes of creative collaboration. We must respond to the expressed need for dancers to spend more time with the digital artists and technologists, enhancing mutual understanding, confluent creative practices and shared aesthetic goals.
Endnotes
[1] The first of these took place in August 2020, then May 2021, with additional material drawn from emailed interviews.
[2] Suture is a film theory concept describing the bridging or closing of the gap between perceived reality and fantasy in the mind of the spectator (see, for instance, Kaja Silverman). In modern film theory, this was attributed to a set of technical conventions to allow us to directly identify with the “look” of the camera upon the action, as if it was our own. In this digital context, it would instead be the way in which the motion capture technology allows us to directly identify with the avatar body to feel, kinaesthetically, as if it were our own.
[3] See our website, mocapstreamer.live, for this supplemental material.
[4] See here for our self-produced video about this test session; and here, for a video made by our technical motion capture partner Target3D.
[5] This live-streamed event can be viewed on our website mocapstreamer.live.
[7] See mocapstreamer.live for details.
[8] This work was first done in the chapter “The Future of Dance: New Telematic Technology for the Remote Creation and Performance of Choreographic Work” (Strutt et al.)
[9] Filed by Fresh Prince of Bel-Air star Alfonso Ribeiro, Russell “Backpack Kid” Horning, unnamed Fortnite fan known as “Orange Shirt Kid” and rappers Terrence “2 Milly” Ferguson and James “BlocBoyJB” Baker in 2019. All claimed that Epic had unlawfully used dance moves they had invented as Fortnite emotes.
Bibliography
Ascott, Roy. Telematic Embrace. U of California P, 2003.
Barbosa, Álvaro. “Displaced Soundscapes: A Survey of Network Systems for Music and Sonic Art Creation.” Leonardo Music Journal, vol. 13, 2003, pp. 53–59.
Birringer, Johannes. Performance, Technology, & Science. New York: PAJ, 2008.
Boucher, Marc. “Kinetic Synaesthesia: Experiencing Dance in Multimedia Scenographies.” Contemporary Aesthetics, vol. 2, no. 13, 2004.
Hockey, Tia May. Personal Interview (2021).
Khoo, Mavin. Personal Interview (2021), and Q&A conversation in showcase ‘Virtual Touch, Virtually Dancing’ (May 2021), mocapstreamer.live.
Langer, Susanne. Feeling and Form: A Theory of Art. Charles Scribner’s Sons, 1953.
Manning, Erin. Relationscapes: Movement, Art, Philosophy, MIT Press, 2012.
Quek, Melissa. Personal Interview (2021).
Ravetto-Biagioli, Kriss. “Dancing with and within the Digital Domain.” Body and Society, vol. 27, no. 2, 2021, pp. 3–31.
Rubidge, Sarah. “Digital Technology in Choreography: Issues and Implications.” 17th Annual Symposium of the Dance Society of Korea, 6 Nov. 200, Seoul, Korea. Symposium Presentation.
Rutherford, Anne. “Cinema and Embodied Affect.” Senses of Cinema, no. 25, 2003.
Silverman, Kaja. The Subject of Semiotics. Oxford UP, 1983.
Strutt, Dan, and Rosa Cisneros. “Virtual Relationships: The Dancer and the Avatar.” Theatre and Performance Design, vol. 7, no. 1–2, 2021, pp. 61–81.
Strutt, Dan, Andreas Schlegel, Neal Coghlan, Clemence Debaig, and Youhong “Friendred” Peng. “New Telematic Technology for the Remote Creation and Performance of Choreographic Work.” The Future of Media, edited by Joanna Zylinska, Goldsmiths Press, 2022. pp. 197–222.
Whitley, Alexander. Q&A conversation in showcase ‘Virtual Touch, Virtually Dancing’ (May 2021) mocapstreamer.live.
*Dan Strutt is a lecturer in the department of Media, Communications and Cultural Studies at Goldsmiths, University of London, where he teaches media theory and analysis alongside creative economy. His book The Digital Image and Reality: Affect, Metaphysics and Post-Cinema was published with Amsterdam UP in 2019. With previous research work on CREATe and the AHRC Creative Economy Programme, he now collaborates with dance and digital artists in AHRC-funded research in the area of motion-capture data streaming and telematic performance.
Copyright © 2022 Dan Strutt
Critical Stages/Scènes critiques e-ISSN: 2409-7411
This work is licensed under the
Creative Commons Attribution International License CC BY-NC-ND 4.0.
