Visiting the Virtual:
Performance Practice in the Virtual Artworks of Rob Hamilton and Christof Ressi

Abstract When moving in the context of computer games, contemporary multimedia music and classical instrumental performance practice, we find ourselves confronted with the different creative aesthetics and goals, agents, and peer groups that are pertinent to each specific context. This multiplicity not only affects the works created, but also has an impact on the audience’s expectations and perceptions. It also touches the work of the performer on several levels. In “Visiting the Virtual,” I will explore the manner in which elements of computer games incorporated into audiovisual artwork in live concert situations achieve an aesthetic effect and performative impact on both the work of the performer and the audience’s perception.

Keywords Game-related audiovisual composition, virtual reality, agency in performance, computer games, contemporary multimedia music.

The artistic research project GAPPP – Gamified Audiovisual Performance and Performance Practice, which offered me the framework for this investigation, was funded by the Austrian Science Fund as a multiannual project in artistic research (PEEK AR 364-G24). The project was conceived and led by the composer and audiovisual artist Marko Ciciliani. It ran from 2016 to 2020 and was based at the Institute of Electronic Music and Acoustics (IEM) of the University of Music and Performing Arts Graz. With GAPPP we set out to develop a thorough understanding of the potential of game-based elements in audiovisual works in the context of contemporary Western art music. New audiovisual works formed the point of departure for our research. These works were created for GAPPP, each addressing specific research questions we wanted to investigate. While one main focus lay on musical applications, all artistic projects that emerged were conceived as audiovisual art, and accordingly the visual and interactive components of the artworks have been as important for our investigation as the musical elements.

This article is part of the monograph Ludified (edited by Marko Ciciliani, Barbara Lüneburg and Andreas Pirchner, released in January 2021), and is published here with the kind permission of the publisher The Green Box, Berlin.

  Introduction and Research Questions

When moving in the context of computer games, contemporary multimedia music and classical instrumental performance practice as we do in GAPPP, we find ourselves confronted with the different creative aesthetics and goals, agents, and peer groups that are pertinent to each specific context. This multiplicity not only affects the works created, but also has an impact on the audience’s expectations and perceptions. It also touches the work of the performer on several levels.

In “Visiting the Virtual,” I will explore the manner in which elements of computer games incorporated into audiovisual artwork in live concert situations achieve an aesthetic effect and performative impact on both the work of the performer and the audience’s perception. Furthermore, I will explore how the use of virtual reality influences this situation. I will do so by looking in detail at two different case studies, Movement II of Trois Machins de la Grâce Aimante (2018), a virtual reality string quartet by Rob Hamilton, and Terrain Studies (2019), an audiovisual com-position for solo instrumentalist and virtual reality system by Christof Ressi. Both works are located within a ‘virtual reality’ that unfolds behind the VR glasses of the per-former(s) and a ‘physical reality’ that is shared with the audience and bears traces of the virtual. My investigation of these cases will therefore address the representational spaces in which the works unfold, namely the physical space of the concert performance and the virtual space in which large parts of the performance take place.

Following on from two earlier papers of mine that were concerned with other GAPPP works (Lüneburg 2018a and b),2 I will touch on questions such as: how does the space of possibility afforded to the performer through the game system, software design, and interfaces influence the players’ ludic and performative involvement and range of expression in live concert situations? How do the differ-ent spaces influence the performers’ tasks and the audi-ence’s perception? How prominently does the audience experience associations with games and how is this feel-ing established in both audience and performers?

  Analytical Tool and Theoretical Background:
  Ciciliani’s Polar Diagram and the “Space of Possibility”

My analysis of Hamilton’s and Ressi’s works seeks to identify creative agencies in the system design and interactivity that give the performer a sense of meaningfulness. According to Salen and Zimmermann, “Meaningful play occurs when the relationship between actions and outcomes in a game are both discernible and integrated into the larger context of the game” (Salen and Zimmermann 2004: 34). In GAPPP works, meaningful play is pertinent to the particular space of possibility they offer. This “space of possibility” is the space of all possible actions and meanings that can emerge during the performance of the ludified artwork. This concept ties together meaning, design systems, and interactivity, and directly influences the performers’ creative strategies and goals, musical objectives, and capacity to share their artistic experiences with the audience.

In order to include both an internal and external perspective in my research, I draw data from my own artistic observations as a performer of both works and from the collaboration with composers and coplayers. Additionally, I code and analyze interviews with composers, perform-ers and audience focus groups. Audio and video docu-mentation from GAPPP’s work labs and lab concerts com-plete the data collection and offer a further source of information when analyzing the artworks and situations in question. I apply Marko Ciciliani’s polar diagram for the analysis of gamified audiovisual works (see figure 1) as a tool with which to analyze the two case studies. Since the polar diagram is explained in detail in the chapter “A Polar Diagram for the Analysis of Gamified Audiovisual Works,” I will offer only a brief summary of its structure and my use of it here.

The polar diagram was created as a visualization tool that provides an overview over key criteria relevant for ludified works. The diagram is divided into an upper half, which describes the work with regard to aspects of its composition, and a lower half, which reveals the perfor-mance-related perspective. Five continuous axes cross both halves, called interface axis, determinism axis, agen-cy axis, presence axis, and ludus axis. Each depicts two parameters that refer either to the same or to closely related phenomena.

Figure 1. Polar diagram developed by Marko Ciciliani to describe ludified audiovisual compositions and the criteria that determine their space of possibility.

The interface axis visualizes how much expertise the performer requires to use the interface of an artwork and its responsiveness to the performer’s actions. The determinism axis relates to the technological system’s algorithmic configuration. It depicts the degree of determinism given through the compositional system and how this translates to the performer. On the performance side it indicates whether the underlying ludified interactive system requires spontaneous actions and decisions, or whether it allows the performer to plan the interpretation of all the performance’s details. The third axis represents the musical and performative agency of the performer in comparison to the agency of the system and the creative leeway granted to the player. The presence axis illustrates how input data is mapped in the artistic result. On the performance side of the diagram, the term “liveness” signifies how “aesthetically meaningful differences in the input sound are mapped to aesthetically meaningful differences in the output sound” (Croft 2007: 61). In other words, this parameter indicates to which degree the system is experienced as an extension of the instrument the performer uses. It includes not only the sonic representation but also potential visual translations of the performers’ actions. On the diagram’s compositional half, the complementary term “input data mapping” refers to the quantity of data extracted from the performers’ playing that are used to trigger sonic or visual events.

The ludus axis demonstrates where the emphasis lies within the gamified system: on either ludus or paidia on the composition half, or on the dominance of rules (gamedriven versus performance-driven rules of play) on the performance half. I adopt the definitions of “ludus” and “paidia” by game researchers Salen and Zimmerman (2004), who define these categories as addressing “a structural understanding of games, a continuum of relationships between structure and play. If for example play edged closer to the ludus end of the spectrum the rules become tighter and more influential. Located on the other end of the spectrum, paidia-based play eschews rigid formal structures in exchange for more freewheeling play” (Salen and Zimmerman 2004: 309).

In the diagram, the values for each parameter are marked graphically. By connecting these marks, a shape results that allows for an intuitive comparison of graphs based on different works. In the following analyses, I concentrate on the agencies and interfaces given to the performers and their influence on the players’ ludic and performative involvement. I investigate the way in which the systems developed further the range of expression in live concert situations and the conditions for the players’ inner involvement or noninvolvement. A special focus will lie on comments by members of the audience, which serve as an external perspective (for a detailed description of the demographics of the audience, cf. “lab concert” in the Glossary in this volume).

Having defined the underlying terms and briefly explained the polar diagram, I will now investigate the following two case studies: Movement II of Trois Machins de la Grâce Aimante by Rob Hamilton and Christof Ressi’s Terrain Studies. I analyze Ressi’s work from the perspective of my own performance at the world premiere. For this book, however, Terrain Studies has been documented in the performance of clarinet player Szilárd Benes, Christof’s longtime collaborative artistic partner.

  Case study 1: Rob Hamilton’s Trois Machins de la Grâce Aimante

Rob Hamilton’s Trois Machins de la Grâce Aimante is an artwork for a virtual reality string quartet and a computer system. Its second movement was premiered in 2018 at the Cube of the IEM Graz by Barbara Lüneburg (1st violin), Osman Eyublu (2nd violin), Francesca Piccioni (viola), Myriam García Fidalgo (violoncello), and Rob Hamilton (VR system), the same instrumentalists who are featured in the documentation on the included USB stick. In this work, the musicians do not perform on their acoustic instruments, but on the “Coretet”, which is a family of virtual musical instruments. The Coretet’s concept and design stems directly from traditional string instruments (violin, viola, and violoncello). However, the Coretet itself is entirely virtual and played via a VR system. The players sit back to back and their faces are covered by VR goggles. Through the goggles, they see their virtual instruments, which they control through wireless controllers and by emulating the typical movements of playing physical string instruments.

Figure 2. Performers on the virtual instrument “Coretet”. © ndbewegtbild

In Rob Hamilton’s own words, in Trois Machins de la Grâce Aimante he explores the ‘virtual’ versus the ‘real’ through the performers’ musical and performative handling of the digital instrument Coretet.

I built a virtual model that allows the performers to use the same performative gesture that they learned on acoustic instruments to push data into this virtual model. So, with the goal of making this feel and sound and act like a traditional bowed instrument, we can then depart and do anything … The focus here is on the instrument and on the performers. (Hamilton 2018)

Visually, the string quartet is represented in three forms: as a projected highly stylized image of its avatars on the main-stage screen, through the bodily performance of the players on stage, and as an image of their individual instrument projected behind the players’ VR glasses. Hamilton uses physical modelling to computationally simulate the reaction of a string to bow pressure and velocity, as well as to being played by different parts of a bow.

However this sound, while it sounds like a string, it does not sound like a violin or a cello. There is an incredible rich harmonic spectrum that is abrasive and distorted and has all kinds of bizarre non-linear characteristics that are hard to control, and yet, these performers are doing a great job controlling it. So working with this constraint, forced me to really focus on making the instrument behave in a traditional sense, even though it is already an extension of that tradition. (Hamilton 2018)

The instrumentalists explore the Coretet in the given virtual environment and play with it without any predetermined game-related obligations, such as fulfilling specified objectives or following goals. Their performance space is the world that unfolds behind their VR goggles, which is designed as a white space that is additionally projected onto the main-stage screen for the audience: “My intention was that this was a neutral, blank space … There is no semantic content in that environment, no context that I am trying to convey as an artist at all” (Hamilton 2018). However, the performers’ field of vision is filled first and foremost by their own virtual instrument. If they want to see their fellow musicians, who are depicted as abstract avatars hovering in the virtual void, they have to deliberately turn towards them.

The relationship of how the performers are sitting in physical space versus in virtual space has become very interesting in that in physical space the performers are seated in a circle facing away one from another, and yet, the representations in virtual space are exactly the opposite: facing one another in traditional string quartet formation. (Hamilton 2018)

The musicians’ sensory experience is split. Their instruments are virtual without a physical body, which means that the musicians are literally playing the air. Their visual experience is determined by the virtual view behind their VR glasses, their sonic experience is based in the physical reality transmitted through the loudspeakers of the performance space, and their haptic experience is reduced to pushing the buttons of their controllers and moving their arms through the air while emulating string player movements. They have to perform in two worlds: the virtual world in which they create the music, and the physical world in which the audience subsequently experiences it.

In musical terms, the performers are guided by a graphical score on the basis of which they improvise and explore the Coretet. The score subdivides Movement II3 of Trois Machins de la Grâce Aimante into six parts while indicating the formal build-up and musical atmospheres. Hamilton allocates a different harmonic principle to each section of the movement by quantizing the fingerboard of the virtual instrument according to different pitch-class settings. He traverses the range by increasingly diminishing the pitch intervals in each section from major triads to a pentatonic scale, a whole-tone scale, a chromatic scale, and finally a nonquantized scale that offers free choice of pitch. The musicians’ actions are focused upon exploring both the music and the virtual instrument. From the audience’s perspective, the projection of the virtual environment onto the main screen and stage props such as individual computers for each player, wearables such as VR glasses and game controllers, and even the playful staging of the quartet back to back effectively set the stage in terms of the game experience.

Next I will apply the polar diagram to the second movement of Trois Machins de la Grâce Aimante. Before I do so, I would like to highlight that I am talking about an experience with a virtual instrument that at that time existed as a prototype only and had never been tested before in a concert situation. Difficulties and limitations in handling the instrument were to be expected. And although we will hear criticism from the performers, they also saw great potential in the instrument and enjoyed performing Trois Machins de la Grâce Aimante. Hamilton has developed a highly sophisticated setting with great artistic promise of which we will hopefully hear and see more in the future.

Applying the Polar Diagram to Trois Machins de la Grâce Aimante

Hamilton describes his composition as feeding from traditional instrumental performance practice, and the instrument Coretet as meant to be played by expert string players: “From the time of its inception, Trois Machins de la Grâce Aimante was intended to exist as a composition firmly descended from traditional ensemble instrumental performance practices with a goal of exploring how virtual implementations of musical instruments could leverage learned expert behaviors of highly skilled musicians” (cf. the chapter “Composing (and Designing) Trois Machins de la Grâce Aimante”). Since Hamilton implements compositional rules rather than game-related rules in Trois Machins de la Grâce Aimante, I will apply the upper half of the polar diagram to purely compositional rather than game-related parameters in my following analysis.

Figure 3. Polar diagram for Rob Hamilton’s Trois Machins de la Grâce Aimante .

Indeed, the polar diagram indicates the use of an interface, the Coretet, that on the compositional side requires its user to be highly specialized, while the performance side of the diagram indicates only a medium degree of responsiveness. This is partly due to the lack of the haptic and tactile feedback that the performers are used to from their physical instruments, which is missing from the virtual instrument. In the focus group interview that we conducted with the members of the quartet, it became clear that the performers struggled for instance with the controllers that replace their normal instrument and the bow. Interview partner 2 (IP2) recounts:

There was the bow issue: I find the game controller that we use instead of a bow extremely sensitive, in fact too sensitive. It responds to many of my movements that would not make any difference with a physical bow, which causes problems. Of course you can turn every problem into something expressive, but here I think this is very difficult. (Quartet 2018, IP2)

In the performers’ opinion, some of the instrumental registers are not yet well adjusted and they feel that the control they have of timbral sound color needs to be improved.

I usually play in high registers and avoid the low, because the lower frequencies come with a lot of noise. Sometimes it sounds more like a ‘swoosh’ than an ordinary tone. I know, there is the concept implemented in the instrument that you can control how much noise you have in the sound, but this I think can still be improved a bit. (Quartet 2018, IP3)

However, the performers see potential for development, both concerning the instrument itself and in their learning of how to handle it:

We are experiencing limitations. Some features do not react as fast as we expect them to do from being used to a normal instrument. And for some things we would perhaps need more training to be able to play more exactly and have more control. We are still struggling a bit with how the machine is reacting, but many features do work. (Quartet 2018, IP2)

We cannot control the pressure of the bow [to influence the dynamics], but we realized that we can replace it with bow speed. Even if there are lots of things that don’t function like with a real instrument, being an instrumentalist and knowing of all the possibilities that you have helps to replace some of the techniques in a way that works on the virtual instrument. (Quartet 2018, IP1)

The determinism axis shows a balance between the system’s degree of determinism and the performers’ resulting feeling of predictability. Both lie within a medium range. The compositional setup—that is, the graphical score—made for relatively strong determinism by providing playing instructions with regard to form, the choice of harmonic scales, and the musical atmospheres of the six sections of Movement II. On the other hand, like all graphical scores it left scope for artistic creativity. Accordingly, the musicians felt that planning was both possible and required, but that creative spontaneity was desired as well and showed in the score’s improvisational aspect. Moreover, the determinism factor is strongly influenced by the virtual interface, which is at the center of the composition, and its to some extent unpredictable behavior. The musicians’ musical planning was thus counterbalanced by the spontaneity they needed to react to the fragile behavior and peculiarities of the Coretet.

On the agency axis, we find that the compositional and system agency ranks considerably higher than the personal agency perceived by the performers. They found for instance that the representational space of the virtual reality comes with a limited field of vision, making it difficult to see their coperformers in the virtual realm and even more difficult to interpret their actions based on the stylized avatars, of which they only see unconnected elements: the shape of a head, a hovering instrument pointing in the wrong direction, a disconnected bow, and a randomly flying hand (see figure 4). They felt that this reduced their musical agency and capacity to make chamber music in the physical concert space. “We want to play together but it is almost impossible to play exactly in unison, which would not be a problem with a normal instrument” (Quartet 2018, IP2).

Figure 4. The highly stylized avatars appear behind their VR glasses and are visible to the audience on the main performance screen. © ndbewegtbild)

The Coretet the musicians performed on existed only in a prototype version that was still in development at the time of their playing. Its (yet) existing limitations had an impact on their musical agency. The performers worked with a number of harmonic schemes set by the composer that were mapped onto the Coretet meaning that they could not freely shape the harmony or melodies or break the given tonal system in their improvisation. Thus, the system in its prototype version allowed for relatively little creative leeway in terms of making chamber music, forming a phrase or a melody, influencing harmonic events, producing concise rhythms, and controlling the bow articulation.

If I try to play staccato really loudly, I pass right through the instrument, since there is no physical body, no strings, that would stop me, and I get a weird doubling of notes. Instead of a regular “dong-dong-dong” I play “da-kaa-da-kaa-da-kaa” because the bow connects twice with the violin on either side of its body. (Quartet 2018, IP4)

On the other hand, there were features of the Coretet the performers really enjoyed, and which they used along with their improvisational expertise to stretch the constraints. For example, they switched between the extreme types of Coretet instruments, using this feature to extend their supply of pitches to higher or lower regions and to expand the expressiveness of their gestures.

We have the option to change the instrument. I switch between violin, viola or cello, which enlarges my scope of pitches. The cello I play in a really weird way, because its body is not fixed to the floor as with a normal cello. Instead I hold it like a huge oversized violin. It stretches in front of me. I use this to extend my performative movements. I want the audience to enjoy these enormous gestures, when I try to reach the end of the fingerboard of my giant cello-violin. That’s fun. (Quartet 2018, IP4)

Figure 5. The Coretet family (from left to right: orb, violin, viola, violoncello, double bass, bow) © Rob Hamilton

The performative agency in Movement II of Trois Machins de la Grâce Aimante is strongly influenced by the playability of the interface, the virtual instrument Coretet, and the constraints given by the harmonic system and the graphical score. At the same time, however, the work builds on the performers’ improvisational skills and creativeness with regard to dynamics, textures, atmospheres, and choice of instrument (the musicians were able to use the five instruments the Coretet offers to enlarge their pitch range, traversing all registers from bass to soprano). As performers, we are thus dealing with two different principles, “ludic play,” which represents the rule-bound composition, and “paidic play,” represented by free-form improvisational play, with ludic play constituting the predominant feature of this work.

Representational Spaces

In order to learn more about how Trois Machins de la Grâce Aimante engages both audience and performers and what kind of connotations the work invoked, I will now turn my attention to the two representational spaces it offers: the physical space of the concert performance, and the virtual space in which the performers move, which is only partially shared with the audience via the main performance screen. Additionally, I will trace how gadgets and aesthetics of the gaming world are tied into the composition and setup.

Hamilton states that “‘gaming’ influences the interaction design, control schema and overall aesthetics of the Coretet and the composition itself” (cf. the chapter “Composing (and Designing)Trois Machins de la Grâce Aimante”). The performers interact with their virtual instrument via the Oculus game controllers they hold in their left and right hands. They manipulate the controllers’ buttons to put their fingers on the virtual finger board, to touch the strings with their virtual bow and latch onto them, or simply to change the orientation of their virtual instrument. Through their goggles, they see their instrument and the avatars of their coplayers from a first-person perspective, whereas the audience is presented with the four avatars as a stylized quartet on the main-stage screen. Images from the virtual reality spill over into the physical concert performance space.

The gestures of the musicians in the air are captured by sensors that track the location of the Oculus game controllers. The data are mapped onto virtual left-hand movements, such as shifting position, glissando, vibrato, and trills, or typical right-hand movements, such as simulating bow strokes at different speeds and in different articulations. The performers’ gestures in the air are movements in the physical environment that manifest in both representational spaces: in the virtual space they make it possible to play the Coretet, while in the physical space they add to the audience’s visual experience by enlarging and emphasizing performative expressiveness. Moreover, the gestures affect the sonic experience in the physical concert space.

It could be argued that the aesthetics of the Oculus Rift glasses, the laptops that are placed in front of each performer capturing the player’s individual perspective on their virtual instrument, the game controllers, and the unreal way of playing transport performers and audience into the realm of computer games. However, the performers did not perceive their task as game-like, but rather like performing a piece of art:

For me the environment is game-like, but we treated the music like serious music, in my opinion we did not include game elements. (Quartet 2018, IP3)

The engine of this piece is built with the software “Unreal Engine 4,” a really famous company which makes a lot of crazy games. So, yes, the work uses game details but it was treated in the prism of art for the purpose of making the music of the composition. (Quartet 2018, IP1)

Nevertheless, they observed that the game-like environment allowed them to break free of their usual serious inner attitude towards classical music, stating that a childlike curiosity popped up that encouraged them to be explorative. They even proposed that audience members could be offered the opportunity to try the system for themselves, perhaps in the context of a music outreach project. However, in their opinion the Coretet is a system built for experts rather than for nonprofessionals.

Since we as string players know the possibilities of our own physical instrument, we know how to replace movements that maybe do not work [on the Coretet] … We know how to make it appear real. We bring along the knowledge of our physical instrument, the structure, movements and techniques. (Quartet 2018, IP1)

The musicians wear VR goggles that visually separate them from their audience, and they have no face-to-face contact. Nevertheless they assumed that the audience could read them through the tools they used (the game controllers and VR goggles), their body language, and the music itself: “I think we can connect, because there are the tools, there is the body present, there is the sound” (Quartet 2018, IP2).

How was the Piece Perceived by the Audience?

According to the results of the audience questionnaire, approximately two thirds of the audience members reacted positively to the piece. Many associated it with games evoked by the VR gadgets and the players’ body movements that reminded them of using Wii controllers. However, they struggled with not having any direct part in the VR experience. Instead, they felt like observers unable to share the fun and almost resented it:

I often asked myself, what do they see that I don’t? How do they communicate? Do they communicate at all? … I hoped all the time that I would see something. Some kind of communication which would let me share the excitement, like “Hey, do something together!” … Almost all the time I felt that I was just an observer, since I had no chance to intervene. I even thought, I would love to just go there and tell them: this thing, why don’t you just take it and just play for yourself, please, don’t make such of show of it! (Focusgroup_1 2018, IP1)4

I often drive racing simulators with VR glasses myself and that is fun as long as you drive yourself … I don’t think it’s as fun as an observer as someone who really does it. (Focusgroup_1 2018, IP2)5

However, there were moments when audience members were able to relate to the piece through their prior knowledge of similar situations. In our audience questionnaire, many mentioned associations with games such as Guitar Hero, Nintendo Wii sports or music games, and interactive games they were familiar with from their PlayStation. This notion was picked up in our focus group interview: “This reminds me of the Wii controller which you have to fidget around with, and you also do it with a group of people” (Focusgroup_1 2018, IP2). 6 Others felt involved when watching the musicians’ chamber-music interactions:

I felt drawn in when I had the feeling now the four are synched, they make music together. Most of the time it felt like each of them was moving independently, but in these moments, mostly when they played fast, it gave me the impression that they harmonized, those were the moments when I felt more involved … For me this was not an effect of gaming, but actually, I found it highly aesthetic and participative. (Focusgroup_2 2018, IP4)7

Space of Possibility

To us performers, Trois Machins de la Grâce Aimantewas a concert piece first and foremost. Musically, we found controlling our musical interface, the Coretet, challenging, but we saw potential in it, even if we did not entirely agree with the composer’s claim that the virtual instrument could be used similarly to an ordinary physical string instrument. We felt that the space of possibility the Coretet offered was—at least in its prototype version—still rather restricted, and we were aiming for something that was (yet) out of reach. Instead of mainly being steered towards the past tradition of playing string instruments, we would have equally liked to explore the instrument’s potential especially with regard to its future-oriented aspects. Therefore, I would like to raise a set of follow-up questions: what does the Coretet offer that the physical string instruments cannot? How can we use these qualities in our performance? Is there an alternative to the way game aesthetics in Trois Machins de la Grâce Aimante could be employed to emphasize a ludified approach or involve the audience? Would any of this increase the agency for the players and offer a richer space of possibility that in consequence would result in more meaningful playing?

What does the Coretet offer that classical string instruments cannot? The Coretet encompasses the whole family of string instruments, from the double bass to the violin, and includes the so-called “orb,” which is meant to be used as a percussion instrument (see figure 5). Moreover, although the Coretet’s sound is based on the physical model of a string, it does not sound like any ordinary string instrument. Hamilton describes its sound as abrasive, distorted in certain ranges, and noisy. What does this mean for the musical agency of the performers? The wide instrumental range—orb, violin, viola, violoncello and double bass—that is accessible simply by pressing a button on the right-hand controller gives the player extra scope in terms of sound color, pitch range and gestural play. The Coretet’s extraordinary sound world should be explored for its uniqueness, instead of modelling the playing on the sound world of the classical string instruments, as it was suggested we do. There are different shades of noise and sound colors to be revealed, its abrasiveness and distortedness to be explored musically, and the percussive qualities of the orb to be discovered.

The virtual instrument Coretet lacks the physicality of an instrument. Instead of holding the body of their instrument, gripping a fingerboard, and bowing strings, the musicians move their arms through the air while emulating playing a physical string instrument. This emphasizes gestural play. As a theatrical tool or means of expression, it adds to the performative agency of the musicians and their feeling of meaningfulness. To give an example: by switching through the different sizes of the Coretet, performers can vary the size of their gestures in quick sequence, changing from the tiniest of movements when playing the smallest instrument of the family, the violin, to huge ones when playing the orb or the double bass. The latter is especially effective if the bass is held like a giant violin instead of playing it in its ordinary upright position. The gestures are stretched out and exaggerated, the musician might even have to get up from their chair, adding a dramatic element to a possibly wild musical atmosphere (see figure 6).

Figure 6. The image shows the gestural play of one performer who is standing up to reach the end of the fingerboard of her oversized Coretet. © ndbewegtbild

The performers can use gestural play as a method of conveying musical meaning. At the premiere, for instance, the musicians had to lock in a regular steady pulse led by the cellist. The musician played discreetly, the rhythm was almost unnoticeable in her movements. By clearly articulating and demarcating her gestures when playing, she could have taken the audience along with her by using her physical movements to illustrate how sound is produced in the invisible virtual space and then sonically projected into the physical concert space, thus making it easier for the public to understand the music, the technological setup and the musical scene.

Another aspect of Trois Machins de la Grâce Aimante that merits further exploration is its cinematographic quality. The audience are able to watch five screens in total: the large main screen at the back of the stage as well as four laptops, each assigned to one performer, sitting on a table vis-à-vis the instrumentalists. These individual computer screens show what each performer sees in the virtual space, thus offering the audience a glimpse of the performers’ first-person perspective on their instruments, which otherwise is hidden behind the VR glasses. Not everybody understood the connection:

Since there were five screens in total, I could not really feel my way into the piece, because I always jumped from one screen to next and wondered, what’s the purpose of the screen? What’s happening here? … I would have found it more invasive if there had been one large screen with four split screens on which I see every screen in one. (Focusgroup_2 2018, IP3)8

Accordingly, how can a connection be established between the individual actions of each performer, what they see, and the way it is staged? Larger gestures could convey musical meaning to the audience. If these gestures were coupled with a video projection of what the acting individual performer was seeing at that moment in the virtual space, their visual experience could be shared with everybody on the main-stage screen. Depending on the musical development, individual screens that show the instrumentalist’s perspective could occasionally be brought to the foreground, alternating with the virtual, white performance space in which the four players are depicted as avatars. This would be particularly effective in combination with solo passages or distinct musical patterns that can clearly be assigned to one individual player.

In this way, everything would come together in the physical world of the performance space, the sonic result, the performers with their physical gestures on stage, and their visual representation on the main-stage screen. Going to an extreme, the role of director of visuals who features players’ projections on the main screen could even be assigned to individual audience members in a participative setting. Ultimately, this could result in a dramaturgical setting in which musical structure, gestural play, and choice of the video screened add to the cinematographic experience for players and audience alike.

I hypothesize that by shifting the conceptional compositional focus from imitating traditional classical instruments to the unique features of the Coretet and the special staging of Trois Machins de la Grâce Aimante, the musicians would gain more musical and performative agency and meaningfulness in their explorative creative actions; additionally, the audience would gain more direct access to the virtual world of the Coretet, which again might counterbalance the feeling of being left out and emotional distance that some of our visitors described.

In addition, one performer suggested that the Coretet could be employed as an educative tool in audience outreach in participative settings in which audience members are allowed to play and explore the instrument. Another performer thought the instrument had even further artistic and social potential:

Let’s say we look a bit further into the future. In my opinion, in maybe fifteen or twenty years from now it will be possible for people from different parts of the planet to play in some kind of international virtual reality online jam session, similar to the way that they connect with each other to play games all night today. An improvisation session for which they can just sit in their home, put on their VR glasses, get connected with each other and make music. Just like that. That would be really great. Well, I think this project is one of the greatest examples of how we can use technology and virtual reality for us, for the future and especially for art. I am sure it will have really big resonance in the future and will serve a good purpose: for art and for the next generation. (Quartet 2018, IP1)

In the following case study, Terrain Studies by Christof Ressi, I investigate how the virtual space the performer moves in and the actual physical performance space are connected and shared between audience and performer and in which way the interplay of both influences the experience of the work.

  Case study 2: Christof Ressi’s Terrain Study

Christof Ressi’s Terrain Study is an audiovisual composition for solo instrumentalist and virtual reality system conceptualized as a ludified 3D performance environment that takes place in both virtual and physical space. “The situation is that of a classic first-person game. There is the camera that is linked to the player’s field of vision. It is … a classic VR game where one moves in a [virtual] environment” (Ressi 2018). 9 The instrumentalist wears a VR headset (model HTC-VIVE) and experiences the performance exclusively in the virtual 3D environment, whereas the audience follows the projection of the player’s VR perspective on a screen placed on the main stage, where the performer is also physically present in the center of attention.

The performer’s tasks encompass the creation of the entire sound world—there are no preproduced or synthesized sounds—and the visual manipulation of the initial landscape, the visual and sonic interaction with objects that appear, the so-called ‘orbs’ (see figure 7), and last but not least the enactment of a performative scene on the physical stage. The landscape and the musical and bodily interaction with it become the performer’s creative interface.

Actually, this landscape is an instrument. You have these orbs which you can interact with and which record your sounds … One’s own behavior in the space influences the sound, in fact more and more so in the course of the game. And then slowly, this transfers to the landscape itself … Whatever you do—soundwise or with your head [the head movement is tracked via the VIVE]—is translated into parameters that eventually deform the environment more and more. (Ressi 2018)10

Figure 7. Screenshot of Terrain Studies showing the landscape with hovering orbs. © ndbewegtbild

In the beginning, the instrumentalist tentatively explores the virtual landscape by improvising. Soon she encounters flying orbs, which invite playful interaction. Whenever the performer touches one of the orbs, they briefly turn red, which signifies that they are now recording the sound produced by the performer. Having finished recording, they start to pulsate and play back the recorded sound while circling around the player. Soon the orbs float away into the landscape, thereby modifying the sonic characteristics of the looping sound. The orbs’ own movement in the landscape and the (computer-tracked) movement of the performer’s head both influence the sound processing. If the orbs descend towards the ground, the pitch of the recorded sample becomes gradually lower, and if they rise into the sky, the pitch rises as well. Likewise, if the performer raises her head, a delay sets in that is spread across the ambisonic loudspeaker system. Over the course of the piece, the performer is increasingly amplified and blends into the overall level of dynamics, which according to the composer subtly suggests that she is becoming more and more part of the artificial environment. Gradually the performer’s presence in the landscape and influence on the environment grows. This is not limited to the sonic world, but has visual consequences as well.

The first thing that happens is that an orb floats towards me. If I approach the orb closely enough, it turns red, which is the moment when I can play for it. So what I’m playing right now is recorded and stays in the orb. It then starts to pulsate and reproduces my sound in a loop, which is immediately modified. It is pitched up or down, mostly sounds a bit rougher, not anymore as I recorded it myself. And the pulse, the delay, can also change. This can sometimes be faster, sometimes slower. The volume also changes depending on how far the orb is away from me. And soon a second orb will be added. And a third. And a fourth. I can always fill them with different sounds. One can also stop an orb by approaching it again until it turns red, and if you then don’t play, you fill it with silence, and the sound stops. (Lüneburg 2018c)

In the virtual world, the player stands on a square island that starts out as an artificial plain surrounded by water. As soon as the performer plays her first note, the topology immediately begins to react to her sonic input and body movements, slowly changing into a hilly island, turning into a volcano, a desert or a mountain formation. The audience is able to follow the performer’s movements in the virtual 3D landscapes on the main-stage screen. Behind her VR glasses she needs to climb steep hills, cross ravines and interact with the floating orbs. In the real world, however, the instrumentalist is plainly standing on the floor of the stage within a marked-off square and the audience sees her move within these boundaries.

First it gets hilly, then there are very, very high mountain peaks and extremely deep gorges, and as a performer I have to go through this landscape again and again. This is sometimes uncomfortable because I have the physical feeling that I am really in a real landscape, that is, one step further and I fall down the slope. Of course, this is not the case in reality, because you always move on the level stage. (Lüneburg 2018c)

The audience takes the role of backseat gamers gathered around the ‘skillful player,’ who negotiates the difficult terrain, interacts with the obstacles thrown in her path, and at the same time composes the music.

The audience always sees the landscape from my perspective. When I turn my head while playing, the stage screen also shows the section of the area I’m looking at right now. (Lüneburg 2018c)

Moreover, via the spatial arrangement of the audio rendition, performer and audience share what game theoretician Gordon Calleja calls a “a sense of habitation within the game environment” (Calleja 2011: 75). What the performer hears within her VR headset and the angle from where a sound source reaches the performer in the virtual world is mapped onto the ambisonic loudspeaker system of the performance space. In this way, the performer has real-time control over the audience’s listening situation. However, this mapping results in a different sonic experience for the audience than for the player, since the instrumentalist is continuously changing her position in the landscape, whereas the audience is sitting completely still, facing in a single direction only. For the instrumentalist the sounds remain stable and naturally related to her movements and her viewing direction, whereas each of her head movements causes a dramatic shift in the spatial placement of sounds for the audience.

Over the course of the piece, the sounds and landscape undergo more and more electronic processing, which again is controlled by the movements of the player. Ressi explains that the virtual world and the player become one:

In principle, my position in the space and my head orientation allow me to shape the sound … which then slowly changes to the landscape itself, which is then deformed more and more. ... In the end there is almost a situation where the space no longer exists without the violin. There is a point where the violin needs to maintain this space entirely. And when she plays certain things, the world is completely broken. It is actually a shift from ‘I am a part in this room’ to ‘I am the room.’ (Ressi 2018)11

Since the performer’s body movements are tracked and influence the visual and sonic scene in multiple ways, her physical presence on stage becomes as important as her music playing. Respondents to the audience questionnaire mention that the choreography of movements and the performer’s stage presence has a clear impact on their experience of the music and the cinematography on the video screen. They mentioned, for instance, that game and performance melt into one. Ideally the player uses this feature to create a situation on the actual performance stage that gesturally underlines what is happening in the virtual scene and is at the same time expressive on the physical stage as well.

Applying the Polar Diagram to Terrain Studies

The interface axis for Terrain Studies reflects the use of an interface that requires a lot of expertise while also being very responsive. The entire virtual reality system, including the 3D performance environment and the physical environment of the concert space, serves as an integral interface. The proficiency needed encompasses instrumental improvisational skills and an understanding of the action-reaction response offered by the interactive elements of the landscape and its actants, such as the orbs. In addition, conceptual thinking is required to perform the work, such as foresight into the formal and cinematographic development of the piece. The work furthermore requires physical presence and an understanding of how gestural play conveys meaning to the audience.

Once performers have familiarized themselves with Terrain Studies, the interactive system makes relatively predictable planning possible. However, on both sides of the determinism axis the mark is only in the middle, since in performance situations the composer himself likes to come up with surprising new interactive features that call for spontaneous reactions and flexibility from the performer: sudden teleporting or self-destruction mechanisms that have not been introduced in the rehearsal or novel actants in the form of further interactive objects will create an unfamiliar, challenging situation for the instrumentalist.

Figure 8. Polar diagram for Christof Ressi’sTerrain Studies.

The mark for the “performer vs. system agency” axis lies in the middle, corresponding to the fact that the performer is presented with an interactive environment and tasks that are both carefully conceptualized. At the same time, the performer’s input is required to shape the scenery in both musical and visual terms. The performer’s stage presence is important and conducive for the audience’s experience. Every action of the performer is meaningful and its short-term and long-term results can be observed by both player and audience. The performer literally shapes the landscape, the music, and the cinematographic scene and constructs the overall form of the performance. Accordingly, the creative leeway given to the performer is marked as very high.

The marks on the presence axis show that audio input and movement data mapping are essential for Terrain Studies. The feeling of “liveness” on the performer’s side corresponds with this and is also perceived by the audience. One interviewee remarked: “The reaction between the musician and the visual effects and sound was really fast and just good” (Focusgroup_2 2018, IP2). The ludus axis indicates a minimum of rules, which were indeed rather general explanations about how the player’s improvisation and movements would influence the system and what to do first: catch a flying orb and fill it with a sound, then watch what happens and react to it. Terrain Studies supports paidic play, and exploration and improvisation stand at the foreground of the player’s experience

How was the Piece Received by the Audience?

More than 90% of the respondents to the questionnaire stated that the game aspect of the performance of Terrain Studies was of central importance to them. This was brought up by the interviewees in the focus group as well: “With the virtual there is of course already an association of play through the performer” (Focusgroup_1 2018, IP4).12 However, they did not feel like backseat players: “Through these balls that come towards you ... you have the feeling that you are more in the middle of it. But really as a backseat player? No, not at all!” (Focusgroup_1 2018, IP3).13 According to the questionnaire, many felt drawn in by the coalescence of the visual, performative, and sonic elements triggered through the performer’s interaction with the orbs and landscape in the virtual environment.

With Ressi I thought it came across pretty well through the visual and auditory level that these orbs were activated. Because they were so distorted and then sounded themselves. It was actually immediately clear that she would go there and record something, then the orb would replay it. (Focusgroup_2 2018, IP3)14

I think that was an essential principle, ... that you could follow the eyes of the performer. To put myself in her position definitely works better this way! (Focusgroup_2 2018, IP1)15

Sounds and visuals typical of computer games, cultural references to well-known game types (such as sports games using Wii controllers), and the use of interfaces that are typical for games (such as the VR headsets or game controllers) underline the impression of a game-related artwork. One interviewee observed: “I mostly found a connection between game, art, and music on the visual level. For me, the game level is always the visual one. Be it the image or the performative physical level” (Focusgroup_1 2018, IP4).17

The game aspect of the works was able to facilitate spectators’ access to the aesthetics of the artwork or at least help them to cope with the work’s ‘weirdness’: “I think that it won’t work without these gaming elements because it is actually weird to listen to for most people. It is very, very weird music” (Focusgroup_1 2018, IP2).18 Nevertheless, the game aspect does not guarantee the inner involvement of the audience: “It is very important for me to say that I was only there as an observer and that it would probably be quite different if I did it myself” (Focusgroup_1 2018, IP1).19 Last but not least, a lack of observable and comprehensible game mechanics, such as rules and goals, can throw the audience off: “For me, the gaming character was gone immediately when I saw no goal of the whole. Why do you do that now? What is the motivation behind it?” (Focusgroup_1 2018, IP1).20

To conclude, I investigate which kind of agency and meaningfulness is created through the fusion of virtual reality and physical reality in the representation of an artwork. How does the given setup aesthetically and performatively impact the work of the performer and the audience’s perception? What influence does it have on the player’s ludic and performative involvement? What could a space of possibility look like that furthers the performer’s creative range of expression in both the physical space of a concert performance and the virtual space beyond the VR glasses?

Although according to the questionnaire most members of the audience experienced Terrain Studies as positive, they also described it as dystopian, nightmarish or overwhelming, and for some it bordered on ‘too much’: “Occasionally I looked away because there were a lot of impressions at once, and that was almost a bit overwhelming, actually” (Focusgroup_1 2018, IP3).16

  Conclusion

Conditions for Involvement

The performers measured Hamilton’s virtual instrument Coretet and the agency it provided them against their traditional physical instrument and missed the reliability and flexibility offered by the latter. They felt they needed this reliability and flexibility for their musical playing and that their musical and performative decision making was at least partially impeded by it. I claim that concentrating instead on the instrument’s cinematographic and gestural as well as performative potential could considerably increase both the performers’ agency and the audience’s involvement. In my opinion, this would allow for a different kind of performative and musical action and creative decision making “that let performers transmit the artistic experience on a social and artistic-communicative level in cognitive, sensor-motoric or emotional ways” (Lüneburg 2018a).

Ressi aims at building a system in which performer and audience and the virtual realm and physical concert space are tightly intertwined. Actions and gestures in the physical space have clearly discernable artistic results in the virtual space. The concert space’s technological and stage setup translates these actions and results to the audience in an understandable way, thereby strengthening involvement and the feeling of meaningfulness for both player and audience members. This could be observed in the performer and focus group interviews and the audience questionnaires from the lab concerts.

Tacit Knowledge

Tacit knowledge helps audience members to relate to a performer’s ludic or performative actions and to identify with the person or the situation. Moreover, it furthers the inner involvement of the observing audience members. As the sociologist Stephen Turner suggests, tacit knowledge of an activity allows for better understanding and communication: “Some activity, inference, or communicative act depends on both the user and the recipient possessing some inferential element or mechanism which allows them to understand, anticipate, co-operate, or co-ordinate with another” (Turner 2014, p. 155). In Hamilton’s work, audience members were able to relate to the VR experience and use of game controllers mostly via their personal playing of sports, dance, or racing games and musical games such as Guitar Hero. Ressi’s work seemed to speak to them through their knowledge of and experience with open-world games and first-person games.

Hamilton states that “an understanding of the interconnected nature of any human-occupied virtual space with our own physical world” has informed his artistic endeavour with the Coretet (cf. the chapter “Composing (and Designing) Trois Machins de la Grâce Aimante”). The same notion has guided my investigation of both Hamilton’s and Ressi’s works. I found two fundamentally different approaches to dealing with virtual reality. Hamilton chose to convert a physical instrument into the virtual idea of one. He focused on the development of an instrument for a new environment that emulates a traditional one and takes it beyond pre-conceived instrumental boundaries. Ressi, by contrast, invented a virtual environment that is manipulated and shaped with the help of an ‘ordinary’ instrument and the player’s body movements in the physical world. What unites both works is that actions in one reality have an impact on the other. However, the two composers pursued different artistic goals.

  References

Calleja, Gordon. (2011). In-Game From Immersion to Incorporation. Cambridge, MA.

Croft, John. (2007). “Theses on liveness.” Organised Sounds 12/ 1: 59–66.

Focusgroup_1. (2018). Interview by Marko Ciciliani and Barbara Lüneburg with the participants of focus group 1 in the framework of the GAPPP lab concert at the Institut für Elektronische Musik und Akustik’s Cube in Graz (Austria) on September 27, 2018.

Focusgroup_2. (2018). Interview by Andreas Pirchner with the participants of focus group 2 in the framework of the GAPPP lab concert at the the Institut für Elektronische Musik und Akustik’s Cube in Graz (Austria) on September 27, 2018

Hamilton, Rob. (2018). Interview by Andreas Pirchner with composer Rob Hamilton on his work Trois Machins de la Grâce Aimante within the framework of GAPPP – Gamified Audiovisual Performance and Performance Practice. Interview conducted on July 29, 2018.

Lüneburg, Barbara [2018a]. “Between ‘Ludic Play’ and ‘Performative Involvement.’” eContact! 202 (2018). Available online at https://econtact.ca/20_2/lueneburg_gamifiedmultimedia.html (last accessed June 18, 2020).

Lüneburg, Barbara [2018b]. “Between Art and Game: Performance Practice in the Gamified Audiovisual Artworks of GAPPP.” The Computer Games Journal 7 (2018): 243–260. Available online at https://doi.org/10.1007/s40869-018-0066-7 (last accessed June 16, 2020).

Lüneburg, Barbara [2018c]. Interview by Marko Ciciliani with performer Barbara Lüneburg within the framework of GAPPP – Gamified Audiovisual Performance and Performance Practice. Interview conducted on September 29, 2018.

Quartet. (2018). Interview by Barbara Lüneburg with the performers of Hamilton’s string quartet in the framework of the GAPPP lab concert at the Institut für Elektronische Musik und Akustik’s Cube in Graz (Austria) on September 27, 2018.

Ressi, Christof. (2018). Interview by Andreas Pirchner with composer Christof Ressi in the framework of the GAPPP performance at the Institut für Elektronische Musik und Akustik’s Cube in Graz (Austria) on September 27, 2018.

Salen, Katie/ Zimmerman, Eric. (2004). Rules of Play: Game Design Fundamentals. Cambridge, MA.

Turner, Stephen P. (2014). Understanding the Tacit. New York.