In the second of two-part series on my time at PAX East, I want to analyze and speculate on the role of user interface (UI) and user experience (UX) in video games. I had the chance to attend a panel featuring prominent UI designers from the gaming industry: Vicki Ebberts [UX, Undead Labs], Alexandria Neonakis [UI/UX Designer, Naughty Dog], and Kate Welch [UI/UX, Freelance]. They offered a perspective on an aspect of game design that I’ve found is often considered an afterthought, despite how central user interface is to the quality of a game experience. Similarly to how I reviewed GeekNights’ panel on the concept of ‘losing’, I will begin by summarizing the key points of the talk, and will then offer my own views on the material presented.
As a point of preface, I should note that the brunt of UI theory discussed was cited from Erik Fagerholt and Magnus Lorentzon’s Beyond the HUD — User Interfaces for Increased Player Immersion in FPS Games, a thesis for Chalmers University of Technology. As a result, the Fagerholt/Lorentzon model will be largely recapitulated at the outset of this article. You can find a summary of their work courtesy of Anthony Stonehouse’s User interface design in video games. Do yourself a favor and hit the link to read Stonehouse’s summary; far as current taxonomies of UI go, I find this particularly useful and clear.
The Fagerholt/Lorentzon model, as outlined by the panel, turns on the fact that video games are a narrative medium, which differentiates them from other applications of user interface. He classifies elements of video game UI based on whether or not they exist in the narrative of the game and/or in the actual world represented by the game (called “the geometry” of the game’s world); in so doing, he establishes four distinct categories of UI elements. I will summarize each in turn.
- Diegetic UI elements. These are the most “immersive” elements, which exist both in the game’s narrative and within the geometry of the game’s world. Examples are the Eagle Vision of “Assassin’s Creed” or the tactical vision of a game like “Arkham City.” Games set in the future or with science fiction are easy subjects for diegetic UI, the argument goes, because they can easily justify UI interfaces as standard technological devices, uniting player and avatar.
- Meta UI elements. These exist within the game’s narrative, but lack direct spatial relation within the world itself. Blood spatters on the screen when low on health are an extremely common example: no one within the game’s world can “see” this, but it is representative of your character being on the verge of bleeding our within the narrative.
- Spatial UI elements. In contrast to meta elements, these UI aspects bear physical relation to the game’s world, but don’t factor into its narrative. Think of arrows over your avatar’s head that point to your next objective: the argument is that such elements, while represented within the world, are not perceptible to anyone but the player, as they are meant to inform the player without actually impacting the storyline directly.
- Non-diegetic elements. This category is thought of as something of a “last resort”: its elements are neither in the game’s geometry nor in its narrative, and typically are presented as such only when it would be cumbersome to present them in any other way. “Halo” and many online games present information on weapons &co in this way.
The PAX panelists pointed to two major motivators in philosophy of UI design, both of which are ultimately concerns of player experience: the first is how intuitive the UI system is for the player to negotiate and manipulate; the second, which is closely related to the first, is the degree to which the player is “immersed” in the world of the game. The line that was repeated throughout the panel was that the sign of an effective, sleek UI is that no player actually comments on or notices the UI. When the future of UI was discussed, motions were made toward the promises of virtual reality to eventually develop games in which UI is ultimately seamless.
This will suffice for my reconstruction. My critique is fairly straightforward. I really just want to pose one question: ought UI to categorically pursue total player immersion as the ideal towards which it strives? The Fagerholt/Lorentzon model is useful because it defines different types of tools available for constructing an interface between a player and a fictional, narrative-driven world. Given all of these tools, what grounds do we have for privileging immersion as the interface’s ultimate goal?
It may not be obvious that there are any competing ideals to consider in UI design, so permit me a moment to flesh out the case with an example. Back when Richard Wagner was writing operas, he developed an aesthetic concept known as “Gesamtkunstwerk“: German for, roughly, “collective work of art.” The guiding principle behind this concept and Wagner’s aesthetics was that the audience of a work of art ought to be completely subsumed by the work of art, such that the world established by the art constitutes the whole of the viewer’s experience while engaged with it. Wagner had an entire opera house built to reflect this philosophy, pioneering such innovations as turning off the theater’s house lights, and hiding the orchestral section so that the music seemed to merely “radiate” from the stage.
So immersion is certainly one sort of possible aesthetic virtue, and in many ways, Wagner’s philosophy anticipates the cinematic productions of modern times. But it is also not the only possible virtue, and many artists actually push back against this principle, striving instead to deliberately estrange the viewer of a piece of art from the art object itself. A common response to Gesamtkunstwerk is actually dangerously subversive because of the degree to which the piece of art aims to strip the viewer of their individuality, immersing them in the collective aesthetics instead. Wagner’s anti-Semitism, along with the Nazi’s appreciation of his operas, only exacerbated this view.
Of course, I don’t aim to make those same arguments against immersion: virtual reality really is a promising medium for the development of even more nuanced aesthetic types and narrative mechanics. I merely wish to push back against the notion that the apex of gaming is a vision of “total immersion.” Firstly, it’s difficult to even conceive of what that means: for, if we take the concept seriously, then it seems that a totally-immersed player would be unable to differentiate the game from reality at all — think “The Matrix” for a model. At that point, does it continue to be a game? Is this even tractable as a theoretical notion of art? We will surely make great progress in the coming years with respect to making the relationship player and game more seamless, but we ought to analyze seamlessness more precisely before extolling it as UI’s goal.
Moreover, we have seen that UIs have at least four distinct types available — diegetic, meta, spatial, non-diegetic. With so many options available, it seems naive to claim that the ultimate goal of UI is to be as unnoticeable as possible. In my own work, I have aimed at articulating how the different relationships between player, avatar, and game world can establish unique aesthetic effects (e.g., the embedded narratology of “Assassin’s Creed,” or the player-dependent metaphysics of “Legend of Zelda: Majora’s Mask”); the most immediate facilitator of these interactions, by virtue of being the conduit between player and avatar, is the UI. So I think it follows that UI ought to explore as many permutations of aesthetic principles as possible, rather than mere design permutations, such that we can explore the broadest boundaries of what sort of stories video games as a medium are capable of telling. Perhaps a counterpoint to immersive UI could be intentionally alienating UI that make the player feel like an utter stranger in spite of controlling the avatar within the game; such a model could be the foundation for an aesthetic of estrangement that, by virtue of being interactive, could be much more successful as a video game than as art in another medium.
What’s more, my intuition is that it’s an artifact of the current state of UI design that we see a conceptual difference between physical space and narrative space in a video game, as the Fagerholt/Lorentzon model suggests. As we develop a more comprehensive theory of video game aesthetics, I think it will become increasingly clear that physical game space and what’s called “narrative” are two different ways of seeing the same aesthetics. Already, the lines between the four categories are blurry at best: we may say that directional markers pointing the player towards a goal are “merely spatial”; yet if we extend the concept of game narrative to include the player as a fundamental, as I have argued that we must, then is this not also a narrative element? And this is the crucial point: for once we accept the player as a part of the game’s narrative, and the totality of the game as its world, then it seems as though all UI, while still aesthetically differentiable, is intrinsically diegetic.
It was wonderful to hear from industry professionals on a part of game design that I had not previously considered with much rigor at all; and of course, I must emphasize that I am in no way a game designer myself. However, I do hope that the trajectory of game design practice and rhetoric in the future focuses not on a goal of immersing the player in the game. Rather, let us accept that the game is already immersed in the player, and take steps to explore all the possible aesthetics that can follow from this relationship.
