Man with a Computer

a summary of

The Language of New Media

a book by

Lev Manovich

used to analyze

Riven

a new media object by

Rand Miller, Robyn Miller, et al.

             Lev Manovich wears his reference points on his sleeve... or rather, his dust jacket. The cover of The Language of New Media features a strip of motion-picture film superimposed with binary code. He explains on his web site that the unofficial subtitle of the book would be Everything You Wanted to Know About New Media But Were Afraid to Ask Dziga Vertov, and the only illustrations used throughout the book are thematically-related stills from Vertov’s Man with a Movie Camera. (Still images of works referred to in the book are instead available on his web site.) A supplemental chapter at the end of the book is devoted to the impact of digital technology on cinema, which he calls “the most important cultural form/medium of the twentieth century.” Although he consciously avoids making predictions about the future of creative media (focusing instead on its continuity with the past), he believes that new media is at roughly the same stage of development today as cinema in the early 1900's.

             This notion is intriguing, because it suggests that there are analogs of D.W. Griffith’s Birth of a Nation to be found, introducing techniques which will become part of a standard vocabulary for this new set of art forms. (Though one hopes these new-media Births will be less contemptible in their content.) In this book, Manovich tries to distill and describe the vocabulary of these experimental works to date. He laments that the early years of cinema - before its well-preserved and heavily-studied “classical” period, in which a great deal of creative experimentation gave way to the mainstream conventions used ever since - weren’t better documented at the time for analysis; this book is his attempt to ensure that the same is not true of new media.

             Manovich begins by defining new media. He outlines five basic principles which characterize it: numerical representation, modularity, automation, variability, and cultural transcoding:

             Numerical representation refers directly to the digital nature of new media objects. Because they are stored numerically, their elements can be defined mathematically, such as “a rectangle of length 4 and width 3” rather than existing any concrete analog form.

             Modularity indicates that new media objects are built from components (images, text, video, shapes, etc.) but those components retain their independent identity. So one image can be replaced with another image, text can be changed, the video can be color-corrected, and so on, without affecting the rest of the object. This is a characteristic derived from the way computer programs themselves are created, combining procedures and subroutines into a larger structure, but with the ability to recode any such module as needed.

             Automation is a consequence of the first two principles. Because a new media object can be taken apart, and then addressed as an abstract set of data, a machine can perform further work on it. Additional human labor is - at least in part - unnecessary. This enables a Photoshop user to transform an entire image - blurring it, adjusting the contrast, or applying a filter to make it look like watercolors - with a keystroke or two. And then do the same to an entire collection of images, with a few keystrokes more. With the application of more advanced mathematical models of artificial intelligence, the computer can generate - from the ether - a moving image of a flock of birds or a crowd of people milling about.

             Variability introduces a wrinkle into the copying of a work. Mass duplication is nothing new; that’s been possible at least since the invention of woodcuts. But new media objects no longer need to be identical. A web site can programmed to display differently depending on the language the visitor’s web browser is configured to use, or whether she has a high-speed connection, or what ZIP code she lives in. Or an interactive virtual world can present different images or events depending on the choices the user makes: what direction to travel in, or what objects to manipulate.

             Transcoding is the most profound difference of new media. By this term, Manovich means that a work in digital form exists in a new mode of thinking. Despite the visual similarity between The Starry Night and a digitized version of the painting, the latter exists only in terms of file sizes, data structure, sampling resolution, and compression format, which are entirely separate characteristics from the terms of content, paint media, meaning, or even color palette and physical dimensions that we as humans would apply. And this difference will inevitably affect the nature of the objects created in this new media.

             In his discussion of new media’s attributes, Manovich follows the model of computer architecture, working from the lowest level of the system to the highest. He begins with the Interface, which he equates with the hardware of the computer, specifically the screen and the controls. The Operations follow, corresponding to some of the basic functions of the modern computer’s operating system, such as select, copy, and print. Next are the Illusions, analogous to the application software we might run on a computer, especially photorealistic rendering. Finally, he ends at the Forms, addressing the organization of content a user would work with, including data and narrative.

             Manovich identifies three primary influences on the interface of new media: the printed word, cinema, and the general-purpose human-computer interface of contemporary operating systems, which is itself derived from the first two but adds unique elements.

             The conventions of the printed word are the oldest, and are made “natural” by virtue of our universal familiarity with them. Computer file systems in the 1980's used the metaphor of sheets of paper in folders in cabinets. The standard metaphor used by the World Wide Web is that of pages of text, all linked to other pages of text, exchanging a hierarchical structure for the peer-to-peer collage of hyperlinking. Other media types are included in web pages through the same means that printed pages include illustrations, photographs, and charts: juxtaposed rectangles on a flat surface.

             New generations of artists and computer users have grown up with cinema (and its cousin television), making it an instinctively natural mode of communication for them. The rectangular frame that serves as a kind of “window” into another world is a centuries-old tradition in our society. The key addition that new media derives from cinema is the moveable motion picture camera (which fascinated Vertov so greatly). Just as a movie camera can turn to look at different things, a computer can scroll back and forth to look at different parts of the contents of a window. Even in a 3D virtual reality environment, where the user theoretically has complete control over the motion of the “camera”, the creator will usually provide a selection of pre-defined viewpoints to choose from, effectively framing her creation. The conventions of cinema have become tools added to the digital computer interface.

             The addition of the toolset is what makes the computer interface more than just a hybrid of print and cinematic metaphors. Although many of the controls for a computer are external to the screen (e.g. mouse, keyboard, joystick), most applications require screen real estate to act as part of the control interface. So the screen is forced to serve dual, competing functions: as a window into a deep illusionary space (a mode favored by cinema), and as a flat surface onto which controls and textual information can be placed (favored by print).

             In addressing the operations of new media, Manovich focuses on the computer’s ability to select, alter, and combine content from multiple sources. In fact, he describes this process as typical of “creating” a work in new media... even if nothing actually new is added to it. Most creative software packages come with a library of prefabricated elements to work with, whether it’s 3D models, texture maps, video transition effects, web site templates, or stock photos. If Post-Modernism is about drawing upon the past to combine things in new ways, new media is genuinely Post-Modern, and the computer legitimizes this approach by embedding the commands for it into the operating system.

             This is similar to the use of montage in old media, but new media tends to look down on montage per se. Instead it favors compositing various elements into an apparently seamless whole. For example, the makers of Jurassic Park labored not just to juxtapose dinosaurs with humans, but to make it seem that they were all part of the same continuous fictional reality. Ironically, doing so required taking the perfectly-rendered digital velociraptors and degrading them to match the flawed photos of the real elements, with their film grain, lens flare, motion blur, limited depth of field, and aerial perspective.

             The user of a computer typically has the ability to customize the environment to their preferences. For example, she can make the text larger, change the colors of the menus, and select a background image to her liking. These capabilities are common in new media objects.

             Illusion can be the ace up new media’s sleeve. Manovich cites the fine art world’s movement away from illusionistic representation in the 20th Century to partially explain the public’s lack of interest in it (and cinema’s complementary popularity). While early computer art in the 1960's was as abstract as the painting of the day, it didn’t take long for computer graphics to attempt realism. Over the course of a few decades it went “from Cimabue to Giotto to Leonardo and beyond,” and can now deliver the photorealism the public apparently wants. And in contrast to pre-Modernist realism, these synthetic images can move.

             Early cinema and early computer graphics used strong linear perspective, deep focus, and lots of motion perpendicular to the frame to trumpet their “realism”. As technology evolved, subtlety of shading, light, and texture became the standard touchstones in both movies and digital creations. Lately, the ability to simulate such barely-conscious natural elements as the motion of grass in a breeze or ripples on the surface of a lake are the ways new media artists show off their mastery.

             Although this illusionism is important to new media, with current technology interactivity often introduces the peculiar oscillation between creating an illusion and breaking it. A user might spend several seconds waiting for a web page to load, during which the activity being animated on her screen is not the content of the page, but its transmission and construction. Finally it finishes and presents itself as whole, she absorbs the content, she clicks a link... and the illusion is shattered again. Virtual-reality environments typically behave the same way.

             Manovich describes two fundamental forms which new media objects tend to take: the interface to a database and the navigable space. The former is usually preferred when the purpose of the object is to provide information and the latter is preferred for recreation, but increasingly we tend to experience both models whether at work or at play (in much the same way that office equipment and home entertainment devices are now the same thing). And most new media objects combine some of both models, giving one primacy over the other depending on the goals of the designer.

             Although the term “database” evokes punch cards and inventory reports, it refers to any searchable collection of items, and those items can be of any media, such as images, prose, animations, video clips, etc. CD-based multimedia encyclopedias are a common example. Many artists’ web sites qualify, though they are often little more than structured indexes to their old media work. The selection of music stored in an Apple iPod is a new media database. But the most ubiquitous such object is the web search site, such as Google or Yahoo, which may provide multiple interfaces through which to search (e.g. hierarchical directory, free-form text query, articles filtered by several specified parameters). Although narrative can be applied to or built from a database, its responsiveness to random queries from the user generally prevents it from having a narrative of its own.

             Although one might make the same assumption regarding new media objects based on navigable space, Manovich points to ancient epics (Homer’s Odyssey being an obvious example) in which the narrative is bound to a travelogue of the hero’s journey. The sequence of events required for a true narrative to exist can be mapped through the sequential geography of the space. This movement can also serve its own purposes, however, providing description rather than narrative. Although navigable space appears to be a continuous thing in itself, a computer-generated space is really just a modular aggregation of discrete objects, making it a highly flexible form, adaptable to a wide variety of games, environments, and narratives, much as text can be used for narration, description, or even puzzles.

             The new media object I’ve selected to examine is a piece entitled Riven, produced by brothers Rand Miller and Robyn Miller (with assistants) from 1993-1997. Although it is commonly stocked in the “games” section of software retailers, it’s a very different animal from Space Invaders, Virtua Fighter 4, or the home version of Who Wants to Be a Millionaire? Not just because it’s really well done (the flawed criterion commonly used to distinguish “art” from “illustration”, or “cinema” from “movies”), but because the “goal” (a fundamental element of any game) is not central to it, or even strictly necessary; Riven can be appreciated on purely aesthetic terms, even if the participant never solves the larger puzzle built into it. In fact, after “finishing” the challenge presented to the participant, return visits to the environment can still provide interesting new experiences. (I had several surprises while revisiting it for this essay.)

             Furthermore, the full range of Manovich’s framework for discussing new media is readily applicable. In fact, he uses various games as examples of new media objects, ranging from the prototypical Spacewar of 1962 to recent releases. So I hope I won’t be penalized for choosing such a “low art” subject.

             Riven can be described most briefly as a virtual reality. Robyn Miller says, “We are creating environments to just wander around inside of. People have been calling it a game for lack of anything better, and we’ve called it a game at times. But that’s not what it really is; it’s a world.” It’s photorealistic. It features substantially open-ended interactivity. It’s vaguely surreal. It draws upon multiple media. It’s multi-sensory. It utilizes the screen in distinct modes with a minimalist interface. But enough of the travel brochure blurbs; let’s begin the tour.

             Starting with the interface, Riven most strongly reflects the influence of cinema. When activated, it presents a pair of animated studio logos, each accompanied by a brief musical interlude, reminiscent of movie studios’ “vanity cards”. (There will also be scrolling credits at the end.) The screen is then filled by a cinematic sequence in which Rand Miller (in the role of the fictional creator of the world Riven, a man who knows the arcane craft of creating worlds via writing) explains the situation and asks the viewer to enter one of his worlds to find someone. The scene then cuts as the viewer is transported to Riven, a brief bit of expository action is presented, and... nothing happens, until the viewer becomes the user, and begins to explore.

             Although the cinematic sequence has concluded, the interface retains its transparency and depth. There is no control panel on screen. There is only the rectangular window providing the user’s perspective on the world, and a mouse-controlled pointer. To move the “camera”, the user points and clicks where she wants to go. To turn, she clicks to either side, and to look down or up, she clicks at the top or bottom of the screen.

             When the pointer strays below the window, a small image of a book (given to the viewer during the opening cinematic) appears. Clicking on it zooms in and then opens the book, switching Riven to a print-derived interface, in which the user can page through the book, reading the information contained in it. Another click closes it, returning the user to the explorable world.

             Riven contains more than places to go, however. The cinematic window also depicts objects which can be manipulated. Rather than flattening the screen into a control panel, however, they remain part of their environments and can be operated by touching their buttons or moving their levers, easily effected with the mouse pointer. This is a strong and successful fusing of content and interface, rendering the distinction between them minimal. Although the cartoony appearance of the pointer (as a hand with its index finger extended or poised to grasp an object) serves as a constant visual indicator that the user is running a simulation, it’s an immersive experience nonetheless.

             The designers of Riven do exercise a degree of control over the “camera” in this interactive environment. The views available to the user are in the thousands, but all pre-defined. The most compelling reason for this is technological economy: it allows the designers to render all of the possible scenes during production rather than providing the playback machine with all of the underlying data in full detail and rendering the scenes on demand. This makes for a smaller set of data to distribute and profoundly faster displays. It also serves the narrative aspect of the work, as I’ll go into later. While this is limiting, the Millers compensate for this by encoding as much descriptive information into these images as they can.

             The space is filled with a large number of elements, very carefully composited together. Although there is a great deal of empty space to be viewed and sometimes traversed, space which adds a great deal of figurative atmosphere to the environment, the density of rendered objects is impressive; it seems that every detail is included. Very few items seen in Riven are physical; other than the actors and their props in the handful of cinematics, everything is rendered from mathematical models and digitally-painted textures. The compositing is not perfect, however; there are sometimes visible “seams” and color variations between static and animated elements of a scene while something is happening in it. The compositing also sags each time the user moves: the scenes change discontinuously. Cinema-style fades and cross-cuts attempt to reduce the impact of these transitions.

             Although the user has a great deal of freedom to choose where to go and what to do in the environment, there is little she can do to customize it, contrary to the expectation with so much other software and the theoretical potential of new media. The scene transitions can be skipped for faster traversal of the space, or slowed for a more illusionistic experience. The sound and certain purely-cosmetic visual effects (which might overwhelm a slower playback device) can be disabled. But that’s about it. This retention of control by the creators at the expense of the user’s control is necessary to maintain the richness of the experience, providing the elements necessary to maintain the illusion.

             The primary means of creating that illusion is the photorealistic rendering of the environment. Not only is attention paid to the shape and textures of objects - making knives look like real oxidized metal, rocks like minerals, and trees like wood - but also to matters such as atmospheric perspective - ensuring that distant object appear distant - and lighting - providing the framework to hold the whole scene together. Seemingly trivial touches like the inclusion of flies buzzing about, a rippling effect on bodies of water, and ambient sounds enhance the illusion strongly, giving even static images a semblance of cinematic realism. Although Manovich was referring to Hollywood movies, it’s a smaller example of what he calls our civilization’s “equivalent to medieval cathedrals... truly epic both in their scale and attention to details. Assembled by thousands of highly skilled craftsmen over the course of years.”

             One of the things that breaks down the illusion, even more than the transition of images does, is the occasional shifting of modes. For example, after turning a lever there may be a second or several of full-motion animation in which something happens as a result. This simulation of action might seem to advance the illusion, but paradoxically it does not. Manovich cites a recurring theme in the development of modern media: the more mobile the image, the less mobile the viewer must be; you can move about while viewing a painting, but you must sit still to watch a movie, and fully-immersive virtual reality could require strapping your entire body into a machine. The same paradox applies here: while the on-screen image moves, the user freezes and becomes a mere viewer, no longer able to interact with the world.

             Despite its cinematic interface, one of the recurring content themes of Riven is the book. (As presented in the opening sequence, the world of Riven itself is purported to have been created through writing in a book.) Not surprisingly, its creators intended it to be substantially narrative in form. But it contains many elements of a database as well. Going behind the scenes to the inner workings of it, one finds a fairly large collection of data: images, animations, sounds, linked lists, rule-sets, and so on. This structure is also reflected in the presented environment. Just like Chekhov’s gun, every object in the environment has a reason for being there. Many (especially the ones the user cannot interact with) are there just to enrich the experience, and a few are narrative red herrings, but the set of places and things the user encounters is roughly congruent with the set of elements required to solve the larger puzzle of the game. The user can explore just for the sake of exploration, but in doing so she must work through many of the smaller puzzles, advancing the narrative (or at least learning the backstory of it) along the way. For example, figuring out how to turn on the power to a particular device enables her to get to another location where she can find a clue to another puzzle or witness an event which is necessary to the final resolution.

             The creators encourage the advancement of the narrative in other ways as well. The user has a great deal of freedom to move about randomly, but only within confines defined by them. Those restrictions serve to shepherd the user along, quite literally compelling her to stay on the path, to go where they want her to go. A virtual reality is in many ways like an installation, and the Millers utilize some of the same techniques used by installation artist Ilya Kabakov. He devised several strategies to exercise a degree of control over how participants experience his installations, including the technique of dividing the works into discrete connected spaces (much like the islands in Riven) with a well-defined path through and between them (again, like the footpaths in Riven), with objects “conveniently” placed to obstruct people from going where he didn’t wish them to go (like Riven’s many natural obstacles), and even pages of text in places where he wished them to stop and spend more time, to effect a rhythm to their navigation of the space (like Riven’s notebooks and visually-interesting locations).

             Riven is consistently praised for its aesthetics: the diverse beauty of its scenes, the contemplative mood of its environment, the understated surreality of its design, the thoughtfulness of its visual themes, the richness of the backstory to its world, even the intellectual fascination of its puzzles. It clearly broke new ground on that level. But formally, as a new media object, it has been alternately hailed as revolutionary for its demonstration of the potential of digital media, and dismissed as regressive, for its reliance on the various conventions of old media like cinema. (At least one critic derided its predecessor Myst as a "slide show".) Manovich would probably regard it instead as necessarily evolutionary. It cannot break entirely with the historical trajectory established by cinema and other forms in the 20th Century, but it must bring new attributes that reflect its individual ontogeny, as an object built in the new media coalescing at the threshold of the 21st. Only time will tell if the techniques used by the Millers will establish them as analogs of Griffith or of Vertov, or just another couple of Men with a Computer.