Mark R. Westmoreland, Annet Pauwelussen, and Silke van Diemen[1]
Recommended citation: Westmoreland, M. R., Pauwelussen, A. and van Diemen, S. (2022). ‘Kaleidoscopic Vision: Immersive Experiments in Maritime Worlds’, entanglements, 5(1/2): 50-70
Abstract
This article critically engages with intersecting tropes of immersion central to both notions of ethnography and technologically-enabled “extended reality” (XR). The authors build on Pauwelussen’s experience of disorientation while doing research within the dynamic land-sea environment of the Makassar Strait in Indonesia by hacking 360° video technologies to juxtapose multiple scenes of ‘flat’ video. These techniques of spatial montage enabled the authors to destabilize the assumption of ‘omni-scopic’ spherical holism and reconceptualize the ontological complexity of these entangled lifeworlds. Unexpectedly, this multimodal assemblage of agencies offered a kaleidoscopic perspective on immersive ethnography comprised of dynamic perception and speculative thinking.
Keywords: kaleidoscope, 360° video, immersion, montage, ethnography
Introduction: Immersive ethnography
Despite the scientific imperative to maintain critical distance, anthropologists have long trafficked in the tropes of immersion. Sharing affinities with other field-based sciences, ethnography’s position in situ and embodied practice of seeing with one’s own eyes (cf. Amit, 2000 among others), not to mention listening with one’s own ears (Helmreich, 2007), promises empirical forms of knowledge available only by ‘being there’. Ethnography’s quintessential ‘immersive experience’ demands a heuristic process of learning for oneself, which involves preparing for the unexpected, particularly in worlds still unknown to the ethnographer (e.g., Amit, 2000; George and Jones, 1980). One can prepare and prepare some more, but in the end it’s also a matter of diving in and learning how to swim in a different environment, with currents that can move the researcher in unexpected directions. These unanticipated shifts in direction are rich opportunities to learn from other perspectives.
When carrying out mobile ethnographic research among seafaring people living in dynamic land-sea environments of the Makassar Strait (Pauwelussen, 2017), Annet’s field research engaged additional dimensions of immersion. Skimming along and submerging into watery terrains, Annet had to both ‘get her sea-legs’ (Pálsson, 1994) among her collaborators and also grasp a dynamic land-sea environment in Indonesia. Attempting to navigate unexpected currents in her fieldwork, Annet started experimenting with different visual strategies as a way to document and translate her immersive experience. Zooming out, she tried to find ways to contextualize these experiences. To make visual sense of the seascapes particular to various kinds of seafarers, Annet utilized mapping and video recording.[2] This collection of maps and film footage, recording human and non-human movements above and below the water surface, offered a kind of a visual ‘thick description’ (Geertz, 1973). However, in line with the multiplicity of the marine space she explored in her research (Pauwelussen, 2017), this assemblage of visualisations did not add up to a seamless whole or lead to a cartographic overview, but instead to an assemblage of entangled storylines.
At the same time, Mark began exploring notions of “immersive ethnography”, enabled by emergent 360° video technology using participatory methods in underground mining sites (Westmoreland, 2020). The emergent field of “extended reality” (XR) gathers several related technologies that include VR (virtual reality), AR (augmented reality), and 360° video under a single conceptual framework. While we have found few companions trying to understand the radically different vision of 360° video (cf. Favero, 2018; Gómez Cruz, 2017), XR applications in ethnographic and field-based documentary research are beginning to proliferate.[3]
Technologists champion the immersive qualities of 360° video for the way the all-encompassing image bubble virtually surrounds its user in a dynamic and interactive environment, however proponents typically renew uncritical notions of immersion as an index of realism. Compelled to look around, thereby combining bodily movement with responsive audio/visual input, these videos generate a feeling of really “being there.” This teleportation feature is particularly apt for generating awe-inspiring reactions when expertly positioned within spectacular landscapes and locations otherwise inaccessible to the viewer, which invariably evokes exotic tropes of traveling to alien worlds. Furthermore, the realist features within immersive journalism and other forms of “immersive witnessing” have also gained significant attention as hi-power “empathy machines” (Milk, 2015). Reproducing the same old tropes of visual realism that earlier advancements in visualization technologies championed, such as the camera obscura, stereoscope, and motion pictures (Crary, 1990), this corollary between immersion and empathy tends to be overly celebratory and neglects the gimmicky quality of these technological spectacles that may actually obstruct the promise of bearing witness (Nash, 2018). Moreover, Sam Gregory, of the well-known human rights organization WITNESS, asks whether we should privilege empathy “over understanding, compassion, solidarity, or action” (Gregory, 2016). Gregory thus signals a conceptual opening for us where we might (re)invent “immersive ethnography” according to different terms than visual realism and personal empathy, terms which might better align with the crucial concerns of contemporary anthropology like intervention, collaboration, and cultural critique, as well as the conceptual frameworks of multiplicity, entanglement, and precarity.
This article recounts our various efforts (independently and together) to navigate the experiences of immersion through different visualizing strategies in order to expand beyond tropes of immersive realism[4]. Whereas Annet’s research on ‘amphibious anthropology’ embodies expanded sensory registers of immersion at sea, Mark’s interest in XR technologies explores the affordances of mediating immersive experiences in the anthropological classroom. Combining Annet’s ethnographic research on maritime worlds in Indonesia with Mark’s interest in using self-reflexive strategies to interrogate the tools of mediation and corresponding knowledge practices, we endeavored to adopt immersive XR technologies without reproducing the tropes of realism and empathy. We argue that the central practice of in situ participant observation privies ethnographers to the problematics of immersion, thus offering us potential insights about alternative conceptual frameworks. By destabilizing the assumption of ‘omni-scopic’ spherical holism in 360º video, we endeavored to use immersive technology to juxtapose different, yet partially connected, scenes of immersion and accentuate the simultaneity of multiplicity.
Our approach aimed to both respond critically, when realist modes of understanding seemed to lead to dead-ends, and speculatively about emergent perspectives that did not fit our expectations or existing categories. These strategies embraced the generative logic of hacking that works to rethink the intended purpose of a given technology. Whereas contemporary uses of hacking emerge as a practice to gain unauthorized access to computational systems and networks, we deployed the principle of hacking as a means of intentionally modifying a piece of technology or a process of utilization in a way other than intended, in order to produce different and potentially more useful kinds of results. In this, we follow McKenzie Wark’s claim, “Whatever code we hack, be it programming language, poetic language, math or music, curves or colourings, we create the possibility of new things entering the world“ (2004 [002]). For our part, we aimed to hack the conventional optics and haptics of virtual realism to show how XR immersion is always multiple and fragmented.
Put another way, if the telescope characterizes the optics of close examination from a critical distance, as well as the imperial gaze of seafaring explorers, our experiments instead revealed a kaleidoscopic perspective; an immersive experience in which fragmented optics and multiple visual strategies produce unique combinations of diverse, overlapping, and constantly shifting patterns of life. The kaleidoscope resonates with Anna Tsing’s efforts to “notice” the dynamic interrelationships in these precarious environments: “I find myself surrounded by patchiness, that is, a mosaic of open-ended assemblages of entangled ways of life, with each further opening into a mosaic of temporal rhythms and spatial arcs” (Tsing, 2015, p. 4). Accordingly, we begin by recounting a series of visual methods that Annet utilized to make sense of her ethnographic immersion while journeying with seafarers through uncharted seascapes. We then elaborate the work we did together using 360º video technology to assemble her materials into new analytical configurations. We therefore approached our collaboration as an experiment using alternative visualizing strategies to enact a different kind of immersive ethnography, premised not on empathy and realism but instead on diversity and juxtaposition.
Incomprehensible cartographies
In 2011, Annet embarked on 18 months of ethnographic research in Indonesia to study the different ways people relate to the sea. The research took place in two different coastal areas: one in Berau, in East Kalimantan (on the island of Borneo), and the other in Pangkep, in Southwest Sulawesi, near the port of Makassar. Situated central to the Coral Triangle—the world’s epicenter of marine biodiversity (Hoeksema, 2007; Verron et al, 2009)—both sites were target areas for the development of marine protected areas, to protect coastal ecosystems—primarily coral reefs—from degradation and overfishing (Glaser et al, 2015; Gunawan and Visser, 2012; Kusumawati and Visser, 2014). Here, Annet set out to explore how seafaring people, as well as Indonesian and foreign marine scientists, know and value the sea and its reefs differently, and how this difference plays out in the way marine conservation measures are supported or resisted (Pauwelussen, 2017; Pauwelussen and Verschoor, 2017).
While Annet’s research started with a comparative framework of two different sites, soon it became clear that the administrative and terrestrial framing of the “fieldsite” did not correspond to the spatial experiences of the people living in the dynamic land-water environments of the Makassar Strait maritime region. Following their lives and practices necessarily took Annet beyond Berau and Pangkep. Coastal and maritime communities appeared highly mobile, their networks shaped through relations of kinship, trade and belonging that enacted the sea between Kalimantan and Sulawesi as a maritime highway, with people, vessels and things on the move. As a way to culturally and physically immerse herself in their world, Annet decided to follow people along their movements in and across the Makassar Strait region, including their underwater practices of dive fishing, gleaning, and restoring marine nature, shifting her research into a mobile and amphibious ethnography (Pauwelussen, 2015; Pauwelussen, 2017, c2). This immersion inverted the terrestrial bias (Jue, 2020) by foregrounding the sea as the center of social practice. Importantly, in these situated and embodied practices of seafaring and doing fieldwork, immersion as cultural and oceanic experience are not ontologically different but inherently entangled.
While fieldwork allows an ethnographer to see with one’s own eyes in close proximity to the perspectives and lived experiences of others, being confronted with different ways of ordering and experiencing the world often involves an initial period of perplexity. For someone who is used to terrestrial orientation, the embodied experience of doing research in and at sea can be unsettling, amplifying the feelings of disorientation and discomfort that embarking on watery immersive ethnography can generate (Helmreich, 2007; Jue, 2020; Merchant, 2011; Pauwelussen, 2017, c2).
Annet experienced this disorientation during her journey, in March 2013, to the Masalima Archipelago, situated in the middle of the sea where the currents of the Makassar Strait join the Java Sea. From the Sea port of Makassar, she joined Indonesian friends on a trading boat to Pamantauan, one of the Masalima Islands. Although she had heard about the island many times through the stories of traders and travelers, she had not been able to locate the island on a map. On most cartographic maps of South-West Sulawesi, the Masalima Archipelago indeed doesn’t exist. As a tiny archipelago situated far from the terrestrial boundaries of the mainland Pangkep District to which it administratively belongs, Masalima is usually excluded from cartographic visualization—with the exception of expensive specialized maps for seafaring purposes. After one day and one night of travelling at sea, Annet arrived on Pamantauan Island in the dark, feeling tired, disoriented and seasick, indicating how doing immersive sea-based fieldwork may come with spatial as well as visceral experiences of confusion. In the weeks that followed, she would gaze out at the sea surrounding the tiny island and wonder where she was, as boats were coming from and going in different directions.
Born from an urge for orientation, and anchoring her place in that dynamic sea world, Annet initiated a mapping exercise, asking fishers and captains to narrate and draw the marine space around them. In a few days, several maps were drawn with men, on the back of spare copies of research permit forms taped together.
The above figure shows one of the maps that resulted from the mapping exercises on Pamantauan Island. The map shows four seafaring routes, starting from a group of islands at the center (the Masalima Archipelago) extending outwards, drawn by three different seamen. Starting with only a few A4 papers taped together, new ones were added whenever the drawing of routes extended beyond the edge of the paper. The lines on the map indicate the routes taken by the fishers and captains travelling along ‘seamarks’ well known to them: islands, sandbanks and reefs, visualized as ovals and names on the map (see also Pauwelussen, 2017: c2). The map faces west: The B on top stands for ‘Barat’, Indonesian for ‘West’. U (Utara) is North (on the right). The map shows the Makassar Strait as a center of activity, a social space reproduced in the travels of seafaring people between the Indonesian islands of Kalimantan in the west, Sulawesi in the east and Madura, Bali and Lombok in the south.
Initially, Annet was confused by the map as she failed to match the lines with the logics of cartographic images that she was accustomed to using for spatial orientation. This incompatibility was not just a matter of scale but involved different modes of narrating marine spatiality. Crucially, the map shows a relational and situated (and not cartographically) defined sea space. While a cartographic map involves a distant view ‘from above’ and assumes visual realism, the map above is a narrated map in which (story)lines constitute strings of islands and reefs familiar to the storyteller, as he has been there before (cf. Haraway, 1988 for a discussion of these different strategies and technologies of visualization). The line-map is seasonal, too, as the men explained how lines would have to be drawn differently when currents changed. The map is also plural, showing different wayfaring journeys and thereby multiplying destinations. On the left side of the map as well as the lower left corner, one can see there are two Islands of Lombok (‘P. Lombok’). This is not a mistake, nor is it illogical in the context of the map according to the men making it. Once we see the map as an assemblage of visualized situated storylines, it makes sense that the different routes drawn by different people lead to their own ‘Lombok’ destinations. In contrast to the cartographic overview, the above map invites us to read it by an immersed following of different storylines which are nevertheless compressed into one image.
At the time of fieldwork, the mapping exercise came at a moment of confusion for Annet, as she did not know how to make sense of what she saw. She was even annoyed by what she initially saw as ‘messiness’. It is in such fieldwork encounters, while making sense of complex and unfamiliar worlds and modes of ordering, that one may struggle with incoherence and disorientation. The experience of confusion in immersive fieldwork encounters is not just a nuisance: it has productive affordances, stimulating the ethnographer to open up to what is radically different (Verran, 2001, p. 5). Marilyn Strathern has referred to the inevitable partiality of immersion as the ‘ethnographic moment’: a dazzling experience that confuses the fieldworker, yet also invites her to follow unanticipated analytical currents. This also requires a process of unlearning taken-for-granted ways of viewing and orienting. Here, we refer to the basic methodological struggle in ethnography of making sense of differing ways of understanding and relating to the world, without reducing this multiplicity to one perspective or frame (Verran, 2001; Viveiros de Castro, 2004).
Snapshots of sea life
While the mapping elicitation exercise visualized the spatial movements as narrated and drawn by seafaring people, Annet also came along on some of the journeys that these seafaring people made, as they engaged with the sea in the Makassar Strait maritime region, to engage in heuristic processes of understanding. Doing fieldwork at sea, in a boat, in the water, or staying in a house built on stilts above the shoreline comes with practical challenges for even the seasoned anthropologist. The use of electronic devices is often compromised by a lack of electricity and being at the mercy of salty water, wind, waves, rain showers and a scorching sun, while an ‘on the move’ ethnography also requires travelling lightly. These environmental circumstances make note-taking a battle with the elements and with one’s own guts, heaving and splashing about at sea. It was in those more mobile moments in the research—following and participating in the practices and movements of scientists, fishers, and shellfish collecting women, among others, to document how they engage with the sea—that Annet started to film what she saw.
More than mere documentation and in line with the urge to create maps, she created snapshots of sea life as a way to capture and make sense of this ever-changing environment and human engagements with it. Shot with a modest waterproof handycam Panasonic HX-WA10, she created a range of photos and short films to aid in writing notes when she had reached solid ground again. Short films appeared more appropriate than photographs to document people’s movements and sequences of practices above and below the water. And while filming human practices, Annet’s attention was often caught by other agencies and elements moving in and out of the scene: fish, jellyfish, dolphins paying a visit, the ever-changing shape and intensity of waves or colors of the seawater, as well as the glittering reflection of the sun while moving under the water’s surface. These presences mattered as they affected the researcher and those she travelled with by diverting attention or influencing action. They therefore formed part of the shared experience of being immersed in the affective and unsteady materiality of a sea world. Mesmerized by the sheer beauty or volatility of it, to capture this experience, too, Annet made a range of short movie clips following these other presences, without a clearly defined purpose other than to catch and document on film this highly dynamic world.
As a result of this filming activity, Annet returned to the Netherlands with 30 hours of video footage, how to best utilize it as yet undetermined. While the footage documented sequences of movements and practices at and in the sea within the same social-spatial reality of the Makassar Strait, the sequences did not add up to one documentary whole. Upon seeing some of Annet’s footage, Mark proposed to remediate Annet’s video footage of immersed sea research through a 360° video editing process. Although incapable of replacing the in situ field experience that anthropologists’ value, or reproducing the specific conditions where access to air, the physics of gravity, and the distortions of light must be individually renegotiated, Annet’s footage of this watery environment nonetheless presented ideal materials to experiment with the multiple immersive affordances of 360° video. Following Melody Jue, the combination of cultural, physical, and virtual forms of immersion with the “milieu-specific” conditions at sea would help us call “attention to the differences between perceptual environments and how we think within and through them as embodied observers“ (2020, p. 3). As such, the assembled nature of Annet’s diverse footage inspired us to curate a multi-channel immersion experience that combined several video sequences within a composite “spherical” projection.
Spherical montage
With the elimination of optical directionality inherent in conventional single-lens cameras that point at something and the expansion of the screen into a totalizing sphere, 360° video presents one of the most radical changes to the cinematic paradigm. By having the freedom to look in any direction, 360° video provides viewers an active role in constructing their own visual experience. And yet, as noted earlier, the adherence to a realism based on feeling present through the affective paradigms of holistic immersion and victim empathy would preclude other ways of interacting with the technology’s agency. Seeking ways to look beyond this normative vision of feeling present in a totalizing sphere, we envisioned an experimental approach premised on hacking the processes of assembling 360° video already encoded in this technology, but with the aim to produce as-of-yet unrealized immersive experiences.
Today, dozens of dual-lens, single-body 360° video cameras are available on the consumer market, but only a few years ago 360° video required using a multi-camera rig and then synchronizing and stitching together the “flat” images in post-production. For example, by placing six GoPro cameras in a cubical arrangement using a specialized tripod head, one could then convert this cube of six video panels into a unified sphere by morphing the edges together and algorithmically distorting them for optimal viewing. The distinct orientation of up/down/forward/backward/right/left becomes a ‘seamless’ bubble that encompasses the viewer. In other words, before dual-lens camera technology existed, 360° video was the result of hacking together new composite forms of video out of a series of discrete clips. While this method utilized six separate cameras unified in time and space to record a single scene in an omni-scopic manner, it occurred to us that six images taken from different moments and locations (like Annet’s research footage) could also be combined to create a six-panel immersive hexaptych.
Annet and Mark thus planned a speculative project premised on artificially creating a 360° audiovisual experience by ‘projecting’ Annet’s regular “flat video” from the field into a virtual spherical space. We hoped that this experiment might suggest ways to repurpose conventional “flat video” within an immersive apparatus to facilitate an expanded understanding of entangled lifeworlds, thereby helping to reconceptualize anthropological questions through a modality of spatial montage.[5] Utilizing the principles of cinematic montage, we juxtaposed images of various humans and nonhumans within an interactive and immersive XR environment. The introduction of such different optical records meant that ruptures between sequences would be more obvious. Rather than aiming to produce seamless spherical images that ultimately kept the edits hidden in the final form, we embraced the disjunctures produced in the hack itself by accentuating the seams in the final output. Instead of trying to suture these images into seamless unity aspiring for virtual realism, we intentionally juxtaposed sequences of video moving adjacent to each other.
The theoretical implications of these visible cuts meant that the totalizing whole became a fragmented series of spatial juxtapositions. Our 360° spherical modality of six stitched video images enabled us to reintroduce the element of montage, conventionally premised upon cuts between shots and frames. The (invisible) edit in most 360° video is not the cut “in the blink of an eye” between shots in time (Murch, 2001), but the faint stitch lines where flat images have been sutured. By combining existing “flat” media into a 360° environment, our hack of “stitching” images together produced a new modality for understanding these materials, one best described as spherical montage. This assemblage of incongruent images, in effect, juxtaposes them and utilizes notions of editing that have otherwise been evacuated from the new 360° vision. By accentuating edges instead of seamlessness, spherical montage radically restructures the spatial conventions of virtual reality (VR).
Rather than linear filmmaking, this juxtaposition of multiple simultaneous perspectives may be compared to the experience of a multichannel gallery installation, in which a viewer can only focus on one or two channels at a time but are nevertheless aware of other videos playing in the same space. This also echoes Suhr and Willerslev’s incorporation of montage techniques in observational filmmaking “to break the mimetic dogma of the humanized camera” (2012, p. 283; see also Suhr and Willerslev, 2013). Rather than conflating human and machine vision, “the multiplication of perspectives” means that ”we find ourselves decentered in an infinite totality of views that no longer affords us the illusion of ourselves as the unique center of the world” (2012, p. 291). We likewise aspired to use montage to stimulate a productive kind of disorientation: one that urges the viewer to reflect on taken-for-granted ways of ordering the world. Although the affordances of VR goggles enable feelings of being immersed (whether evoking snorkeling/scuba gear or not), our juxtaposition of different panels forces one to choose between the multiple scenes and confront the tension between the known and unknown.[6] While the combination of the contiguous frames facilitates tension between them, it simultaneously provides a singular encompassing experience whereby the diversity of being within a common geographic context (Makassar Strait) becomes recognizable. This combination of multiple narrative sequences of video creates an effect of simultaneous diverging realities.
Recasting cutaways
With funding from the Center for Innovation at Leiden University, we had the opportunity to explore these possibilities by producing immersive learning modules for anthropology students. To begin, Annet organized her 30 hours of footage from three different sea-based journeys and grouped this into folders. One folder contained footage of dive fishers who use an air compressor to collect sea cucumbers and lobsters from the reef at 10 to 30 meters deep. A second folder contained footage of coral gleaners, semi-nomadic Bajau women and, occasionally, men, who collect giant clams and other creatures from the reef at low tide, while also sustaining relations with reef spirits. A third folder contained footage of a trip Annet undertook with Indonesian marine ecologists who set out juvenile giant clams in artificial reef structures.
Silke joined the collaboration as our video editor. Reviewing the material from beginning to end, she created a log-list with descriptions, metadata and other characteristics. In Silke’s process of organizing Annet’s footage, she looked creatively for similarities within the footage. For instance, she made selections of video based on color and edited a few compilations from different clips based on these similarities. Subsequently, she continued the process by experimenting with different modes of viewing, including upside-down perspective, fisheye, equirectangular, connecting horizons in different groups, and ultraviolet fisheye filter. These different perspectives helped to conceptually bring the fragments together. During this process of experimenting with categorization and viewing, Silke began to notice parallel action and movement between the different groups. She continued to order these similarities of action into a series of chronological sequences, beginning with moving over water, preparing gear, diving underwater, being immersed underwater, and finally returning to land with the vessel. We applied this narrative framework to the three different sea journeys (described above) so they all begin with a contextual perspective, traveling on the boat, going underwater, and then engaging with marine creatures on the reef or sandy floor below the water surface. One panel shows the dive fishers, another the coral gleaning women, and a third the ecologists doing reef reconstruction work.
While these three human-centric storylines provided important contrasting worldviews, we needed a fourth horizontal panel in order to mimic 360° video composed of a cubical layout with four horizontal panels, plus the two vertical panels. In addition to three folders of different activity, Annet had instinctively placed the short sequences of sea-life and sea elements—waves, rain, reefs, fish and other animals—in a separate folder to be used as cutaways. In conventional documentary terminology, a cutaway is a shot of an incidental perspective inserted to make an edit within a continuous sequence of film less noticeable. Filmmakers generally plan for cutaways by shooting “b-roll”, thus signaling its diminished importance next to the main content. While sophisticated uses of cutaways may build meaning by association, cutaways do not themselves constitute a primary referent. The use of a cutaway implies cutting away from the main action. While working on the edit of our composite spherical video, the materials relegated to this lesser role began to take on a new and more important role that accentuated the multiplicity of being.
As such, we decided to dedicate the fourth panel to these non-human participants to better correspond with Annet’s own experience of being immersed in a lively sea world. While at sea, such material and animal activity appear not as a passive environment but rather as a spectrum of agencies that affect and condition the practices of people, both above and below the water surface. Life at sea involves humans but is not necessarily human-centered. Indeed, for seafaring and sea-dwelling people in the Makassar Strait, the sea as a life world and living world is constituted by a spectrum of human and non-human agencies. Accordingly, the fourth panel shows corals, fish, and jellyfish, as well as the activity of waves, colors, and light, to become agencies in the scene rather than a background for human action.
In order to create the complete cubical layout used in the 6x GoPro configuration, we needed to add top and bottom panels to the four horizontal scenes. Upper and lower panels provided spatial orientation and contextual coherency. In the sequences on the boat looking out across the horizon, shots of the sky above and sea below enclosed the vertical perspectives. For the underwater sequences, we added the water’s surface above and the seabed below—perspectives familiar to anyone who has snorkeled staring facedown at the seafloor or dived to the sea depths and turned to look up at the shimmering surface where water meets air.
Now that we had the six sides of our cube created, immersing a viewer within a box of images, the next step was to create a single spherical video. Without examples or templates that stitched together regular flat video to create a 360° video to follow, Silke began experimenting with combining these panels into the video editing software to find out what it would look like when played all together. The computer processing load initially crashed Silke’s laptop, but after exporting the assemblage as a single file on another computer, we could finally watch all the channels concurrently. The hack juxtaposed the different panels into a shared whole while keeping explicit the assembled nature of this combination. The composite video was a bit clunky; however, due to the heavy technical processing requirements, Silke’s achievement seemed miraculous. We anticipated that viewing through VR goggles may amplify the experience of incoherence, but also had to consider the corporeal similarity to wearing a snorkel and scuba mask, the possibility of feeling seasick or nauseous, and worse, a case of aquaphobia.[7] Furthermore, the cacophony of all the sound channels playing simultaneously added to the feeling of entangled multiplicity.[8] While the final result looked disorientating and fragmented, the resulting film experience resonated in unexpected ways with Annet’s ethnographic insights, showing the potential to imagine different kinds of immersive applications for 360° video.
Amphibious movements
The video editing intervention—the act of hacking 360° video technology to produce something unexpected—stimulated Annet to reflect on how her fieldwork footage related to the methodological challenge of doing immersive ethnographic research. The states of confusion produced by viewing partially connected and simultaneously unfolding storylines somehow resemble the ethnographer’s experience of disorientation while immersed in unfamiliar worlds and their inherent multiplicities. To Annet, one of the most challenging aspects of doing fieldwork in the Makassar Strait in 2011–2013 was the continuous consideration of where to go, what to focus on, and how to listen in order to weave together an enhanced and comprehensive understanding of how and why marine conservation is contested. Inevitably, there are multiple events happening simultaneously, and how we position and focus ourselves in relation to them may have a considerable effect on how we draw our ethnographic pictures. The practice of producing fieldwork material (e.g. notes, transcripts, photos, video) is situated, bringing forth an assemblage of richly detailed but partial accounts of the reality/-ies under study. Still, ethnographic accounts often (implicitly) suggest a coherent world in which a linear sequence of events is taking place. While ethnographic accounts usually embrace and build on situated knowledge practices (Haraway, 1988), scientific conventions of objectivity and coherent overviews often still linger in methodological expectations (cf. Law, 2004).
As Annet carried out her own study of different perspectives on the sea, she encountered different notions of what the sea, nature, or coral is and how humans relate to it. Following the work and narratives of ecologists gave her access to a way of understanding the sea that was incongruent with the spirit-world of sea people. While unraveling conflicts about marine conservation plans, Annet came across incompatible philosophies of what the sea—or the coral reef—is, how it is known, and how humans relate to it (Pauwelussen and Verschoor, 2017). Similar to the different narrative maps in the mapping exercise, these philosophies did not neatly ‘map’ onto one another. Attempting to create a coherent story about one relegated the other to the background. The sea appeared ontologically multiple, or incoherent, yet people also took it as self-evident that they all moved in and through the same sea. Sea life and life at sea took place in one world and, simultaneously, in different worlds.
At the intersection of anthropology and STS (Science and Technology Studies), an assemblage of thought has emerged around the description and conceptualization of ontological complexity, or multiplicity (e.g. Blaser, 2014; Bonelli, 2015; Haraway, 2016; Law and Mol, 2002). These studies invite one to think beyond perspectivism—the idea that there is one world, seen from different perspectives—to embrace the possibility that the world, or reality itself, is multiple, as there are continuously different worlds in the making. Still, such worlds are not clearly separated. In practice, they partially connect (Strathern, 1991) as world-making practices intermingle and interact. As with the narrative maps, in which places cannot be reduced to each other if arrived at through different storylines, they are also not distinct in any clear-cut way. There is movement in-between, by people, things, concepts, and currents, just as Annet—as an ethnographer—tried to translate between them.
Inspired by the work of STS scholars Jensen (2017) and Morita (2015), Annet has conceptualized this movement in-between as ‘amphibiousness’ (Pauwelussen, 2017; Pauwelussen and Verschoor, 2017). Such amphibiousness can also be used to conceptualize the immersive methodology of the researcher or fieldworker who moves between worlds, bodily and cognitively, to develop a sensitivity to them. Such immersion with incongruent yet intersecting worlds may very well bring about an experience of confusion. Yet, this confusion, the ability to allow for incoherence and ‘staying with the trouble’ (Haraway, 2016) without pre-imposing the researcher’s own logic, is a vital part of ethnography as a translation practice (cf. Pauwelussen, 2017, c2; Verran, 2001).
What if reality is made of a multiplicity of worlds and logics? What if uncontainable messiness and excess is the default nature of our ethnographic fields? How then to do a proper translation that stays true to the multiplicity of the worlds we experience in ethnography? Our 360° video stimulated this methodological reflection, juxtaposing multiple simultaneous perspectives and making explicit how the immersed experience is actually stitched together from realities that, whilst not wholly distinct, cannot be reduced to one. Just as the viewer can only focus on one or two channels at a time, the ethnographer likewise focuses attention while remaining nevertheless aware that other scenes and narratives are playing.
Kaleidoscopic vision
Melody Jue has shown how “photomosaic epistemology and aesthetics” already characterize knowledge-making practices in contemporary oceanography (Jue, 2021). Whereas scientific methodologies like the photomosaic attempt to assemble a coherent composite out of individual pieces of data, our mosaicking efforts intentionally disrupted realistic patterns of sealife to emphasize a more fragmentary understanding of the ocean. The narrative map, drawn with fishers and seafarers, stitches together not only different pieces of paper but also different spatial narratives.[9] As a mosaicking effort of sorts, in which different people draw on an expanding piece of paper at different moments, the resulting map visualizes storylines together on one surface while also leading the map-reader into different spatial directions, into different time frames, rupturing the logics and coherence of cartography. In line with this, the hacking of 360° video using Annet’s fieldwork footage involved using disruptive techniques of montage and then reassembling the pieces into a single mosaic, thereby challenging the coherence of the spherical image. The resulting multiplicity, in both the map and the video, resonated with Annet’s thinking about an immersive, amphibious anthropology (and methodology) in a world that is ontologically multiple. We have shown how the creation of a spherical montage, particularly in the enclosed environment of VR goggles, forces the viewer to make decisions on what to watch, while nevertheless being aware of other images adjoining along the frame of each panel and possibly sounds emanating from elsewhere.
It is important to note that even in the fragmented composition of the video, a gravitational orientation continues to impose a realist proprioception. Despite the freedom of the omni-scopic perspective, 360° technology does not really challenge the privileged orientation of the horizon. Indeed, for humans, looking away from the horizon line with VR goggles can cause gravitational disorientation and potentially “seasickness.” Furthermore, the privileged perspective afforded the VR goggles, which allow the sensorimotor contingencies of head movement, thus enhancing the feeling of immersion, is based on forward-facing eye morphology. In other words, whereas some animals of prey can see nearly 360˚, humans and other stereoscopic predators cannot see an entire spherical image all at once. Simply put, 360° vision is inherently not human vision. Whether moving with goggles or dragging the cursor across a screen, the normative vision maintains a forward-facing rectangular selection, a movable window that renders most of the sphere out of view.
When confused and disoriented, a regular thing to do is to ‘zoom out’, to try to ‘see the big picture’ and retrieve solid ground, but zooming out of a spherical image does not necessarily offer a coherent “wide angle,” just like the stitched-together line map does not provide a cartographically coherent overview “from above.” Although not possible with goggles, computer and tablet 360˚ media browsers with flat screens enable viewers to zoom out, or rather squeeze the sphere into a flat, albeit distorted, whole. In this uniquely distanced perspective, in which the sphere becomes flattened, one can still drag the image with the cursor, rendering the horizon a supple line that moves according to the image’s orientation. The ability to see a complete 360˚ image from radically different perspectives offers a kind of Möbius strip effect, in which the most sublime perspectives of the recorded landscapes often come by looking along the zenith/nadir vector—straight up to the sky or straight down to the ground. Among 360˚ video aficionados, these perspectives are known as the “hamster wheel” and “tiny planet” effects. Whereas the tiny planet requires looking straight down at the ground, which produces a small inner earthen sphere encircled by an outer ring of sky, the hamster wheel refers to people’s movement around an outer ring with a hollow sky in the center.
When fully zoomed out on the zenith/nadir vector, the six-panel juxtaposition of Annet’s materials of the Makassar Strait now takes on a new, multifaceted perspective. The “tiny planet” effect wraps the different image layers into concentric circles with a well of water in the center, encircled by the different horizon perspectives, and then surrounded by the perimeter of the sky. The movement of clouds and waves in different concentric circles creates a double swirling effect, while four perspectives of the horizon play simultaneously in an in-between ring. In contrast to the scopic regime of the telescope, zooming out in our 360° video mashup creates a kaleidoscopic view of constantly moving patterns. Instead of solid ground, the background of sky or sea becomes liquid and moving. When pulling back into a single image, the viewer can attempt to re-center the composition of the video-assemblage, but they cannot escape the multiplicity of this immersion. As we move in this kaleidoscopic environment, we may also become reflective of our own positionality and how it affects the composition of assembled fragments we see moving in front of us.
A kaleidoscope is best known as a toy or instrument of imagination that is usually shaped like a telescope.[10] The telescope as a technology of authority, with its roots in seafaring exploration and colonialism, resonates with a mainstream anthropological gaze that makes distant objects seem nearer by magnifying the small, remote, and obscure. Whereas a telescope uses a stacked series of lenses to aid the human eye’s perceptual limitations and enable seeing distant objects on the horizon (or celestial bodies in the heavens), the kaleidoscope has a different optical logic premised on intentionally placing obstructions between its stack of lenses, assembled from fragments of colored glass and/or other materials, thus confronting the human eye up close with a dazzling sample of colorful objects and unexpected fractal-like patterns. Rather than the mastery of the telescope to make distant objects seem nearer, the busyness of all these moving patterns has a kaleidoscopic effect of juxtaposing shots, movement, color, and texture in unexpected patterns—the result of a multimodal assemblage of technologies. In a more general sense, kaleidoscope refers to a constantly changing pattern or sequence of elements. The kaleidoscopic vision produced by hacking 360° video thus facilitates a conceptual attunement that ideally makes the viewer of our work more aware of the parallel patterns of activity in the discrete panels. In other words, this fragmented mosaic pattern serves as a model for dynamic perception and imaginative thinking about the diversity and precarity of these seascapes.
Science and Technology Scholar John Law has argued that our social science research methods are designed to create pockets of coherence and clarity out of the above-mentioned messy multiplicity (Law 2004). In the process, we distort reality into clarity. There is nothing ‘wrong’ with this; we all need order to act in the world, and to make sense of it. Still, it is important to remind ourselves of the fact that such order and coherence are contingent, an effect of our ordering practices—both cognitive and affective—rather than the building blocks of reality. This argument contains an ontological and methodological inversion: definitions, concepts, framing, logics, and established ways of seeing are not just tools to explain reality; they are the very ordering processes that we need to study and critically reflect upon, if only to open up to ways of doing, perceiving, and thinking differently, and perhaps to experiment with that.
While the counterintuitive affordances of the various distorted visions outlined above may be more abstract than representational, Wark reminds us, “abstraction is what every hack produces and affirms. To abstract is to construct a plane upon which otherwise different and unrelated matters may be brought into many possible relations“ (2004 [008]). The hacking of these immersive technologies thus enabled us to find multimodal synergy between our collective conceptual, methodological, technical, and aesthetical frameworks. Careful analysis and critical engagement with emerging audiovisual technologies can help ethnographers critically evaluate the scopic regimes that fashion our normative modes of vision. Instead of succumbing to either the continual promise of technology’s ability to produce more realistic visions of the world or the deriding rhetoric of ‘visualism’ that alienates anthropologists from ways of seeing, we should cultivate the unexpected affordances and unique perspectives beyond the assumptions of observational realism. Our kaleidoscopic video is one way to challenge these scopic regimes and speculate in embodied ways on the entangled multiplicity of ethnographic immersion. In a kaleidoscopic anthropology there is no solid ground to retrieve, no position that feigns completeness, no claim to immersive realism; there is only a constellation of shifting perspectives.
Acknowledgements
We are grateful to the communities in Masalima and Berau and scientists at the Center for Ocean, Coastal, and Small Island Research and Development of the Hasanuddin University in Makassar for their trust, support and guidance during Annet’s fieldwork. A special thanks goes to fieldwork assistant and friend Kadir, who facilitated the mapping and the filming with fishers and gleaners. We would like to thank the Center for Innovation at Leiden University for their support, in particular Thomas Ginn, Thomas Hurkxkens, and Leontine van Melle for their inspiration, questions, and contributions. We are grateful for the insightful and constructive feedback given by Veerle Boekestijn, Melody Jue, and Kate Maxwell on earlier versions of this paper. Annet’s fieldwork was supported by a research grant from the Wageningen School of Social Sciences.
Notes
[1] Mark and Annet collaborated on this article with mutual input. While Annet provided the content and the underlying theoretical basis of amphibious anthropology, Mark initiated the idea of assembling the materials into a 360° video and drove the theoretical ideas around hacking spherical images. Silke served as the lead editor and contributed her reflections on the process.
[2] The fieldwork research was carried out over a total of 20 months in 2011–2013, during which Annet lived and traveled with coastal communities as well as scientists and NGO staff. Annet has been open about the purpose of her study to the people involved and all interviews, mapping and filming were carried out with prior informed consent. The footage of Bajau fishers and gleaners, as well as the map, was also shown and discussed with the people involved and their families on different occasions. For a more elaborative discussion and justification of methodology and fieldwork ethics see Pauwelussen 2017, chapter 1 and 2.
[3] Also notable is work by Rossella Schillaci, Ezekiel Morgan, Juliet Brown, Iza Kavedzija, Rob Eagle, Josepha Wessels, Zul Tinarbuko, and Adonis Durado who all presented work at the 2021 RAI Film Festival and Conference, 25-8 March 2021.
[4] Mark worked closely with his former student, Silke, to develop a series of video modules that explored the potential of 360° video to convey ethnographic insights to anthropology students. Annet contributed her own unedited research materials from the Makassar Strait in Indonesia about dynamic land-sea environments as a form of amphibious anthropology. Silke executed editing solutions that combined Annet’s ethnographic research captured on “flat” video with Mark’s interest in hacking 360° video.
[5] We presented this work multiple times at the culmination of a first-year course on multimodal anthropology. Unfortunately, elaboration of the student responses is beyond the scope of this article.
[6] This is also the case whether viewing on a flatscreen or with VR goggles.
[7] During our pilot usage of this module, we became aware of an under-considered “trigger warning” as one student asked to be excused due to severe aquaphobia. The anxieties about motion sickness are pervasive in the literature with common camera-handling recommendations for how to avoid this. See details about “virtual reality sickness” at this site: http://www.vrglossary.org/glossary/motion-sickness/
[8] While 360° video editors have begun to design the spherical soundscapes to direct viewers’ attention in particular directions, thereby reintroducing narrative authorship in spaces that remain visually open-ended, we did not want to prioritize a single frame over any other. Although a bit crude, we found the jumbled cacophony of all the audio tracks playing together consistent with our larger argument. Furthermore, users report a gradual and perpetual process of deciphering which audio corresponds with which image.
[9] Of course, the resulting map did not so much represent the fishers’ spatial understanding, which is embedded in an oral knowledge-sharing tradition, but rather the messy and experimental process of (an ethnographer) trying to translate between situated knowledge practices and conventional cartographic visualization.
[10] Apple dictionary ‘telescope’: an optical instrument designed to make distant objects appear nearer, containing an arrangement of lenses, or of curved mirrors and lenses, by which rays of light are collected and focused and the resulting image magnified [make appear larger]. Etymology: tele- ‘at a distance’ + -scopium ‘look at’.
References
Amit, V. (2000) Introduction: Constructing the Field. In: Amit, V. (ed.). Constructing the Field: Ethnographic Fieldwork in the Contemporary World, London: Routledge. pp. 1–18.
Blaser, M. (2014) Ontology and indigeneity: on the political ontology of heterogeneous assemblages, Cultural Geographies, 21(1), pp. 49-58.
Bonelli, C. (2015) To see that which cannot be seen: ontological difference and public health policies in Southern Chile, Journal of the Royal Anthropological Institute, 21(4), pp. 872–891.
Crary, J. (1990) Techniques of the Observer. Cambridge, MA: MIT Press.
Favero, P. S. H. (2018) The Present Image: Visible Stories in a Digital Habitat. Cham: Palgrave Macmillan.
Geertz, C. (1973) The Interpretation of Cultures: Selected Essays. New York: Basic Books.
George, R. A. and Jones, M. O. (1980) People Studying People: The Human Element in Fieldwork. Berkeley: University of California Press.
Glaser, M., Breckwoldt, A., Deswandi, R., Radjawali, I., Baitoningsih, W., and Ferse, S. C. A. (2015) Of Exploited Reefs and Fiseres – A Holistic View on Participatory Coastal and Marine Management in an Indonesian Archipelago, Ocean & Coastal Management, 116, pp. 193–213.
Gregory, S. (2016) Immersive Witnessing: From Empathy and Outrage to Action. Retrieved January 27, 2019, from https://blog.witness.org/2016/08/immersive-witnessing-from-empathy-and-outrage-to-action/
Gómez Cruz, E. (2017) Immersive Reflexivity: Using 360° Cameras in Ethnographic Fieldwork. In: Gómez Cruz, E., Sumartojo, S. and Pink, S. (eds.). Refiguring Techniques in Digital Visual Research. Cham: Springer International Publishing. pp. 25–38. http://link.springer.com/10.1007/978-3-319-61222-5_3, accessed October 28, 2018.
Gunawan, B. I. and Visser, L. E. (2012) Permeable Boundaries: Outsiders and Access to Fishing Grounds in the Berau Marine Protected Area, Anthropological Forum, 22(2), pp. 187–207.
Haraway, D. (1988) Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective, Feminist Studies, 14(3), pp. 575–599.
Haraway, D. (2016) Staying with the Trouble: Making Kin in the Chthulucene. Durham: Duke University Press.
Helmreich, S. (2007) An Anthropologist Underwater: Immersive Soundscapes, Submarine Cyborgs, and Transductive Ethnography, American Ethnologist, 34(4), pp. 621–641.
Hoeksema, B. W. (2007) Delineation of the Indo-Malayan centre of maximum marine biodiversity: the Coral Triangle. In: Renema, W. (ed.). Biogeography, time, and place: distributions, barriers, and islands. Dordrecht: Springer. pp. 117–178.
Jensen, C. B. (2017) Amphibious Worlds: Environments, Infrastructures, Ontologies, Engaging Science, Technology, and Society, 3, pp. 224–234.
Jue, M. (2020) Introduction: Thinking through Seawater. In: Melody, J (ed.). Wild Blue Media: Thinking Through Seawater. Durham: Duke University Press. pp. 1–33.
Jue, M. (2021) ‘Pixels May Lose Kelp Canopy’: The Photomosaic as Epistemic Figure for the Satellite Mapping and Modeling of Seaweeds, Media+Environment, 3(2).
Kusumawati, R. and Visser, L. E. (2014) Collaboration or Contention? Decentralized Marine Governance in Berau, Anthropological Forum, 24(1), pp. 21–46.
Law, J. and Mol, A. (2002) Complexities: An Introduction. In: Law, J. and Mol, A. (eds.). Complexities: Social Studies in Knowledge Practices. Durham: Duke University Press. pp. 1–22.
Law, J. (2004) After Method: Mess in Social Science Research. New York: Routledge.
Merchant, S. (2011) Negotiating Underwater Space: The Sensorium, the Body and the Practice of Scuba-Diving, Tourism Studies 13(3), pp. 214–234.
Milk, C. (2015) How virtual reality can create the ultimate empathy machine. Retrieved from https://www.ted.com/talks/chris_milk_how_virtual_reality_can_create_the_ultimate_empathy_machine?language=en
Morita, A. (2015) Infrastructuring the Amphibious Space: The Interplay of Aquatic and Terrestrial Infrastructures in the Chao Phraya Delta in Thailand, Science as Culture, 25(1), pp. 117–140.
Murch, W. (2001) In the Blink of an Eye: A Perspective on Film Editing (2nd ed.). Los Angeles: Silman-James Press.
Nash, K. (2018) Virtual reality witness: exploring the ethics of mediated presence, Studies in Documentary Film, 12(2), pp. 119–131.
Pálsson, G. (1994) Enskilment at Sea, Man 29(4), pp. 901–927.
Pauwelussen, A. P. (2015) The Moves of a Bajau Middlewoman: Understanding the Disparity between Trade networks and Marine Conservation, Anthropological Forum, 25(4), pp. 329–349.
Pauwelussen, A. P. (2017) Amphibious Anthropology: Engaging with Maritime Worlds in Indonesia. PhD Thesis, Wageningen University.
Pauwelussen, A. P. (2021) Leaky Bodies: Masculinity and Risk in the Practice of Dive Fishing in Indonesia, Gender, Place & Culture. DOI: 10.1080/0966369X.2021.1950642.
Pauwelussen, A. P. and Verschoor, G. M. (2017) Amphibious Encounters: Coral and People in Conservation Outreach in Indonesia, Engaging Science, Technology and Society, 3, pp. 292–314.
Strathern, M. (1991) Partial Connections. Savage, Maryland: Rowman & Littlefield.
Strathern, M. (1999) Property, Substance and Effect. Anthropological Essays on Persons and Things. London: The Athlone Press.
Suhr, C. and Willerslev, R. (2012) Can Film Show the Invisible? The Work of Montage in Ethnographic Filmmaking, Current Anthropology, 53(3), pp. 282–301.
Suhr, C. and Willerslev, R. (2013) Transcultural Montage. New York: Berghahn Books.
Tsing, A. L. (2015) The Mushroom at the End of the World. Princeton and Oxfordshire: Princeton University Press.
Veron, J. E. N., Devantier, L. M., Turak, E., Green, A. L., Kininmonth, S., Stafford-Smith, M. and Peterson, N. (2009) Delineating the Coral Triangle, Galaxea, Journal of Coral Reef Studies, 11(2), pp. 91–100.
Verran, H. (2001) Science and an African Logic. Chicago: University of Chicago Press.
Viveiros de Castro, E. (2004) Perspectival Anthropology and the Method of Controlled Equivocation, Tipití, 2(1), pp. 3–22.
Wark, M. (2004) A Hacker Manifesto. Cambridge, MA: Harvard University Press.
Westmoreland, M. R. (2020) 360° Video. In: Vannini, P. (ed.). Handbook of Ethnographic Film and Video. Abingdon, Oxon ; New York, NY: Routledge. pp. 256–66.
Author bio
Mark R. Westmoreland is Associate Professor of Visual Anthropology in the Institute of Cultural Anthropology and Development Sociology at Leiden University where he coordinates the Visual Ethnography specialisation. Westmoreland formerly served as co-editor of Visual Anthropology Review before co-founding the Writing with Light journal for anthropological photo-essays. His work engages both scholarly and practice-based approaches at the intersection between art, ethnography, and politics. He has written extensively on the interface between sensory embodiment and media aesthetics in on-going legacies of contentious politics. In Lebanon, he addresses the crucial role experimental documentary practices play in addressing recurrent political violence, while in Egypt he has focused on the activist mode of resistance-by-recording in mass street protests.
ORCiD: 0000-0002-5669-8781
Facebook: https://www.facebook.com/mark.westmoreland.587
Website: https://www.universiteitleiden.nl/en/staffmembers/mark-westmoreland
Annet Pauwelussen is assistant professor in marine governance with the Environmental Policy Group at Wageningen University (NL), and a research fellow with the Ocean Nexus Center of the University of Washington. With a background in anthropology, her work combines elements of science and technology studies, political ecology, multispecies ethnography and feminist theory to explore matters of equity and environmental justice in marine conservation and restoration practices. Her amphibious anthropology experiments with inclusive approaches and methodologies that are responsive to diversity and dialogue across epistemological and ontological difference. Focusing on Southeast Asia and Europe, she has published and lectured on human-sea relations, gender, diving, illegal fishing and trade networks, aquaculture, marine protected areas, and mobile methods.
ORCiD: 0000-0002-5669-8781
Facebook: https://www.facebook.com/annet.pauwelussen
Website: https://www.wur.nl/nl/personen/annet-dr.ir.-ap-annet-pauwelussen.htm
Silke van Diemen is educated as a cultural and visual anthropologist and is specialized in ethnographic filmmaking. After working as a teacher and research assistant for the visual ethnography department at Leiden University, she now works as a researcher for non-fiction (film)projects and as an independent filmmaker. She approaches her work from the notion that no matter how different our lifestyles may seem, in the end there is more that connects us as human beings than what differentiates us. For her the challenge is to find those universal human connections underneath our cultural, social and religious differences.
Website: www.silkevandiemen.com
Past issues 