Posted on Tue 19 Sep 2017
Virtually Useful: Step Two - What is VR/AR/MR/360?
Second in a series of short reads, Studio Managing Producer Verity McIntosh attempts to demystify some virtual reality terminology.
There are a few similar sounding terms and acronyms to negotiate when it comes to talking about VR (there's one for starters). In this chapter I will attempt to make sense of what is meant by some of the key terms; VR, AR, MR and 360 video.
Virtual reality (VR) is sometimes used as an umbrella term to mean ‘immersive experiences’, however more specifically, and for the purposes of this blog, the term refers to content experienced in specialist headsets (Head Mounted Displays – HMDs).
The current generation of headsets make use of advances in sensor technology such as gyroscopes and accelerometers found in most phones, giving the wearer the ability to look around their environment. Improvements in computer processing speeds have enabled there to be an incredibly low latency (delay) between movement in real life and in the virtual world. This convinces your brain that the things that you see are solid, and located in a real place, contributing to a powerful sense of presence, immediacy, and perhaps immersion. The feeling that you have been transported somewhere else.
People often recount their experiences in VR in terms of things that they did, and things that happened to them, not things that they saw as they might with film or TV. A Studio resident, Jane Gauntlett recently shared with me that when she was running a VR experience in a cafe, a group of teenage boys kept running in and out, jeering and being disruptive. Being the lovely human that she is, Jane invited the boys in to experience her piece, In My Shoes: Dancing with Myself. One of the first things that you to do in this piece, is look down at the pair of hands on the table, Jane’s hands. The teenager mimicked the gesture, putting his hands on the table and said “Wow, my nails are fucking beautiful”. Not those nails, not her nails. My nails. This happens time and again.
In My Shoes: Dancing With Myself by Jane Gauntlett
I was recently trying to describe the moment when I spotted a disturbance off in the distance whilst playing a space adventure game called Lone Echo. I gestured wildly to my other half - 'you know, that space-time-rift-y thing, it was over there'. I was pointing furiously into the middle distance and a little to the left of the train station we were passing as we walked home. The sense of depth and distance I knew we had both experienced in the game had been so physical to me, it seemed perfectly reasonable to describe it as a place in real world terms, as though referencing somewhere we had both been on holiday.
360 Video Increasingly, practitioners are drawing a clear distinction between Virtual reality and 360 video. The latter usually refers to live action film, recorded with specialist 360 cameras like Nokia Ozo or Samsung’s Gear 360, or an array of cameras, e.g. six Go Pro’s in a cube formation to capture all of the surrounding action. 360 film can be streamed live if you have the processor power and bandwidth, but content is more often stitched together later in dedicated software such as Kolor, as part of the post production process.
One of the main affordances to consider with this form, is that the only place that your viewer can stand/sit/float, is the spot where you placed the camera(s). Your participant can look all around them, and perhaps teleport from one camera position to another, but any off-axis motion, even the craning their neck forward to have a closer look, will result in the whole world appearing to move with them, as though they are in the centre of a world which is glued to their head (which in some ways they are). This can quickly break the illusion of presence and can make some people feel nauseous.
360 video is well suited to experiences that might naturally be encountered in a seated, relatively passive environment such as sports spectatorship, proscenium arch theatre, a roller coaster, a park bench, a restaurant, riding a bike, squatting in a trench, lying in a grave, lolling in the back of a gondola perhaps. The opportunities are many and varied, but it’s worth noting that this form is less satisfying for any environment your viewer might naturally expect to move around and through – wide open spaces, mazes, corridors, cityscapes etc.
To create more navigable worlds, the solution for now at least, is to use programmes such as Unity or Unreal to create computer generated worlds, as games designers have done for many years. With these tools you can design every aspect of the world that the user may encounter from all angles, with dynamic light, shade, textures, spatially located sound etc to create a rich and textured world.
A note on the kit: Mobile VR devices - headsets that you put a phone into such as Google Cardboard and the Samsung Gear VR, as opposed to those that have a built in screen - use what is known as 'rotational tracking'. Using the on-board sensors, the phones interpret the rotational movement of the wearer’s head as they look around, up and down which, as you can imagine, works well for 360 video. The more expensive headsets such as Oculus Rift and the HTC Vive come with external sensors that work in harmony with data gathered by the headset itself to track the user as they move around a space. This is known as 'positional tracking' and allows for six axis or ‘degrees of freedom’ (DOF), to be translated into the virtual world. The largest space that HTC suggest that you can track is 5m x 5m. It is slightly less for Oculus systems as they use different tracking technology.
A note on what is next: In the future, the affordances of computer generated VR and 360 video are likely to combine, offering photo-realistic, navigable worlds. This is likely to be enabled by the development of ‘light field’ technology, a means by which to record depth in live-action video. Some light field tech exists now, with pioneers such as Lytro and otoy making steps towards making this a reality, however the cost and computational power required to usefully capture and process this amount of data, puts it beyond the reach of most outside of the lab for now.
360 is proving an attractive new frontier for filmmakers and documentary makers who are used to working in a form where audiences expect to be routed to one spot, and where the visually surrounding nature of the content offers an expanded and engrossing new opportunity for storytelling. It can also be reasonably quick and low cost to create, with a better chance of finding an audience as mobile VR headsets continue to outsell their more expensive cousins. For some however, this is not really VR at all, and there is some concern that this more limited experience, with its potential to cause sickness will undermine the potential of full VR, and erode public confidence in the form.
Augmented reality (AR) Whilst VR effectively works as a blindfold to the world around you, placing you in the centre of a virtual world, augmented reality seeks to overlay virtual objects over and into your view of the real world. This may be achieved using a viewing pane such as a tablet or phone, overlaying images or animations onto the camera view.
Pokémon Go was a breakthrough AR app in 2016, indicating public appetite for more of this sort of thing. Apple and Google’s next major phone operating system updates both contain software specifically designed to make it much easier for developers to integrate AR elements into their apps (called ARKit and AR Core respectively) so this style of AR is likely to become much more familiar in the coming year.
The term AR is also used to refer to another type of head mounted display, one that is more akin to a pair of glasses that allows you to see the world around you, and introduce digital elements directly into your line of sight. Early entrants to mode of AR include the divisive Google Glass, and a number of 'heads up display' models designed for things like off-piste skiing. Recently MotoGP team Aprilia announced that their whole pit crew would start to wear AR helmets during races to give them quick access to thermal imaging and bike data without needing to split attention between rider and screens.
Mixed reality (MR) There is some discussion about whether AR and MR are ostensibly the same thing, and the terms are often used interchangeably. As far as I can tell, the main distinguishing factor is that mixed, merged, or hybrid reality, tends to be used when the digital objects are anchored or tethered to a particular real world element, giving the impression that they are physically present in the surrounding environment.
Recent advances in machine vision (a computers’ ability to scan an image and interpret what it contains) and contextual markers (markers that can be placed in the real world like stickers and recognised by a computer) allow content creators to anchor digital media to real world items.
At this year’s VR World Congress in Bristol we watched Mike Taulty and Peter Daukintis from Microsoft HoloLens place a pirouetting ballerina into the aisle in the midst of the conference audience. We could see this from Mike’s point of view through the HoloLens he was wearing, via a live link from the HoloLens to the screen on stage.
— Verity McIntosh (@veritymcintosh) April 12, 2017
There has been a lot of hype about an as yet unreleased MR technology Magic Leap which is said to offer an extraordinarily compelling and realistic mixed reality experience. However, those who have tried it have been under strict NDAs for years, and recent leaks seem to suggest that they are still some distance away from consumer release.
So the question remains, what form do you use when designing a new creative piece? In a nutshell this probably comes down to your own enthusiasms, your budget, and whether you want to reach audiences now or in the future. 360 video and mobile VR are by far the more pervasive technologies right now, and although distribution and exhibition channels are still in their infancy, there is a good chance your work will find its way to your audience.
In five to ten years it is likely that mixed reality will supersede all of the above so you may prefer to dedicate your energy to understanding, exploring and unlocking this emerging form, putting yourself in a strong position to lead the field when it actually becomes a thing. The downside of course, is that in the interim the technology can be expensive, constantly changing, and in many ways – especially as far as an audience is concerned – it doesn’t yet exist! Even the Microsoft Hololens is only currently available as a developer kit, the consumer version is yet to be announced. VR headsets such as Oculus Rift and HTC Vive are getting a little closer to the mainstream, and can be great for visitor experiences and installation pieces, but are still extremely rare in the home. We would love to hear what you are currently choosing to make and why via twitter @PMStudioUK, or if you’re near Bristol pop in and tell us at one of our Open Studio Friday’s.