We're introducing a new series based on one of the blog topics our readers overwhelmingly wanted: The Science Behind AR Hardware. While we briefly explained some of the technical capabilities of AR headsets in the first of our AR Anti-patterns series, this new series goes in-depth to show how headset components work and why headsets have been designed the way they've been (hint: it's because today's technology, which was not designed for AR and VR in mind, is being used to make new technology).
In this first part, we take a look at display and ocular technology design in today's headsets. As you read through, you'll understand why large fields of view (FOV) are crucial to good AR experiences, the three categories of ocularity all headsets fall under, and how headsets display holograms. With the wide variety of display and imaging technologies, you'll see why there won't be "one headset to rule them all" anytime soon (if at all). Given that this is a longform article, feel free to use the following jumplinks to get to the sections you want to read first: ocularity categories, display types, and FOV.
Key Takeaways from The Science Behind AR Hardware, Part 1
→ Considering that AR headsets need to be human-centered devices, headset manufacturers have to consider the tradeoffs between the quality of the AR experience vs. the cost, size, and technological capabilities of various optical elements that could go into a headset.
→ High quality AR headsets (read: the ones that provide a highly immersive experience and crisp holograms) are large and expensive because of the constraints of current optical display technology (think back to how the first computers, e.g., ENIAC, took up entire rooms in the 1940s and 1950s). As optical technology evolves, smaller and cheaper form factors will become the norm.
→ Large FOV (greater than 30°) is key to more immersive, higher quality AR experiences. Put another way, seeing holographic info with small FOV is like looking at things through a small porthole: you have to constantly move your head (even your entire body at times) in order to see everything you want to see.
If you take a look at the AR and VR industry today, you'd be amazed to see all the headsets that have hit the market in the past few years. And if you think about it, it's a remarkable feat for those companies to build this new technology by adapting and using existing technology – and it's way harder than you think. But, it's astounding to see so many companies being committed to advancing the industry as a whole. Just take a look at how large and expensive current-gen AR headsets are and you'll start to see why they mostly tend to look large and not-so-user-friendly.
That question is answered based on headsets' ocularity, or how they show images to our eyes. Headsets are categorized as being monocular, bi-ocular, or binocular. Each category progressively has more features, and subsequently becomes more expensive, than the previous category:
This table shows the pros and cons of each headset ocularity type.
A more visual representation of the ocularity types can be seen in a 1998 study comparing the efficacy between the three types:
NASA research scientist Stephen Ellis and Lockheed-Martin aerospace engineer Brian Menges looked at how well people could see virtual objects up close.
Monocular Headsets: Images Shown Only to One Eye
Monocular headsets present images to only one eye, with the other eye being free to focus on other tasks and the surrounding environment. Monocular headsets, like the Google Glass, tend to be small and lightweight, and thus are great at projecting digital information and images onto real-world environments (akin to wearing a heads-up display aka HUD). However, that is the extent of their AR capabilities because monocular headsets have no FOV and lack stereo depth cues (such peripheral cues are needed in order to display holograms). More optical components are needed in order for monocular headsets to provide highly immersive AR experiences. But at that point, why not check out bi-ocular and or binocular headsets?
Bi-ocular Headsets: Same Image Shown to Both Eyes
Compared to their monocular counterparts, bi-ocular headsets are larger because more components and larger lenses are needed to present the same image to both eyes. Bi-ocular headsets tend to be used for hands-on training and practice, e.g., teaching people how to assemble things on a factory floor, because they don't have as many technological limitations as monocular headsets (bi-ocular headsets have small FOV and can present some peripheral cues). Despite being a better option than monocular headsets (but still lacking stereo depth cues), the current sentiment within the AR / VR industry leans towards building headsets that are either monocular (such as the Google Glass) or binocular, e.g., the Meta 2 Development Kit.
Binocular Headsets: Different Images Shown to Each Eye
When you think of AR headsets, the kinds that most often come to mind are the ones that display rich, interactive 3D holograms:
These kinds of headsets are called binocular headsets because they show different images to each eye from individual displays, just like the way our eyes work together to aggregate a single image (each eye sees slightly different images), perceive depth, and see things in 3D. Similarly, most VR headsets, such as the Vive and Rift, are binocular headsets.
From a technological perspective, binocular AR headsets are more technologically advanced, larger (because they include bigger lenses and optical systems that allow for wider FOV and more depth cues), and altogether more complex than their monocular and bi-ocular counterparts. Because they include larger lenses and optical systems that allow for wider FOV and more depth cues, binocular AR headsets are able to provide highly immersive AR experiences. And by virtue of being feature-rich, they cost a lot to build, have bigger components, and are the most expensive type of headset on the market right now.
Now that we've covered the different types of AR headsets, we're going to take a closer look at display technology and how it's the crux of what allows AR to happen.
In AR, both the real and virtual environments are combined and shown through a lens, which is called a see-through display. This display is based on the concept of ocularity, and comes in two types: video see-through and optical see-through. Both types are used for different purposes, with optical see-through displays currently being the preferred display type among AR headset manufacturers.
Video see-through Displays
At a high level, video see-through displays use video cameras to capture and record real-world environments. The real-world images are then combined with computer-generated (virtual) images, with the computer-generated images layered on top of the real-world images. The combined real and virtual images are then displayed on a screen.
A diagram showing how a video see-through display works (Vallino)
Companies like Volkswagen have turned iPads into video see-through AR displays that help their technicians understand and troubleshoot different parts of cars:
Volkswagen's MARTA iPad app in action (PSFK)
Optical see-through Displays
Whereas video see-through displays use electronics to blend the real and virtual world together, optical see-through displays use something low tech, but still just as effective: semi-reflective and semi-transparent optical elements, such as mirrors. The semi-transparent mirrors allow enough light from the real-world environment to pass through so that users can directly see the real world. Computer-generated images are then shown near the mirror (usually to the side of or over the mirror), allowing for the virtual images to be reflected in the mirror and subsequently projected onto the real-world environment.
A diagram showing how an optical see-through display works (Vallino)
For example, the Meta 2 Development Kit uses an optical see-through display:
Trying on the Meta 2 DK at SOLIDWORKS World. Both the monitor and laptop screen are showing what the user sees through the Meta 2. (Martel)
So now that you have a better understanding of how the display and ocular technologies in AR headsets work, you can see why high-end headsets are large and expensive. Fortunately headset manufacturers are constantly striving to produce smaller headsets with better displays and larger FOV – critical ingredients to making highly immersive and quality AR experiences. This brings us to the last component of AR headsets: field of view.
For AR and VR headsets (and more broadly, optical instruments), FOV describes the amount of information that is visible to both eyes. FOV is expressed in degrees and is one of the key features that people tend to consider, alongside comfort and resolution, when they are evaluating AR and VR headsets. You may be wondering "Why is FOV such a key feature in AR?" and "If VR headsets have average FOV of 100°+, then why haven't AR headsets been able to have larger FOV?".
To put it in perspective, humans' average FOV is 120° measured horizontally and 130° vertically (both measurements are in the same FOV range as today's VR headsets). For most of their 60-year existence, AR headsets have had narrow FOV because of two reasons: 1) the imaging technologies (LCD and OLED panels) in them were not originally designed to be placed into headsets and 2) it takes a lot of computing processing power to render holographic information while simultaneously tracking users' hands and mapping the surrounding environment – something that older processors weren't able to handle until now. It wasn't until the past few years were AR headset manufacturers able to achieve wide FOV (read: > 30°) on par with their VR counterparts – no small feat given that the average FOV among the available current-gen AR headsets stands at 33°.
To paraphrase AR researchers Dieter Schmalstieg and Tobias Hollerer, if you want a highly immersive AR experience where holograms are lifelike and "occupy" physical space like their real-world counterparts (you know, the kind science fiction has been promising us for all these years) then you need as large of a FOV as possible. If this year's Consumer Electronics Show (CES) was any indicator of the trends AR headsets are going in 2017, you'll quickly see that larger FOV has become a major design consideration among various manufacturers.
Meetings of the future will be more interactive thanks to AR (Lucasfilm)
As the technology underpinning AR headsets evolves, components will become smaller and cheaper (meaning smaller and sleeker headsets) and features will become richer – a rising tide that will lift all boats and benefit everyone ranging from headset manufacturers to component vendors to end consumers. In the next installment of The Science Behind AR Hardware, we're going to take a deep dive into imaging technology and examine how LCD, OLED, LCoS, and waveguides work. In the meantime, read this Forbes article for a solid high-level read on how imaging technology works.