Introduction
In mid-2016, much of the world got its first taste of augmented reality (AR) in the form of the wildly popular mobile game Pokemon Go. For weeks, people came together to explore the communities around them in the hopes of catching creatures called Pokemon, filling up once empty parks and other public areas with children, teenagers, and adults alike. Though it may have been the earliest success story involving AR, it will not be the last as this new technology finds its footing and rises as an industry.
Although the public has only seen AR used in games like Pokemon Go, the military has been using the technology for many years in training and to provide better data access to ground troops, improving situational awareness [12]. It has also been utilized in the health field to better educate students and provide them another way to simulate complex operations, as well as to improve surgical accuracy during clinical use [13]. Because it has a purpose in a wide variety of fields – gaming, advertising, technical work, and more – MarketsandMarkets, a market research company, claims that the global AR industry is projected to reach $72.7 billion by 2024, up from the $10 billion it is worth today [11]. As AR takes advantage of new technology and innovations in hardware and software to continue delivering new experiences, the massive jump in investments shows that companies are seeing its value.
What is AR?
Augmented reality (AR) is an up-and-coming technology that is closely related to virtual reality (VR). AR and VR are often mistaken as two interchangeable ideas, but they are fundamentally very different (Figure 1). Virtual reality typically requires users to put on headsets that alter their perceptions of reality. For example, someone using a VR headset could go from sitting in their safe and cozy room at home one second to walking on a tightrope 1000 feet above the ground in the next. Users can become so immersed in this tightrope simulation that a quarter of them won’t even be able to take a single step the first time they try it [1]. While VR transports users to a new environment at the cost of isolating them from the physical world, AR overlays virtual images on top of reality. AR also does not require users to put on headsets, though there are devices that function that way; these headsets, otherwise known as smartglasses, allow users to still see the world around them. The most common way to experience AR would be through a device almost everyone uses every day – a smartphone, making it much more accessible. Virtual content is then superimposed onto a live feed (whether it be your phone’s camera or your smartglasses) and made to seem as if it is a part of reality.
The Tech Behind AR
There are a multitude of different innovations that have contributed to AR as we know it, but the most critical development is computer vision. As humans, we can see an image and easily identify what objects are in the picture. This process of breaking down images into recognizable objects is incredibly difficult for computers to do, and computer vision is the branch of computer science dedicated to solving this problem. Computer vision deals with computers gaining an understanding of images or videos and being able to pick out specific details such as how many people are in an image and what objects are present. It is crucial to AR because to overlay digital content properly onto the screen, the computer needs to answer two questions – what is it looking at and how is it oriented [4]? It needs to determine what object it is looking at to decide what to render into the digital scene. The geometry of the object is also vital in determining how the orientation and position of the digital rendering should be placed (Figure 2). Thanks to advancements in computer vision, AR has been able to progress to where it is today.
Types of AR
Because AR is the combination of the digital world and the physical world, there are many ways to achieve the same goal but with different levels of interaction. The main differences we will discuss are marker-based vs marker-less AR and the different devices we use to experience AR.
Marker-based vs Marker-less
Marker-based AR refers to AR that requires users to scan a physical predefined image to generate a digital rendering on top of it. The most common type of marker is a QR code, but markers can range from barcodes to book covers and even box art. Once a camera scans the marker, the virtual object’s orientation will be determined by that of the marker. For example, if a user were to rotate their phone after the virtual object was created, the virtual object would stay stagnant, allowing users to see different angles of the object. Though marker-based AR is much easier to implement than marker-less AR, virtual models are tethered to the marker. It, therefore, has limited ability to interact with the world and is mainly used for marketing and promotional materials. Examples include books, cereal boxes, and trading cards that display a video or graphic once scanned.
Marker-less AR, as the name implies, does not require there to be a marker. Instead, the camera scans the physical world and creates a 3-D mesh of the environment. This process is called Simultaneous Localization and Mapping (SLAM) [2]. Thus, when a virtual object is placed into the digital scene, it is placed into the 3-D environment itself. This has many more applications, such as allowing users to visualize objects placed in the physical world and allow for a much more immersive gaming experience (Figure 3). Ikea has utilized this in its AR App called Ikea Place, in which users can scan a room and place virtual pieces of furniture into it [14]. This allows users to preview how items might look in their homes without ever entering the store.
Different Devices
Currently, the main ways to experience AR are through streaming, smartglasses, and phones. While streaming may not typically be interpreted as a form of AR, it certainly qualifies. AR is defined as overlaying content onto a live feed. For example, during a live football game, the first down line displayed on the screen can be considered augmented reality. Furthermore, in video game streams, companies have used AR to put on astonishing spectacles for the audiences at home, as we will discuss later. Alternatively, smartglasses are devices that use projectors to display digital images onto the physical lens. While arguably more immersive, they are still far from accessible for many due to their $2,500 to $4,000 price tag. Besides streaming, most people experience AR through their mobile devices. In recent years, interest in AR and phones has begun growing exponentially, especially due to an explosion of popular apps such as Pokemon Go and face lenses in Snapchat. Because of its ease of accessibility and individualized experiences, the mobile AR market has become such a lucrative field that it is estimated to reach a billion users and generate $60 billion in revenue by 2021, according to Digi-capital, an industry analytics platform [3].
Uses of AR
AR may seem like a gimmicky new fad with very few practical applications. Though we may be ages away from developing holograms and other tools taken straight out of sci-fi movies, AR has been an asset in many industries. In the sections below, we will discuss AR’s various functions in several industries.
Industrial
As AR headsets cost several thousands of dollars and are still far from being common consumer goods, a significant portion of the AR field has been focused on targeting industry needs [5]. The best example of this would be Microsoft’s HoloLens, which is a pair of smartglasses that are aimed at helping people across a business collaborate and work more efficiently.
Imagine trying to sell a potential product to investors, but the product is still in development. However, you do have a digital model of the product. Upon putting on the HoloLens, the investors would be able to see an exact model of the product overlaid upon reality and interact with it. Or imagine yourself as a technician, trying to follow a manual to fix the wiring in an electrical system. Without AR technology, the book would need to be propped up and consulted every minute. The repair would take longer and be more tedious, making for an overall more inconvenient experience. Using the Hololens, a virtual version of the guide would be superimposed right next to the project, allowing you to see more information condensed within a single glance. There would be no more switching back and forth between the project at hand and the guide, as everything would be contained in one view. Moreover, points of interest can be automatically highlighted, as seen in Figure 4, decreasing the amount of time needed to figure out which physical parts line up according to the images in the book [6].
Streaming
In sports broadcasting of both traditional sports and electronic sports (esports), AR has been used for many years in the form of graphical analyses, visualizations, and animated replays. However, the most impressive use of AR in streaming thus far was by Riot Games, for their 2017 League of Legends World Championships. The audience physically present at the game were treated to a live band and performance, while the viewers at home watching the stream witnessed an amazing feat of engineering. During the opening, these live stream viewers saw a dragon from the game swoop down into the stadium, complete with perfectly timed roars to truly sell the experience (Figure 5). This required the developers to create an animation that would run at the exact same frame rate as the online broadcast, in order to seem like it was swooping into the stadium in real-time. They also had to build in lighting and color adjustments into the dragon’s animation to account for all possible weather scenarios, and block off parts of the animation so that the dragon would appear to be behind the stadium as it entered, allowing for maximum realism [7]. Finally, this specially crafted animation was superimposed onto video and streamed from strategically placed cameras, making it seem like a part of reality. This unbelievable scene that lasted only a couple of minutes required months of planning and development, along with dozens of technical innovations. However, it made use of AR in a way never seen before and amazed the millions watching. As companies see the increasing potential in AR, we are bound to see even more creative and awe-inspiring scenes in the future.
Figure 5. A screenshot of the dragon as stream viewers would have seen it
Mobile AR
Mobile AR refers to the overlaying of digital information onto the physical world, as seen through a phone’s lens. Popular due to its accessibility, the two main focuses of mobile AR are creative AR and contextual AR. Creative AR refers to applications that allow users to create novel content. A familiar example would be Snapchat face lenses. Through computer vision, the device can recognize faces and extract landmarks from the image such as the positions of the eyes, ears, and nose [8]. Then, it overlays the digital content over the appropriate locations, resulting in the fun and creative filters that transform your friends into dogs.
Contextual AR refers to AR that aids in providing information about the world around us. Imagine being able to aim your phone’s camera at a building and have the screen show information about the shops located in the building, as seen in Figure 6. For example, pretend you find yourself in a new city, not knowing anything about what is around you. You could point your phone at a monument, and have it display its name, creator, and cultural significance. Or you could point your phone at a plaza and have it show you what shops are in it, what they sell, and their hours and reviews. This futuristic-sounding application is already available today in the form of Monocle, a feature built into the Yelp app [9]. Though far from perfect, as businesses begin to see the practical uses of incorporating AR in their products, we will steadily advance our understanding of AR and its limits.
Looking Towards the Future
Technology is constantly expanding and integrating itself into our daily lives. As HoloLens inventor Alex Kipman said, “We will all live in a world very soon where we will interface and interact and instinctually manipulate technology ubiquitously through our days,” [10]. AR has infinite potential to significantly improve our daily lives by providing context to the world around us along with new and innovative content to consume. Automobile companies such as Toyota, Lexus, and BME have already begun experimenting with AR features, hopeful that one day we will have data and navigation details projected onto our windshield, creating the “display of the future” [15]. Companies are already seeing the benefit of investing in AR apps that allow customers to visualize products to scale in the comfort of their own homes. It not only eliminates the need for a physical store by reducing overhead costs but also is inspiring a revolution in electronic commerce and how people shop [16]. AR can also aid many other industries through realistic training simulations, virtual medical procedures, and enhanced communications. Today, AR is already implemented in ways that would have been inconceivable only a decade ago. Every day, we inch a little closer to the world of tomorrow, and AR most certainly has a place in it.
References
[1] L. Hudson, “Terror in 360 Degrees,” Slate Magazine, 02-May-2016. [Online]. Available: https://slate.com/culture/2016/05/virtual-reality-can-be-more-amazing-and-more-terrifying-than-a-traditional-video-game-experience.html. [Accessed: 26-Feb-2020].
[2] S. Mayekar, “The Future of AR Is SLAM Technology. But What Is SLAM?,” Analytics Insight, 06-Jul-2019. [Online]. Available: https://www.analyticsinsight.net/future-ar-slam-technology-slam/. [Accessed: 26-Feb-2020].
[3] C. Hackl, “Apple & Augmented Reality: AR Will Contextualize Consumer’s Realities & Transform Marketing.,” HuffPost, 13-Sep-2017. [Online]. Available: https://www.huffpost.com/entry/apple-augmented-reality-ar-will-contextualize-consumers_b_59b82db9e4b02c642e4a1208. [Accessed: 26-Feb-2020].
[4] L. Blippar, “What is augmented reality (AR) and how does it work? – Blog,” Blippar, 21-Aug-2018. [Online]. Available: https://www.blippar.com/blog/2018/08/21/what-is-augmented-reality-and-how-does-augmented-reality-work. [Accessed: 26-Feb-2020].
[5] H. Tsukayama, “Augmented-reality start-up Magic Leap is finally selling something,” The Washington Post, 08-Aug-2018. [Online]. Available: https://www.washingtonpost.com/technology/2018/08/08/augmented-reality-startup-magic-leap-is-finally-selling-something/. [Accessed: 26-Feb-2020].
[6] K. Weise, “You’re Hired. Now Wear This Headset to Learn the Job.,” The New York Times, 10-Jul-2019. [Online]. Available: https://www.nytimes.com/2019/07/10/business/microsoft-hololens-job-training.html. [Accessed: 26-Feb-2020].
[7] A. Horn and J. Restaino, “/dev: Summoning the Worlds Dragon,” /dev: Summoning the Worlds Dragon – League of Legends, 2018. [Online]. Available: https://nexus.leagueoflegends.com/en-us/2017/12/dev-summoning-the-worlds-dragon/. [Accessed: 26-Feb-2020].
[8] J. Le, “Snapchat’s Filters: How computer vision recognizes your face,” Medium, 22-Jul-2018. [Online]. Available: https://medium.com/cracking-the-data-science-interview/snapchats-filters-how-computer-vision-recognizes-your-face-9907d6904b91. [Accessed: 26-Feb-2020].
[9] D. Silverman, “The strange tale of Monocle, the pioneering AR app you’ve never heard of,” HoustonChronicle.com, 21-Nov-2017. [Online]. Available: https://www.houstonchronicle.com/techburger/article/The-strange-tale-of-Monocle-the-AR-pioneer-12371889.php. [Accessed: 26-Feb-2020].
[10] S. Stein, “Why AR is going to give you ‘superpowers’ in the future,” CNET, 28-Feb-2019. [Online]. Available: https://www.cnet.com/news/the-future-of-ar-according-to-microsoft/. [Accessed: 26-Feb-2020].
[11] ReportLinker, “The global augmented reality market size is estimated to grow from USD 10.7 billion in 2019 and projected to reach USD 72.7 billion by 2024; it is expected to grow at a CAGR of 46.6% from 2019 to 2024,” GlobeNewswire News Room, 21-Jan-2020. [Online]. Available: https://www.globenewswire.com/news-release/2020/01/21/1972730/0/en/The-global-augmented-reality-market-size-is-estimated-to-grow-from-USD-10-7-billion-in-2019-and-projected-to-reach-USD-72-7-billion-by-2024-it-is-expected-to-grow-at-a-CAGR-of-46-6.html. [Accessed: 29-Apr-2020].
[12] J. Harper, “Pentagon Eyeing More Advanced Virtual, Augmented Reality Headwear,” Pentagon Eyeing More Advanced Virtual, Augmented Reality Headwear, 27-Nov-2019. [Online]. Available: https://www.nationaldefensemagazine.org/articles/2019/11/27/pentagon-eyeing-more-advanced-virtual-augmented-reality-headwear. [Accessed: 29-Apr-2020].
[13] “Augmented Reality In Healthcare Will Be Revolutionary: 9 Examples,” The Medical Futurist, 16-Nov-2019. [Online]. Available: https://medicalfuturist.com/augmented-reality-in-healthcare-will-be-revolutionary/. [Accessed: 29-Apr-2020].
[14] A. Liszewski, “Ikea Expands Its Augmented Reality App, Lets You Fill an Entire Room With Virtual Swedish Furniture,” Gizmodo, 23-Sep-2019. [Online]. Available: https://gizmodo.com/ikeas-expands-augmented-reality-app-lets-you-fill-an-e-1838362429. [Accessed: 29-Apr-2020].
[15] B. Marr, “Are You Ready for Augmented Reality in your Car?,” Forbes, 26-Aug-2019. [Online]. Available: https://www.forbes.com/sites/bernardmarr/2019/08/26/are-you-ready-for-augmented-reality-in-your-car/#4cbd002e3144. [Accessed: 29-Apr-2020].
[16] O. Chia, “Shopping in AR: A look into retail’s future,” SlashGear, 23-Feb-2019. [Online]. Available: https://www.slashgear.com/shopping-in-ar-lego-shows-us-retails-future-23567009/. [Accessed: 29-Apr-2020].
