research papers on virtual reality

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 25 October 2021

Augmented reality and virtual reality displays: emerging technologies and future perspectives

• Jianghao Xiong 1 ,
• En-Lin Hsiang 1 ,
• Ziqian He 1 ,
• Tao Zhan   ORCID: orcid.org/0000-0001-5511-6666 1 &
• Shin-Tson Wu   ORCID: orcid.org/0000-0002-0943-0440 1  

Light: Science & Applications volume  10 , Article number:  216 ( 2021 ) Cite this article

93k Accesses

308 Citations

26 Altmetric

Metrics details

• Liquid crystals

With rapid advances in high-speed communication and computation, augmented reality (AR) and virtual reality (VR) are emerging as next-generation display platforms for deeper human-digital interactions. Nonetheless, to simultaneously match the exceptional performance of human vision and keep the near-eye display module compact and lightweight imposes unprecedented challenges on optical engineering. Fortunately, recent progress in holographic optical elements (HOEs) and lithography-enabled devices provide innovative ways to tackle these obstacles in AR and VR that are otherwise difficult with traditional optics. In this review, we begin with introducing the basic structures of AR and VR headsets, and then describing the operation principles of various HOEs and lithography-enabled devices. Their properties are analyzed in detail, including strong selectivity on wavelength and incident angle, and multiplexing ability of volume HOEs, polarization dependency and active switching of liquid crystal HOEs, device fabrication, and properties of micro-LEDs (light-emitting diodes), and large design freedoms of metasurfaces. Afterwards, we discuss how these devices help enhance the AR and VR performance, with detailed description and analysis of some state-of-the-art architectures. Finally, we cast a perspective on potential developments and research directions of these photonic devices for future AR and VR displays.

Introduction

Recent advances in high-speed communication and miniature mobile computing platforms have escalated a strong demand for deeper human-digital interactions beyond traditional flat panel displays. Augmented reality (AR) and virtual reality (VR) headsets 1 , 2 are emerging as next-generation interactive displays with the ability to provide vivid three-dimensional (3D) visual experiences. Their useful applications include education, healthcare, engineering, and gaming, just to name a few 3 , 4 , 5 . VR embraces a total immersive experience, while AR promotes the interaction between user, digital contents, and real world, therefore displaying virtual images while remaining see-through capability. In terms of display performance, AR and VR face several common challenges to satisfy demanding human vision requirements, including field of view (FoV), eyebox, angular resolution, dynamic range, and correct depth cue, etc. Another pressing demand, although not directly related to optical performance, is ergonomics. To provide a user-friendly wearing experience, AR and VR should be lightweight and ideally have a compact, glasses-like form factor. The above-mentioned requirements, nonetheless, often entail several tradeoff relations with one another, which makes the design of high-performance AR/VR glasses/headsets particularly challenging.

In the 1990s, AR/VR experienced the first boom, which quickly subsided due to the lack of eligible hardware and digital content 6 . Over the past decade, the concept of immersive displays was revisited and received a new round of excitement. Emerging technologies like holography and lithography have greatly reshaped the AR/VR display systems. In this article, we firstly review the basic requirements of AR/VR displays and their associated challenges. Then, we briefly describe the properties of two emerging technologies: holographic optical elements (HOEs) and lithography-based devices (Fig. 1 ). Next, we separately introduce VR and AR systems because of their different device structures and requirements. For the immersive VR system, the major challenges and how these emerging technologies help mitigate the problems will be discussed. For the see-through AR system, we firstly review the present status of light engines and introduce some architectures for the optical combiners. Performance summaries on microdisplay light engines and optical combiners will be provided, that serve as a comprehensive overview of the current AR display systems.

The left side illustrates HOEs and lithography-based devices. The right side shows the challenges in VR and architectures in AR, and how the emerging technologies can be applied

Key parameters of AR and VR displays

AR and VR displays face several common challenges to satisfy the demanding human vision requirements, such as FoV, eyebox, angular resolution, dynamic range, and correct depth cue, etc. These requirements often exhibit tradeoffs with one another. Before diving into detailed relations, it is beneficial to review the basic definitions of the above-mentioned display parameters.

Definition of parameters

Taking a VR system (Fig. 2a ) as an example. The light emitting from the display module is projected to a FoV, which can be translated to the size of the image perceived by the viewer. For reference, human vision’s horizontal FoV can be as large as 160° for monocular vision and 120° for overlapped binocular vision 6 . The intersection area of ray bundles forms the exit pupil, which is usually correlated with another parameter called eyebox. The eyebox defines the region within which the whole image FoV can be viewed without vignetting. It therefore generally manifests a 3D geometry 7 , whose volume is strongly dependent on the exit pupil size. A larger eyebox offers more tolerance to accommodate the user’s diversified interpupillary distance (IPD) and wiggling of headset when in use. Angular resolution is defined by dividing the total resolution of the display panel by FoV, which measures the sharpness of a perceived image. For reference, a human visual acuity of 20/20 amounts to 1 arcmin angular resolution, or 60 pixels per degree (PPD), which is considered as a common goal for AR and VR displays. Another important feature of a 3D display is depth cue. Depth cue can be induced by displaying two separate images to the left eye and the right eye, which forms the vergence cue. But the fixed depth of the displayed image often mismatches with the actual depth of the intended 3D image, which leads to incorrect accommodation cues. This mismatch causes the so-called vergence-accommodation conflict (VAC), which will be discussed in detail later. One important observation is that the VAC issue may be more serious in AR than VR, because the image in an AR display is directly superimposed onto the real-world with correct depth cues. The image contrast is dependent on the display panel and stray light. To achieve a high dynamic range, the display panel should exhibit high brightness, low dark level, and more than 10-bits of gray levels. Nowadays, the display brightness of a typical VR headset is about 150–200 cd/m 2 (or nits).

a Schematic of a VR display defining FoV, exit pupil, eyebox, angular resolution, and accommodation cue mismatch. b Sketch of an AR display illustrating ACR

Figure 2b depicts a generic structure of an AR display. The definition of above parameters remains the same. One major difference is the influence of ambient light on the image contrast. For a see-through AR display, ambient contrast ratio (ACR) 8 is commonly used to quantify the image contrast:

where L on ( L off ) represents the on (off)-state luminance (unit: nit), L am is the ambient luminance, and T is the see-through transmittance. In general, ambient light is measured in illuminance (lux). For the convenience of comparison, we convert illuminance to luminance by dividing a factor of π, assuming the emission profile is Lambertian. In a normal living room, the illuminance is about 100 lux (i.e., L am  ≈ 30 nits), while in a typical office lighting condition, L am  ≈ 150 nits. For outdoors, on an overcast day, L am  ≈ 300 nits, and L am  ≈ 3000 nits on a sunny day. For AR displays, a minimum ACR should be 3:1 for recognizable images, 5:1 for adequate readability, and ≥10:1 for outstanding readability. To make a simple estimate without considering all the optical losses, to achieve ACR = 10:1 in a sunny day (~3000 nits), the display needs to deliver a brightness of at least 30,000 nits. This imposes big challenges in finding a high brightness microdisplay and designing a low loss optical combiner.

Tradeoffs and potential solutions

Next, let us briefly review the tradeoff relations mentioned earlier. To begin with, a larger FoV leads to a lower angular resolution for a given display resolution. In theory, to overcome this tradeoff only requires a high-resolution-display source, along with high-quality optics to support the corresponding modulation transfer function (MTF). To attain 60 PPD across 100° FoV requires a 6K resolution for each eye. This may be realizable in VR headsets because a large display panel, say 2–3 inches, can still accommodate a high resolution with acceptable manufacture cost. However, for a glasses-like wearable AR display, the conflict between small display size and the high solution becomes obvious as further shrinking the pixel size of a microdisplay is challenging.

To circumvent this issue, the concept of the foveated display is proposed 9 , 10 , 11 , 12 , 13 . The idea is based on that the human eye only has high visual acuity in the central fovea region, which accounts for about 10° FoV. If the high-resolution image is only projected to fovea while the peripheral image remains low resolution, then a microdisplay with 2K resolution can satisfy the need. Regarding the implementation method of foveated display, a straightforward way is to optically combine two display sources 9 , 10 , 11 : one for foveal and one for peripheral FoV. This approach can be regarded as spatial multiplexing of displays. Alternatively, time-multiplexing can also be adopted, by temporally changing the optical path to produce different magnification factors for the corresponding FoV 12 . Finally, another approach without multiplexing is to use a specially designed lens with intended distortion to achieve non-uniform resolution density 13 . Aside from the implementation of foveation, another great challenge is to dynamically steer the foveated region as the viewer’s eye moves. This task is strongly related to pupil steering, which will be discussed in detail later.

A larger eyebox or FoV usually decreases the image brightness, which often lowers the ACR. This is exactly the case for a waveguide AR system with exit pupil expansion (EPE) while operating under a strong ambient light. To improve ACR, one approach is to dynamically adjust the transmittance with a tunable dimmer 14 , 15 . Another solution is to directly boost the image brightness with a high luminance microdisplay and an efficient combiner optics. Details of this topic will be discussed in the light engine section.

Another tradeoff of FoV and eyebox in geometric optical systems results from the conservation of etendue (or optical invariant). To increase the system etendue requires a larger optics, which in turn compromises the form factor. Finally, to address the VAC issue, the display system needs to generate a proper accommodation cue, which often requires the modulation of image depth or wavefront, neither of which can be easily achieved in a traditional geometric optical system. While remarkable progresses have been made to adopt freeform surfaces 16 , 17 , 18 , to further advance AR and VR systems requires additional novel optics with a higher degree of freedom in structure design and light modulation. Moreover, the employed optics should be thin and lightweight. To mitigate the above-mentioned challenges, diffractive optics is a strong contender. Unlike geometric optics relying on curved surfaces to refract or reflect light, diffractive optics only requires a thin layer of several micrometers to establish efficient light diffractions. Two major types of diffractive optics are HOEs based on wavefront recording and manually written devices like surface relief gratings (SRGs) based on lithography. While SRGs have large design freedoms of local grating geometry, a recent publication 19 indicates the combination of HOE and freeform optics can also offer a great potential for arbitrary wavefront generation. Furthermore, the advances in lithography have also enabled optical metasurfaces beyond diffractive and refractive optics, and miniature display panels like micro-LED (light-emitting diode). These devices hold the potential to boost the performance of current AR/VR displays, while keeping a lightweight and compact form factor.

Formation and properties of HOEs

HOE generally refers to a recorded hologram that reproduces the original light wavefront. The concept of holography is proposed by Dennis Gabor 20 , which refers to the process of recording a wavefront in a medium (hologram) and later reconstructing it with a reference beam. Early holography uses intensity-sensitive recording materials like silver halide emulsion, dichromated gelatin, and photopolymer 21 . Among them, photopolymer stands out due to its easy fabrication and ability to capture high-fidelity patterns 22 , 23 . It has therefore found extensive applications like holographic data storage 23 and display 24 , 25 . Photopolymer HOEs (PPHOEs) have a relatively small refractive index modulation and therefore exhibits a strong selectivity on the wavelength and incident angle. Another feature of PPHOE is that several holograms can be recorded into a photopolymer film by consecutive exposures. Later, liquid-crystal holographic optical elements (LCHOEs) based on photoalignment polarization holography have also been developed 25 , 26 . Due to the inherent anisotropic property of liquid crystals, LCHOEs are extremely sensitive to the polarization state of the input light. This feature, combined with the polarization modulation ability of liquid crystal devices, offers a new possibility for dynamic wavefront modulation in display systems.

The formation of PPHOE is illustrated in Fig. 3a . When exposed to an interfering field with high-and-low intensity fringes, monomers tend to move toward bright fringes due to the higher local monomer-consumption rate. As a result, the density and refractive index is slightly larger in bright regions. Note the index modulation δ n here is defined as the difference between the maximum and minimum refractive indices, which may be twice the value in other definitions 27 . The index modulation δ n is typically in the range of 0–0.06. To understand the optical properties of PPHOE, we simulate a transmissive grating and a reflective grating using rigorous coupled-wave analysis (RCWA) 28 , 29 and plot the results in Fig. 3b . Details of grating configuration can be found in Table S1 . Here, the reason for only simulating gratings is that for a general HOE, the local region can be treated as a grating. The observation of gratings can therefore offer a general insight of HOEs. For a transmissive grating, its angular bandwidth (efficiency > 80%) is around 5° ( λ  = 550 nm), while the spectral band is relatively broad, with bandwidth around 175 nm (7° incidence). For a reflective grating, its spectral band is narrow, with bandwidth around 10 nm. The angular bandwidth varies with the wavelength, ranging from 2° to 20°. The strong selectivity of PPHOE on wavelength and incident angle is directly related to its small δ n , which can be adjusted by controlling the exposure dosage.

a Schematic of the formation of PPHOE. Simulated efficiency plots for b1 transmissive and b2 reflective PPHOEs. c Working principle of multiplexed PPHOE. d Formation and molecular configurations of LCHOEs. Simulated efficiency plots for e1 transmissive and e2 reflective LCHOEs. f Illustration of polarization dependency of LCHOEs

A distinctive feature of PPHOE is the ability to multiplex several holograms into one film sample. If the exposure dosage of a recording process is controlled so that the monomers are not completely depleted in the first exposure, the remaining monomers can continue to form another hologram in the following recording process. Because the total amount of monomer is fixed, there is usually an efficiency tradeoff between multiplexed holograms. The final film sample would exhibit the wavefront modulation functions of multiple holograms (Fig. 3c ).

Liquid crystals have also been used to form HOEs. LCHOEs can generally be categorized into volume-recording type and surface-alignment type. Volume-recording type LCHOEs are either based on early polarization holography recordings with azo-polymer 30 , 31 , or holographic polymer-dispersed liquid crystals (HPDLCs) 32 , 33 formed by liquid-crystal-doped photopolymer. Surface-alignment type LCHOEs are based on photoalignment polarization holography (PAPH) 34 . The first step is to record the desired polarization pattern in a thin photoalignment layer, and the second step is to use it to align the bulk liquid crystal 25 , 35 . Due to the simple fabrication process, high efficiency, and low scattering from liquid crystal’s self-assembly nature, surface-alignment type LCHOEs based on PAPH have recently attracted increasing interest in applications like near-eye displays. Here, we shall focus on this type of surface-alignment LCHOE and refer to it as LCHOE thereafter for simplicity.

The formation of LCHOEs is illustrated in Fig. 3d . The information of the wavefront and the local diffraction pattern is recorded in a thin photoalignment layer. The volume liquid crystal deposited on the photoalignment layer, depending on whether it is nematic liquid crystal or cholesteric liquid crystal (CLC), forms a transmissive or a reflective LCHOE. In a transmissive LCHOE, the bulk nematic liquid crystal molecules generally follow the pattern of the bottom alignment layer. The smallest allowable pattern period is governed by the liquid crystal distortion-free energy model, which predicts the pattern period should generally be larger than sample thickness 36 , 37 . This results in a maximum diffraction angle under 20°. On the other hand, in a reflective LCHOE 38 , 39 , the bulk CLC molecules form a stable helical structure, which is tilted to match the k -vector of the bottom pattern. The structure exhibits a very low distorted free energy 40 , 41 and can accommodate a pattern period that is small enough to diffract light into the total internal reflection (TIR) of a glass substrate.

The diffraction property of LCHOEs is shown in Fig. 3e . The maximum refractive index modulation of LCHOE is equal to the liquid crystal birefringence (Δ n ), which may vary from 0.04 to 0.5, depending on the molecular conjugation 42 , 43 . The birefringence used in our simulation is Δ n  = 0.15. Compared to PPHOEs, the angular and spectral bandwidths are significantly larger for both transmissive and reflective LCHOEs. For a transmissive LCHOE, its angular bandwidth is around 20° ( λ  = 550 nm), while the spectral bandwidth is around 300 nm (7° incidence). For a reflective LCHOE, its spectral bandwidth is around 80 nm and angular bandwidth could vary from 15° to 50°, depending on the wavelength.

The anisotropic nature of liquid crystal leads to LCHOE’s unique polarization-dependent response to an incident light. As depicted in Fig. 3f , for a transmissive LCHOE the accumulated phase is opposite for the conjugated left-handed circular polarization (LCP) and right-handed circular polarization (RCP) states, leading to reversed diffraction directions. For a reflective LCHOE, the polarization dependency is similar to that of a normal CLC. For the circular polarization with the same handedness as the helical structure of CLC, the diffraction is strong. For the opposite circular polarization, the diffraction is negligible.

Another distinctive property of liquid crystal is its dynamic response to an external voltage. The LC reorientation can be controlled with a relatively low voltage (<10 V rms ) and the response time is on the order of milliseconds, depending mainly on the LC viscosity and layer thickness. Methods to dynamically control LCHOEs can be categorized as active addressing and passive addressing, which can be achieved by either directly switching the LCHOE or modulating the polarization state with an active waveplate. Detailed addressing methods will be described in the VAC section.

Lithography-enabled devices

Lithography technologies are used to create arbitrary patterns on wafers, which lays the foundation of the modern integrated circuit industry 44 . Photolithography is suitable for mass production while electron/ion beam lithography is usually used to create photomask for photolithography or to write structures with nanometer-scale feature size. Recent advances in lithography have enabled engineered structures like optical metasurfaces 45 , SRGs 46 , as well as micro-LED displays 47 . Metasurfaces exhibit a remarkable design freedom by varying the shape of meta-atoms, which can be utilized to achieve novel functions like achromatic focus 48 and beam steering 49 . Similarly, SRGs also offer a large design freedom by manipulating the geometry of local grating regions to realize desired optical properties. On the other hand, micro-LED exhibits several unique features, such as ultrahigh peak brightness, small aperture ratio, excellent stability, and nanosecond response time, etc. As a result, micro-LED is a promising candidate for AR and VR systems for achieving high ACR and high frame rate for suppressing motion image blurs. In the following section, we will briefly review the fabrication and properties of micro-LEDs and optical modulators like metasurfaces and SRGs.

Fabrication and properties of micro-LEDs

LEDs with a chip size larger than 300 μm have been widely used in solid-state lighting and public information displays. Recently, micro-LEDs with chip sizes <5 μm have been demonstrated 50 . The first micro-LED disc with a diameter of about 12 µm was demonstrated in 2000 51 . After that, a single color (blue or green) LED microdisplay was demonstrated in 2012 52 . The high peak brightness, fast response time, true dark state, and long lifetime of micro-LEDs are attractive for display applications. Therefore, many companies have since released their micro-LED prototypes or products, ranging from large-size TVs to small-size microdisplays for AR/VR applications 53 , 54 . Here, we focus on micro-LEDs for near-eye display applications. Regarding the fabrication of micro-LEDs, through the metal-organic chemical vapor deposition (MOCVD) method, the AlGaInP epitaxial layer is grown on GaAs substrate for red LEDs, and GaN epitaxial layers on sapphire substrate for green and blue LEDs. Next, a photolithography process is applied to define the mesa and deposit electrodes. To drive the LED array, the fabricated micro-LEDs are transferred to a CMOS (complementary metal oxide semiconductor) driver board. For a small size (<2 inches) microdisplay used in AR or VR, the precision of the pick-and-place transfer process is hard to meet the high-resolution-density (>1000 pixel per inch) requirement. Thus, the main approach to assemble LED chips with driving circuits is flip-chip bonding 50 , 55 , 56 , 57 , as Fig. 4a depicts. In flip-chip bonding, the mesa and electrode pads should be defined and deposited before the transfer process, while metal bonding balls should be preprocessed on the CMOS substrate. After that, thermal-compression method is used to bond the two wafers together. However, due to the thermal mismatch of LED chip and driving board, as the pixel size decreases, the misalignment between the LED chip and the metal bonding ball on the CMOS substrate becomes serious. In addition, the common n-GaN layer may cause optical crosstalk between pixels, which degrades the image quality. To overcome these issues, the LED epitaxial layer can be firstly metal-bonded with the silicon driver board, followed by the photolithography process to define the LED mesas and electrodes. Without the need for an alignment process, the pixel size can be reduced to <5 µm 50 .

a Illustration of flip-chip bonding technology. b Simulated IQE-LED size relations for red and blue LEDs based on ABC model. c Comparison of EQE of different LED sizes with and without KOH and ALD side wall treatment. d Angular emission profiles of LEDs with different sizes. Metasurfaces based on e resonance-tuning, f non-resonance tuning and g combination of both. h Replication master and i replicated SRG based on nanoimprint lithography. Reproduced from a ref. 55 with permission from AIP Publishing, b ref. 61 with permission from PNAS, c ref. 66 with permission from IOP Publishing, d ref. 67 with permission from AIP Publishing, e ref. 69 with permission from OSA Publishing f ref. 48 with permission from AAAS g ref. 70 with permission from AAAS and h , i ref. 85 with permission from OSA Publishing

In addition to manufacturing process, the electrical and optical characteristics of LED also depend on the chip size. Generally, due to Shockley-Read-Hall (SRH) non-radiative recombination on the sidewall of active area, a smaller LED chip size results in a lower internal quantum efficiency (IQE), so that the peak IQE driving point will move toward a higher current density due to increased ratio of sidewall surface to active volume 58 , 59 , 60 . In addition, compared to the GaN-based green and blue LEDs, the AlGaInP-based red LEDs with a larger surface recombination and carrier diffusion length suffer a more severe efficiency drop 61 , 62 . Figure 4b shows the simulated result of IQE drop in relation with the LED chip size of blue and red LEDs based on ABC model 63 . To alleviate the efficiency drop caused by sidewall defects, depositing passivation materials by atomic layer deposition (ALD) or plasma enhanced chemical vapor deposition (PECVD) is proven to be helpful for both GaN and AlGaInP based LEDs 64 , 65 . In addition, applying KOH (Potassium hydroxide) treatment after ALD can further reduce the EQE drop of micro-LEDs 66 (Fig. 4c ). Small-size LEDs also exhibit some advantages, such as higher light extraction efficiency (LEE). Compared to an 100-µm LED, the LEE of a 2-µm LED increases from 12.2 to 25.1% 67 . Moreover, the radiation pattern of micro-LED is more directional than that of a large-size LED (Fig. 4d ). This helps to improve the lens collection efficiency in AR/VR display systems.

Metasurfaces and SGs

Thanks to the advances in lithography technology, low-loss dielectric metasurfaces working in the visible band have recently emerged as a platform for wavefront shaping 45 , 48 , 68 . They consist of an array of subwavelength-spaced structures with individually engineered wavelength-dependent polarization/phase/ amplitude response. In general, the light modulation mechanisms can be classified into resonant tuning 69 (Fig. 4e ), non-resonant tuning 48 (Fig. 4f ), and combination of both 70 (Fig. 4g ). In comparison with non-resonant tuning (based on geometric phase and/or dynamic propagation phase), the resonant tuning (such as Fabry–Pérot resonance, Mie resonance, etc.) is usually associated with a narrower operating bandwidth and a smaller out-of-plane aspect ratio (height/width) of nanostructures. As a result, they are easier to fabricate but more sensitive to fabrication tolerances. For both types, materials with a higher refractive index and lower absorption loss are beneficial to reduce the aspect ratio of nanostructure and improve the device efficiency. To this end, titanium dioxide (TiO 2 ) and gallium nitride (GaN) are the major choices for operating in the entire visible band 68 , 71 . While small-sized metasurfaces (diameter <1 mm) are usually fabricated via electron-beam lithography or focused ion beam milling in the labs, the ability of mass production is the key to their practical adoption. The deep ultraviolet (UV) photolithography has proven its feasibility for reproducing centimeter-size metalenses with decent imaging performance, while it requires multiple steps of etching 72 . Interestingly, the recently developed UV nanoimprint lithography based on a high-index nanocomposite only takes a single step and can obtain an aspect ratio larger than 10, which shows great promise for high-volume production 73 .

The arbitrary wavefront shaping capability and the thinness of the metasurfaces have aroused strong research interests in the development of novel AR/VR prototypes with improved performance. Lee et al. employed nanoimprint lithography to fabricate a centimeter-size, geometric-phase metalens eyepiece for full-color AR displays 74 . Through tailoring its polarization conversion efficiency and stacking with a circular polarizer, the virtual image can be superimposed with the surrounding scene. The large numerical aperture (NA~0.5) of the metalens eyepiece enables a wide FoV (>76°) that conventional optics are difficult to obtain. However, the geometric phase metalens is intrinsically a diffractive lens that also suffers from strong chromatic aberrations. To overcome this issue, an achromatic lens can be designed via simultaneously engineering the group delay and the group delay dispersion 75 , 76 , which will be described in detail later. Other novel and/or improved near-eye display architectures include metasurface-based contact lens-type AR 77 , achromatic metalens array enabled integral-imaging light field displays 78 , wide FoV lightguide AR with polarization-dependent metagratings 79 , and off-axis projection-type AR with an aberration-corrected metasurface combiner 80 , 81 , 82 . Nevertheless, from the existing AR/VR prototypes, metasurfaces still face a strong tradeoff between numerical aperture (for metalenses), chromatic aberration, monochromatic aberration, efficiency, aperture size, and fabrication complexity.

On the other hand, SRGs are diffractive gratings that have been researched for decades as input/output couplers of waveguides 83 , 84 . Their surface is composed of corrugated microstructures, and different shapes including binary, blazed, slanted, and even analogue can be designed. The parameters of the corrugated microstructures are determined by the target diffraction order, operation spectral bandwidth, and angular bandwidth. Compared to metasurfaces, SRGs have a much larger feature size and thus can be fabricated via UV photolithography and subsequent etching. They are usually replicated by nanoimprint lithography with appropriate heating and surface treatment. According to a report published a decade ago, SRGs with a height of 300 nm and a slant angle of up to 50° can be faithfully replicated with high yield and reproducibility 85 (Fig. 4g, h ).

Challenges and solutions of VR displays

The fully immersive nature of VR headset leads to a relatively fixed configuration where the display panel is placed in front of the viewer’s eye and an imaging optics is placed in-between. Regarding the system performance, although inadequate angular resolution still exists in some current VR headsets, the improvement of display panel resolution with advanced fabrication process is expected to solve this issue progressively. Therefore, in the following discussion, we will mainly focus on two major challenges: form factor and 3D cue generation.

Form factor

Compact and lightweight near-eye displays are essential for a comfortable user experience and therefore highly desirable in VR headsets. Current mainstream VR headsets usually have a considerably larger volume than eyeglasses, and most of the volume is just empty. This is because a certain distance is required between the display panel and the viewing optics, which is usually close to the focal length of the lens system as illustrated in Fig. 5a . Conventional VR headsets employ a transmissive lens with ~4 cm focal length to offer a large FoV and eyebox. Fresnel lenses are thinner than conventional ones, but the distance required between the lens and the panel does not change significantly. In addition, the diffraction artifacts and stray light caused by the Fresnel grooves can degrade the image quality, or MTF. Although the resolution density, quantified as pixel per inch (PPI), of current VR headsets is still limited, eventually Fresnel lens will not be an ideal solution when a high PPI display is available. The strong chromatic aberration of Fresnel singlet should also be compensated if a high-quality imaging system is preferred.

a Schematic of a basic VR optical configuration. b Achromatic metalens used as VR eyepiece. c VR based on curved display and lenslet array. d Basic working principle of a VR display based on pancake optics. e VR with pancake optics and Fresnel lens array. f VR with pancake optics based on purely HOEs. Reprinted from b ref. 87 under the Creative Commons Attribution 4.0 License. Adapted from c ref. 88 with permission from IEEE, e ref. 91 and f ref. 92 under the Creative Commons Attribution 4.0 License

It is tempting to replace the refractive elements with a single thin diffractive lens like a transmissive LCHOE. However, the diffractive nature of such a lens will result in serious color aberrations. Interestingly, metalenses can fulfil this objective without color issues. To understand how metalenses achieve achromatic focus, let us first take a glance at the general lens phase profile \(\Phi (\omega ,r)\) expanded as a Taylor series 75 :

where \(\varphi _0(\omega )\) is the phase at the lens center, \(F\left( \omega \right)\) is the focal length as a function of frequency ω , r is the radial coordinate, and \(\omega _0\) is the central operation frequency. To realize achromatic focus, \(\partial F{{{\mathrm{/}}}}\partial \omega\) should be zero. With a designed focal length, the group delay \(\partial \Phi (\omega ,r){{{\mathrm{/}}}}\partial \omega\) and the group delay dispersion \(\partial ^2\Phi (\omega ,r){{{\mathrm{/}}}}\partial \omega ^2\) can be determined, and \(\varphi _0(\omega )\) is an auxiliary degree of freedom of the phase profile design. In the design of an achromatic metalens, the group delay is a function of the radial coordinate and monotonically increases with the metalens radius. Many designs have proven that the group delay has a limited variation range 75 , 76 , 78 , 86 . According to Shrestha et al. 86 , there is an inevitable tradeoff between the maximum radius of the metalens, NA, and operation bandwidth. Thus, the reported achromatic metalenses at visible usually have limited lens aperture (e.g., diameter < 250 μm) and NA (e.g., <0.2). Such a tradeoff is undesirable in VR displays, as the eyepiece favors a large clear aperture (inch size) and a reasonably high NA (>0.3) to maintain a wide FoV and a reasonable eye relief 74 .

To overcome this limitation, Li et al. 87 proposed a novel zone lens method. Unlike the traditional phase Fresnel lens where the zones are determined by the phase reset, the new approach divides the zones by the group delay reset. In this way, the lens aperture and NA can be much enlarged, and the group delay limit is bypassed. A notable side effect of this design is the phase discontinuity at zone boundaries that will contribute to higher-order focusing. Therefore, significant efforts have been conducted to find the optimal zone transition locations and to minimize the phase discontinuities. Using this method, they have demonstrated an impressive 2-mm-diameter metalens with NA = 0.7 and nearly diffraction-limited focusing for the designed wavelengths (488, 532, 658 nm) (Fig. 5b ). Such a metalens consists of 681 zones and works for the visible band ranging from 470 to 670 nm, though the focusing efficiency is in the order of 10%. This is a great starting point for the achromatic metalens to be employed as a compact, chromatic-aberration-free eyepiece in near-eye displays. Future challenges are how to further increase the aperture size, correct the off-axis aberrations, and improve the optical efficiency.

Besides replacing the refractive lens with an achromatic metalens, another way to reduce system focal length without decreasing NA is to use a lenslet array 88 . As depicted in Fig. 5c , both the lenslet array and display panel adopt a curved structure. With the latest flexible OLED panel, the display can be easily curved in one dimension. The system exhibits a large diagonal FoV of 180° with an eyebox of 19 by 12 mm. The geometry of each lenslet is optimized separately to achieve an overall performance with high image quality and reduced distortions.

Aside from trying to shorten the system focal length, another way to reduce total track is to fold optical path. Recently, polarization-based folded lenses, also known as pancake optics, are under active development for VR applications 89 , 90 . Figure 5d depicts the structure of an exemplary singlet pancake VR lens system. The pancake lenses can offer better imaging performance with a compact form factor since there are more degrees of freedom in the design and the actual light path is folded thrice. By using a reflective surface with a positive power, the field curvature of positive refractive lenses can be compensated. Also, the reflective surface has no chromatic aberrations and it contributes considerable optical power to the system. Therefore, the optical power of refractive lenses can be smaller, resulting in an even weaker chromatic aberration. Compared to Fresnel lenses, the pancake lenses have smooth surfaces and much fewer diffraction artifacts and stray light. However, such a pancake lens design is not perfect either, whose major shortcoming is low light efficiency. With two incidences of light on the half mirror, the maximum system efficiency is limited to 25% for a polarized input and 12.5% for an unpolarized input light. Moreover, due to the existence of multiple surfaces in the system, stray light caused by surface reflections and polarization leakage may lead to apparent ghost images. As a result, the catadioptric pancake VR headset usually manifests a darker imagery and lower contrast than the corresponding dioptric VR.

Interestingly, the lenslet and pancake optics can be combined to further reduce the system form. Bang et al. 91 demonstrated a compact VR system with a pancake optics and a Fresnel lenslet array. The pancake optics serves to fold the optical path between the display panel and the lenslet array (Fig. 5e ). Another Fresnel lens is used to collect the light from the lenslet array. The system has a decent horizontal FoV of 102° and an eyebox of 8 mm. However, a certain degree of image discontinuity and crosstalk are still present, which can be improved with further optimizations on the Fresnel lens and the lenslet array.

One step further, replacing all conventional optics in catadioptric VR headset with holographic optics can make the whole system even thinner. Maimone and Wang demonstrated such a lightweight, high-resolution, and ultra-compact VR optical system using purely HOEs 92 . This holographic VR optics was made possible by combining several innovative optical components, including a reflective PPHOE, a reflective LCHOE, and a PPHOE-based directional backlight with laser illumination, as shown in Fig. 5f . Since all the optical power is provided by the HOEs with negligible weight and volume, the total physical thickness can be reduced to <10 mm. Also, unlike conventional bulk optics, the optical power of a HOE is independent of its thickness, only subject to the recording process. Another advantage of using holographic optical devices is that they can be engineered to offer distinct phase profiles for different wavelengths and angles of incidence, adding extra degrees of freedom in optical designs for better imaging performance. Although only a single-color backlight has been demonstrated, such a PPHOE has the potential to achieve full-color laser backlight with multiplexing ability. The PPHOE and LCHOE in the pancake optics can also be optimized at different wavelengths for achieving high-quality full-color images.

Vergence-accommodation conflict

Conventional VR displays suffer from VAC, which is a common issue for stereoscopic 3D displays 93 . In current VR display modules, the distance between the display panel and the viewing optics is fixed, which means the VR imagery is displayed at a single depth. However, the image contents are generated by parallax rendering in three dimensions, offering distinct images for two eyes. This approach offers a proper stimulus to vergence but completely ignores the accommodation cue, which leads to the well-known VAC that can cause an uncomfortable user experience. Since the beginning of this century, numerous methods have been proposed to solve this critical issue. Methods to produce accommodation cue include multifocal/varifocal display 94 , holographic display 95 , and integral imaging display 96 . Alternatively, elimination of accommodation cue using a Maxwellian-view display 93 also helps to mitigate the VAC. However, holographic displays and Maxwellian-view displays generally require a totally different optical architecture than current VR systems. They are therefore more suitable for AR displays, which will be discussed later. Integral imaging, on the other hand, has an inherent tradeoff between view number and resolution. For current VR headsets pursuing high resolution to match human visual acuity, it may not be an appealing solution. Therefore, multifocal/varifocal displays that rely on depth modulation is a relatively practical and effective solution for VR headsets. Regarding the working mechanism, multifocal displays present multiple images with different depths to imitate the original 3D scene. Varifocal displays, in contrast, only show one image at each time frame. The image depth matches the viewer’s vergence depth. Nonetheless, the pre-knowledge of the viewer’s vergence depth requires an additional eye-tracking module. Despite different operation principles, a varifocal display can often be converted to a multifocal display as long as the varifocal module has enough modulation bandwidth to support multiple depths in a time frame.

To achieve depth modulation in a VR system, traditional liquid lens 97 , 98 with tunable focus suffers from the small aperture and large aberrations. Alvarez lens 99 is another tunable-focus solution but it requires mechanical adjustment, which adds to system volume and complexity. In comparison, transmissive LCHOEs with polarization dependency can achieve focus adjustment with electronic driving. Its ultra-thinness also satisfies the requirement of small form factors in VR headsets. The diffractive behavior of transmissive LCHOEs is often interpreted by the mechanism of Pancharatnam-Berry phase (also known as geometric phase) 100 . They are therefore often called Pancharatnam-Berry optical elements (PBOEs). The corresponding lens component is referred as Pancharatnam-Berry lens (PBL).

Two main approaches are used to switch the focus of a PBL, active addressing and passive addressing. In active addressing, the PBL itself (made of LC) can be switched by an applied voltage (Fig. 6a ). The optical power of the liquid crystal PBLs can be turned-on and -off by controlling the voltage. Stacking multiple active PBLs can produce 2 N depths, where N is the number of PBLs. The drawback of using active PBLs, however, is the limited spectral bandwidth since their diffraction efficiency is usually optimized at a single wavelength. In passive addressing, the depth modulation is achieved through changing the polarization state of input light by a switchable half-wave plate (HWP) (Fig. 6b ). The focal length can therefore be switched thanks to the polarization sensitivity of PBLs. Although this approach has a slightly more complicated structure, the overall performance can be better than the active one, because the PBLs made of liquid crystal polymer can be designed to manifest high efficiency within the entire visible spectrum 101 , 102 .

Working principles of a depth switching PBL module based on a active addressing and b passive addressing. c A four-depth multifocal display based on time multiplexing. d A two-depth multifocal display based on polarization multiplexing. Reproduced from c ref. 103 with permission from OSA Publishing and d ref. 104 with permission from OSA Publishing

With the PBL module, multifocal displays can be built using time-multiplexing technique. Zhan et al. 103 demonstrated a four-depth multifocal display using two actively switchable liquid crystal PBLs (Fig. 6c ). The display is synchronized with the PBL module, which lowers the frame rate by the number of depths. Alternatively, multifocal displays can also be achieved by polarization-multiplexing, as demonstrated by Tan et al. 104 . The basic principle is to adjust the polarization state of local pixels so the image content on two focal planes of a PBL can be arbitrarily controlled (Fig. 6d ). The advantage of polarization multiplexing is that it does not sacrifice the frame rate, but it can only support two planes because only two orthogonal polarization states are available. Still, it can be combined with time-multiplexing to reduce the frame rate sacrifice by half. Naturally, varifocal displays can also be built with a PBL module. A fast-response 64-depth varifocal module with six PBLs has been demonstrated 105 .

The compact structure of PBL module leads to a natural solution of integrating it with above-mentioned pancake optics. A compact VR headset with dynamic depth modulation to solve VAC is therefore possible in practice. Still, due to the inherent diffractive nature of PBL, the PBL module face the issue of chromatic dispersion of focal length. To compensate for different focal depths for RGB colors may require additional digital corrections in image-rendering.

Architectures of AR displays

Unlike VR displays with a relatively fixed optical configuration, there exist a vast number of architectures in AR displays. Therefore, instead of following the narrative of tackling different challenges, a more appropriate way to review AR displays is to separately introduce each architecture and discuss its associated engineering challenges. An AR display usually consists of a light engine and an optical combiner. The light engine serves as display image source, while the combiner delivers the displayed images to viewer’s eye and in the meantime transmits the environment light. Some performance parameters like frame rate and power consumption are mainly determined by the light engine. Parameters like FoV, eyebox and MTF are primarily dependent on the combiner optics. Moreover, attributes like image brightness, overall efficiency, and form factor are influenced by both light engine and combiner. In this section, we will firstly discuss the light engine, where the latest advances in micro-LED on chip are reviewed and compared with existing microdisplay systems. Then, we will introduce two main types of combiners: free-space combiner and waveguide combiner.

Light engine

The light engine determines several essential properties of the AR system like image brightness, power consumption, frame rate, and basic etendue. Several types of microdisplays have been used in AR, including micro-LED, micro-organic-light-emitting-diodes (micro-OLED), liquid-crystal-on-silicon (LCoS), digital micromirror device (DMD), and laser beam scanning (LBS) based on micro-electromechanical system (MEMS). We will firstly describe the working principles of these devices and then analyze their performance. For those who are more interested in final performance parameters than details, Table 1 provides a comprehensive summary.

Working principles

Micro-LED and micro-OLED are self-emissive display devices. They are usually more compact than LCoS and DMD because no illumination optics is required. The fundamentally different material systems of LED and OLED lead to different approaches to achieve full-color displays. Due to the “green gap” in LEDs, red LEDs are manufactured on a different semiconductor material from green and blue LEDs. Therefore, how to achieve full-color display in high-resolution density microdisplays is quite a challenge for micro-LEDs. Among several solutions under research are two main approaches. The first is to combine three separate red, green and blue (RGB) micro-LED microdisplay panels 106 . Three single-color micro-LED microdisplays are manufactured separately through flip-chip transfer technology. Then, the projected images from three microdisplay panels are integrated by a trichroic prism (Fig. 7a ).

a RGB micro-LED microdisplays combined by a trichroic prism. b QD-based micro-LED microdisplay. c Micro-OLED display with 4032 PPI. Working principles of d LCoS, e DMD, and f MEMS-LBS display modules. Reprinted from a ref. 106 with permission from IEEE, b ref. 108 with permission from Chinese Laser Press, c ref. 121 with permission from Jon Wiley and Sons, d ref. 124 with permission from Spring Nature, e ref. 126 with permission from Springer and f ref. 128 under the Creative Commons Attribution 4.0 License

Another solution is to assemble color-conversion materials like quantum dot (QD) on top of blue or ultraviolet (UV) micro-LEDs 107 , 108 , 109 (Fig. 7b ). The quantum dot color filter (QDCF) on top of the micro-LED array is mainly fabricated by inkjet printing or photolithography 110 , 111 . However, the display performance of color-conversion micro-LED displays is restricted by the low color-conversion efficiency, blue light leakage, and color crosstalk. Extensive efforts have been conducted to improve the QD-micro-LED performance. To boost QD conversion efficiency, structure designs like nanoring 112 and nanohole 113 , 114 have been proposed, which utilize the Förster resonance energy transfer mechanism to transfer excessive excitons in the LED active region to QD. To prevent blue light leakage, methods using color filters or reflectors like distributed Bragg reflector (DBR) 115 and CLC film 116 on top of QDCF are proposed. Compared to color filters that absorb blue light, DBR and CLC film help recycle the leaked blue light to further excite QDs. Other methods to achieve full-color micro-LED display like vertically stacked RGB micro-LED array 61 , 117 , 118 and monolithic wavelength tunable nanowire LED 119 are also under investigation.

Micro-OLED displays can be generally categorized into RGB OLED and white OLED (WOLED). RGB OLED displays have separate sub-pixel structures and optical cavities, which resonate at the desirable wavelength in RGB channels, respectively. To deposit organic materials onto the separated RGB sub-pixels, a fine metal mask (FMM) that defines the deposition area is required. However, high-resolution RGB OLED microdisplays still face challenges due to the shadow effect during the deposition process through FMM. In order to break the limitation, a silicon nitride film with small shadow has been proposed as a mask for high-resolution deposition above 2000 PPI (9.3 µm) 120 .

WOLED displays use color filters to generate color images. Without the process of depositing patterned organic materials, a high-resolution density up to 4000 PPI has been achieved 121 (Fig. 7c ). However, compared to RGB OLED, the color filters in WOLED absorb about 70% of the emitted light, which limits the maximum brightness of the microdisplay. To improve the efficiency and peak brightness of WOLED microdisplays, in 2019 Sony proposed to apply newly designed cathodes (InZnO) and microlens arrays on OLED microdisplays, which increased the peak brightness from 1600 nits to 5000 nits 120 . In addition, OLEDWORKs has proposed a multi-stacked OLED 122 with optimized microcavities whose emission spectra match the transmission bands of the color filters. The multi-stacked OLED shows a higher luminous efficiency (cd/A), but also requires a higher driving voltage. Recently, by using meta-mirrors as bottom reflective anodes, patterned microcavities with more than 10,000 PPI have been obtained 123 . The high-resolution meta-mirrors generate different reflection phases in the RGB sub-pixels to achieve desirable resonant wavelengths. The narrow emission spectra from the microcavity help to reduce the loss from color filters or even eliminate the need of color filters.

LCoS and DMD are light-modulating displays that generate images by controlling the reflection of each pixel. For LCoS, the light modulation is achieved by manipulating the polarization state of output light through independently controlling the liquid crystal reorientation in each pixel 124 , 125 (Fig. 7d ). Both phase-only and amplitude modulators have been employed. DMD is an amplitude modulation device. The modulation is achieved through controlling the tilt angle of bi-stable micromirrors 126 (Fig. 7e ). To generate an image, both LCoS and DMD rely on the light illumination systems, with LED or laser as light source. For LCoS, the generation of color image can be realized either by RGB color filters on LCoS (with white LEDs) or color-sequential addressing (with RGB LEDs or lasers). However, LCoS requires a linearly polarized light source. For an unpolarized LED light source, usually, a polarization recycling system 127 is implemented to improve the optical efficiency. For a single-panel DMD, the color image is mainly obtained through color-sequential addressing. In addition, DMD does not require a polarized light so that it generally exhibits a higher efficiency than LCoS if an unpolarized light source is employed.

MEMS-based LBS 128 , 129 utilizes micromirrors to directly scan RGB laser beams to form two-dimensional (2D) images (Fig. 7f ). Different gray levels are achieved by pulse width modulation (PWM) of the employed laser diodes. In practice, 2D scanning can be achieved either through a 2D scanning mirror or two 1D scanning mirrors with an additional focusing lens after the first mirror. The small size of MEMS mirror offers a very attractive form factor. At the same time, the output image has a large depth-of-focus (DoF), which is ideal for projection displays. One shortcoming, though, is that the small system etendue often hinders its applications in some traditional display systems.

Comparison of light engine performance

There are several important parameters for a light engine, including image resolution, brightness, frame rate, contrast ratio, and form factor. The resolution requirement (>2K) is similar for all types of light engines. The improvement of resolution is usually accomplished through the manufacturing process. Thus, here we shall focus on other three parameters.

Image brightness usually refers to the measured luminance of a light-emitting object. This measurement, however, may not be accurate for a light engine as the light from engine only forms an intermediate image, which is not directly viewed by the user. On the other hand, to solely focus on the brightness of a light engine could be misleading for a wearable display system like AR. Nowadays, data projectors with thousands of lumens are available. But the power consumption is too high for a battery-powered wearable AR display. Therefore, a more appropriate way to evaluate a light engine’s brightness is to use luminous efficacy (lm/W) measured by dividing the final output luminous flux (lm) by the input electric power (W). For a self-emissive device like micro-LED or micro-OLED, the luminous efficacy is directly determined by the device itself. However, for LCoS and DMD, the overall luminous efficacy should take into consideration the light source luminous efficacy, the efficiency of illumination optics, and the efficiency of the employed spatial light modulator (SLM). For a MEMS LBS engine, the efficiency of MEMS mirror can be considered as unity so that the luminous efficacy basically equals to that of the employed laser sources.

As mentioned earlier, each light engine has a different scheme for generating color images. Therefore, we separately list luminous efficacy of each scheme for a more inclusive comparison. For micro-LEDs, the situation is more complicated because the EQE depends on the chip size. Based on previous studies 130 , 131 , 132 , 133 , we separately calculate the luminous efficacy for RGB micro-LEDs with chip size ≈ 20 µm. For the scheme of direct combination of RGB micro-LEDs, the luminous efficacy is around 5 lm/W. For QD-conversion with blue micro-LEDs, the luminous efficacy is around 10 lm/W with the assumption of 100% color conversion efficiency, which has been demonstrated using structure engineering 114 . For micro-OLEDs, the calculated luminous efficacy is about 4–8 lm/W 120 , 122 . However, the lifetime and EQE of blue OLED materials depend on the driving current. To continuously display an image with brightness higher than 10,000 nits may dramatically shorten the device lifetime. The reason we compare the light engine at 10,000 nits is that it is highly desirable to obtain 1000 nits for the displayed image in order to keep ACR>3:1 with a typical AR combiner whose optical efficiency is lower than 10%.

For an LCoS engine using a white LED as light source, the typical optical efficiency of the whole engine is around 10% 127 , 134 . Then the engine luminous efficacy is estimated to be 12 lm/W with a 120 lm/W white LED source. For a color sequential LCoS using RGB LEDs, the absorption loss from color filters is eliminated, but the luminous efficacy of RGB LED source is also decreased to about 30 lm/W due to lower efficiency of red and green LEDs and higher driving current 135 . Therefore, the final luminous efficacy of the color sequential LCoS engine is also around 10 lm/W. If RGB linearly polarized lasers are employed instead of LEDs, then the LCoS engine efficiency can be quite high due to the high degree of collimation. The luminous efficacy of RGB laser source is around 40 lm/W 136 . Therefore, the laser-based LCoS engine is estimated to have a luminous efficacy of 32 lm/W, assuming the engine optical efficiency is 80%. For a DMD engine with RGB LEDs as light source, the optical efficiency is around 50% 137 , 138 , which leads to a luminous efficacy of 15 lm/W. By switching to laser light sources, the situation is similar to LCoS, with the luminous efficacy of about 32 lm/W. Finally, for MEMS-based LBS engine, there is basically no loss from the optics so that the final luminous efficacy is 40 lm/W. Detailed calculations of luminous efficacy can be found in Supplementary Information .

Another aspect of a light engine is the frame rate, which determines the volume of information it can deliver in a unit time. A high volume of information is vital for the construction of a 3D light field to solve the VAC issue. For micro-LEDs, the device response time is around several nanoseconds, which allows for visible light communication with bandwidth up to 1.5 Gbit/s 139 . For an OLED microdisplay, a fast OLED with ~200 MHz bandwidth has been demonstrated 140 . Therefore, the limitation of frame rate is on the driving circuits for both micro-LED and OLED. Another fact concerning driving circuit is the tradeoff between resolution and frame rate as a higher resolution panel means more scanning lines in each frame. So far, an OLED display with 480 Hz frame rate has been demonstrated 141 . For an LCoS, the frame rate is mainly limited by the LC response time. Depending on the LC material used, the response time is around 1 ms for nematic LC or 200 µs for ferroelectric LC (FLC) 125 . Nematic LC allows analog driving, which accommodates gray levels, typically with 8-bit depth. FLC is bistable so that PWM is used to generate gray levels. DMD is also a binary device. The frame rate can reach 30 kHz, which is mainly constrained by the response time of micromirrors. For MEMS-based LBS, the frame rate is limited by the scanning frequency of MEMS mirrors. A frame rate of 60 Hz with around 1 K resolution already requires a resonance frequency of around 50 kHz, with a Q-factor up to 145,000 128 . A higher frame rate or resolution requires a higher Q-factor and larger laser modulation bandwidth, which may be challenging.

Form factor is another crucial aspect for the light engines of near-eye displays. For self-emissive displays, both micro-OLEDs and QD-based micro-LEDs can achieve full color with a single panel. Thus, they are quite compact. A micro-LED display with separate RGB panels naturally have a larger form factor. In applications requiring direct-view full-color panel, the extra combining optics may also increase the volume. It needs to be pointed out, however, that the combing optics may not be necessary for some applications like waveguide displays, because the EPE process results in system’s insensitivity to the spatial positions of input RGB images. Therefore, the form factor of using three RGB micro-LED panels is medium. For LCoS and DMD with RGB LEDs as light source, the form factor would be larger due to the illumination optics. Still, if a lower luminous efficacy can be accepted, then a smaller form factor can be achieved by using a simpler optics 142 . If RGB lasers are used, the collimation optics can be eliminated, which greatly reduces the form factor 143 . For MEMS-LBS, the form factor can be extremely compact due to the tiny size of MEMS mirror and laser module.

Finally, contrast ratio (CR) also plays an important role affecting the observed images 8 . Micro-LEDs and micro-OLEDs are self-emissive so that their CR can be >10 6 :1. For a laser beam scanner, its CR can also achieve 10 6 :1 because the laser can be turned off completely at dark state. On the other hand, LCoS and DMD are reflective displays, and their CR is around 2000:1 to 5000:1 144 , 145 . It is worth pointing out that the CR of a display engine plays a significant role only in the dark ambient. As the ambient brightness increases, the ACR is mainly governed by the display’s peak brightness, as previously discussed.

The performance parameters of different light engines are summarized in Table 1 . Micro-LEDs and micro-OLEDs have similar levels of luminous efficacy. But micro-OLEDs still face the burn-in and lifetime issue when driving at a high current, which hinders its use for a high-brightness image source to some extent. Micro-LEDs are still under active development and the improvement on luminous efficacy from maturing fabrication process could be expected. Both devices have nanosecond response time and can potentially achieve a high frame rate with a well-designed integrated circuit. The frame rate of the driving circuit ultimately determines the motion picture response time 146 . Their self-emissive feature also leads to a small form factor and high contrast ratio. LCoS and DMD engines have similar performance of luminous efficacy, form factor, and contrast ratio. In terms of light modulation, DMD can provide a higher 1-bit frame rate, while LCoS can offer both phase and amplitude modulations. MEMS-based LBS exhibits the highest luminous efficacy so far. It also exhibits an excellent form factor and contrast ratio, but the presently demonstrated 60-Hz frame rate (limited by the MEMS mirrors) could cause image flickering.

Free-space combiners

The term ‘free-space’ generally refers to the case when light is freely propagating in space, as opposed to a waveguide that traps light into TIRs. Regarding the combiner, it can be a partial mirror, as commonly used in AR systems based on traditional geometric optics. Alternatively, the combiner can also be a reflective HOE. The strong chromatic dispersion of HOE necessitates the use of a laser source, which usually leads to a Maxwellian-type system.

Traditional geometric designs

Several systems based on geometric optics are illustrated in Fig. 8 . The simplest design uses a single freeform half-mirror 6 , 147 to directly collimate the displayed images to the viewer’s eye (Fig. 8a ). This design can achieve a large FoV (up to 90°) 147 , but the limited design freedom with a single freeform surface leads to image distortions, also called pupil swim 6 . The placement of half-mirror also results in a relatively bulky form factor. Another design using so-called birdbath optics 6 , 148 is shown in Fig. 8b . Compared to the single-combiner design, birdbath design has an extra optics on the display side, which provides space for aberration correction. The integration of beam splitter provides a folded optical path, which reduces the form factor to some extent. Another way to fold optical path is to use a TIR-prism. Cheng et al. 149 designed a freeform TIR-prism combiner (Fig. 8c ) offering a diagonal FoV of 54° and exit pupil diameter of 8 mm. All the surfaces are freeform, which offer an excellent image quality. To cancel the optical power for the transmitted environmental light, a compensator is added to the TIR prism. The whole system has a well-balanced performance between FoV, eyebox, and form factor. To release the space in front of viewer’s eye, relay optics can be used to form an intermediate image near the combiner 150 , 151 , as illustrated in Fig. 8d . Although the design offers more optical surfaces for aberration correction, the extra lenses also add to system weight and form factor.

a Single freeform surface as the combiner. b Birdbath optics with a beam splitter and a half mirror. c Freeform TIR prism with a compensator. d Relay optics with a half mirror. Adapted from c ref. 149 with permission from OSA Publishing and d ref. 151 with permission from OSA Publishing

Regarding the approaches to solve the VAC issue, the most straightforward way is to integrate a tunable lens into the optical path, like a liquid lens 152 or Alvarez lens 99 , to form a varifocal system. Alternatively, integral imaging 153 , 154 can also be used, by replacing the original display panel with the central depth plane of an integral imaging module. The integral imaging can also be combined with varifocal approach to overcome the tradeoff between resolution and depth of field (DoF) 155 , 156 , 157 . However, the inherent tradeoff between resolution and view number still exists in this case.

Overall, AR displays based on traditional geometric optics have a relatively simple design with a decent FoV (~60°) and eyebox (8 mm) 158 . They also exhibit a reasonable efficiency. To measure the efficiency of an AR combiner, an appropriate measure is to divide the output luminance (unit: nit) by the input luminous flux (unit: lm), which we note as combiner efficiency. For a fixed input luminous flux, the output luminance, or image brightness, is related to the FoV and exit pupil of the combiner system. If we assume no light waste of the combiner system, then the maximum combiner efficiency for a typical diagonal FoV of 60° and exit pupil (10 mm square) is around 17,000 nit/lm (Eq. S2 ). To estimate the combiner efficiency of geometric combiners, we assume 50% of half-mirror transmittance and the efficiency of other optics to be 50%. Then the final combiner efficiency is about 4200 nit/lm, which is a high value in comparison with waveguide combiners. Nonetheless, to further shrink the system size or improve system performance ultimately encounters the etendue conservation issue. In addition, AR systems with traditional geometric optics is hard to achieve a configuration resembling normal flat glasses because the half-mirror has to be tilted to some extent.

Maxwellian-type systems

The Maxwellian view, proposed by James Clerk Maxwell (1860), refers to imaging a point light source in the eye pupil 159 . If the light beam is modulated in the imaging process, a corresponding image can be formed on the retina (Fig. 9a ). Because the point source is much smaller than the eye pupil, the image is always-in-focus on the retina irrespective of the eye lens’ focus. For applications in AR display, the point source is usually a laser with narrow angular and spectral bandwidths. LED light sources can also build a Maxwellian system, by adding an angular filtering module 160 . Regarding the combiner, although in theory a half-mirror can also be used, HOEs are generally preferred because they offer the off-axis configuration that places combiner in a similar position like eyeglasses. In addition, HOEs have a lower reflection of environment light, which provides a more natural appearance of the user behind the display.

a Schematic of the working principle of Maxwellian displays. Maxwellian displays based on b SLM and laser diode light source and c MEMS-LBS with a steering mirror as additional modulation method. Generation of depth cues by d computational digital holography and e scanning of steering mirror to produce multiple views. Adapted from b, d ref. 143 and c, e ref. 167 under the Creative Commons Attribution 4.0 License

To modulate the light, a SLM like LCoS or DMD can be placed in the light path, as shown in Fig. 9b . Alternatively, LBS system can also be used (Fig. 9c ), where the intensity modulation occurs in the laser diode itself. Besides the operation in a normal Maxwellian-view, both implementations offer additional degrees of freedom for light modulation.

For a SLM-based system, there are several options to arrange the SLM pixels 143 , 161 . Maimone et al. 143 demonstrated a Maxwellian AR display with two modes to offer a large-DoF Maxwellian-view, or a holographic view (Fig. 9d ), which is often referred as computer-generated holography (CGH) 162 . To show an always-in-focus image with a large DoF, the image can be directly displayed on an amplitude SLM, or using amplitude encoding for a phase-only SLM 163 . Alternatively, if a 3D scene with correct depth cues is to be presented, then optimization algorithms for CGH can be used to generate a hologram for the SLM. The generated holographic image exhibits the natural focus-and-blur effect like a real 3D object (Fig. 9d ). To better understand this feature, we need to again exploit the concept of etendue. The laser light source can be considered to have a very small etendue due to its excellent collimation. Therefore, the system etendue is provided by the SLM. The micron-sized pixel-pitch of SLM offers a certain maximum diffraction angle, which, multiplied by the SLM size, equals system etendue. By varying the display content on SLM, the final exit pupil size can be changed accordingly. In the case of a large-DoF Maxwellian view, the exit pupil size is small, accompanied by a large FoV. For the holographic display mode, the reduced DoF requires a larger exit pupil with dimension close to the eye pupil. But the FoV is reduced accordingly due to etendue conservation. Another commonly concerned issue with CGH is the computation time. To achieve a real-time CGH rendering flow with an excellent image quality is quite a challenge. Fortunately, with recent advances in algorithm 164 and the introduction of convolutional neural network (CNN) 165 , 166 , this issue is gradually solved with an encouraging pace. Lately, Liang et al. 166 demonstrated a real-time CGH synthesis pipeline with a high image quality. The pipeline comprises an efficient CNN model to generate a complex hologram from a 3D scene and an improved encoding algorithm to convert the complex hologram to a phase-only one. An impressive frame rate of 60 Hz has been achieved on a desktop computing unit.

For LBS-based system, the additional modulation can be achieved by integrating a steering module, as demonstrated by Jang et al. 167 . The steering mirror can shift the focal point (viewpoint) within the eye pupil, therefore effectively expanding the system etendue. When the steering process is fast and the image content is updated simultaneously, correct 3D cues can be generated, as shown in Fig. 9e . However, there exists a tradeoff between the number of viewpoint and the final image frame rate, because the total frames are equally divided into each viewpoint. To boost the frame rate of MEMS-LBS systems by the number of views (e.g., 3 by 3) may be challenging.

Maxwellian-type systems offer several advantages. The system efficiency is usually very high because nearly all the light is delivered into viewer’s eye. The system FoV is determined by the f /# of combiner and a large FoV (~80° in horizontal) can be achieved 143 . The issue of VAC can be mitigated with an infinite-DoF image that deprives accommodation cue, or completely solved by generating a true-3D scene as discussed above. Despite these advantages, one major weakness of Maxwellian-type system is the tiny exit pupil, or eyebox. A small deviation of eye pupil location from the viewpoint results in the complete disappearance of the image. Therefore, to expand eyebox is considered as one of the most important challenges in Maxwellian-type systems.

Pupil duplication and steering

Methods to expand eyebox can be generally categorized into pupil duplication 168 , 169 , 170 , 171 , 172 and pupil steering 9 , 13 , 167 , 173 . Pupil duplication simply generates multiple viewpoints to cover a large area. In contrast, pupil steering dynamically shifts the viewpoint position, depending on the pupil location. Before reviewing detailed implementations of these two methods, it is worth discussing some of their general features. The multiple viewpoints in pupil duplication usually mean to equally divide the total light intensity. In each time frame, however, it is preferable that only one viewpoint enters the user’s eye pupil to avoid ghost image. This requirement, therefore, results in a reduced total light efficiency, while also conditioning the viewpoint separation to be larger than the pupil diameter. In addition, the separation should not be too large to avoid gap between viewpoints. Considering that human pupil diameter changes in response to environment illuminance, the design of viewpoint separation needs special attention. Pupil steering, on the other hand, only produces one viewpoint at each time frame. It is therefore more light-efficient and free from ghost images. But to determine the viewpoint position requires the information of eye pupil location, which demands a real-time eye-tracking module 9 . Another observation is that pupil steering can accommodate multiple viewpoints by its nature. Therefore, a pupil steering system can often be easily converted to a pupil duplication system by simultaneously generating available viewpoints.

To generate multiple viewpoints, one can focus on modulating the incident light or the combiner. Recall that viewpoint is the image of light source. To duplicate or shift light source can achieve pupil duplication or steering accordingly, as illustrated in Fig. 10a . Several schemes of light modulation are depicted in Fig. 10b–e . An array of light sources can be generated with multiple laser diodes (Fig. 10b ). To turn on all or one of the sources achieves pupil duplication or steering. A light source array can also be produced by projecting light on an array-type PPHOE 168 (Fig. 10c ). Apart from direct adjustment of light sources, modulating light on the path can also effectively steer/duplicate the light sources. Using a mechanical steering mirror, the beam can be deflected 167 (Fig. 10d ), which equals to shifting the light source position. Other devices like a grating or beam splitter can also serve as ray deflector/splitter 170 , 171 (Fig. 10e ).

a Schematic of duplicating (or shift) viewpoint by modulation of incident light. Light modulation by b multiple laser diodes, c HOE lens array, d steering mirror and e grating or beam splitters. f Pupil duplication with multiplexed PPHOE. g Pupil steering with LCHOE. Reproduced from c ref. 168 under the Creative Commons Attribution 4.0 License, e ref. 169 with permission from OSA Publishing, f ref. 171 with permission from OSA Publishing and g ref. 173 with permission from OSA Publishing

Nonetheless, one problem of the light source duplication/shifting methods for pupil duplication/steering is that the aberrations in peripheral viewpoints are often serious 168 , 173 . The HOE combiner is usually recorded at one incident angle. For other incident angles with large deviations, considerable aberrations will occur, especially in the scenario of off-axis configuration. To solve this problem, the modulation can be focused on the combiner instead. While the mechanical shifting of combiner 9 can achieve continuous pupil steering, its integration into AR display with a small factor remains a challenge. Alternatively, the versatile functions of HOE offer possible solutions for combiner modulation. Kim and Park 169 demonstrated a pupil duplication system with multiplexed PPHOE (Fig. 10f ). Wavefronts of several viewpoints can be recorded into one PPHOE sample. Three viewpoints with a separation of 3 mm were achieved. However, a slight degree of ghost image and gap can be observed in the viewpoint transition. For a PPHOE to achieve pupil steering, the multiplexed PPHOE needs to record different focal points with different incident angles. If each hologram has no angular crosstalk, then with an additional device to change the light incident angle, the viewpoint can be steered. Alternatively, Xiong et al. 173 demonstrated a pupil steering system with LCHOEs in a simpler configuration (Fig. 10g ). The polarization-sensitive nature of LCHOE enables the controlling of which LCHOE to function with a polarization converter (PC). When the PC is off, the incident RCP light is focused by the right-handed LCHOE. When the PC is turned on, the RCP light is firstly converted to LCP light and passes through the right-handed LCHOE. Then it is focused by the left-handed LCHOE into another viewpoint. To add more viewpoints requires stacking more pairs of PC and LCHOE, which can be achieved in a compact manner with thin glass substrates. In addition, to realize pupil duplication only requires the stacking of multiple low-efficiency LCHOEs. For both PPHOEs and LCHOEs, because the hologram for each viewpoint is recorded independently, the aberrations can be eliminated.

Regarding the system performance, in theory the FoV is not limited and can reach a large value, such as 80° in horizontal direction 143 . The definition of eyebox is different from traditional imaging systems. For a single viewpoint, it has the same size as the eye pupil diameter. But due to the viewpoint steering/duplication capability, the total system eyebox can be expanded accordingly. The combiner efficiency for pupil steering systems can reach 47,000 nit/lm for a FoV of 80° by 80° and pupil diameter of 4 mm (Eq. S2 ). At such a high brightness level, eye safety could be a concern 174 . For a pupil duplication system, the combiner efficiency is decreased by the number of viewpoints. With a 4-by-4 viewpoint array, it can still reach 3000 nit/lm. Despite the potential gain of pupil duplication/steering, when considering the rotation of eyeball, the situation becomes much more complicated 175 . A perfect pupil steering system requires a 5D steering, which proposes a challenge for practical implementation.

Pin-light systems

Recently, another type of display in close relation with Maxwellian view called pin-light display 148 , 176 has been proposed. The general working principle of pin-light display is illustrated in Fig. 11a . Each pin-light source is a Maxwellian view with a large DoF. When the eye pupil is no longer placed near the source point as in Maxwellian view, each image source can only form an elemental view with a small FoV on retina. However, if the image source array is arranged in a proper form, the elemental views can be integrated together to form a large FoV. According to the specific optical architectures, pin-light display can take different forms of implementation. In the initial feasibility demonstration, Maimone et al. 176 used a side-lit waveguide plate as the point light source (Fig. 11b ). The light inside the waveguide plate is extracted by the etched divots, forming a pin-light source array. A transmissive SLM (LCD) is placed behind the waveguide plate to modulate the light intensity and form the image. The display has an impressive FoV of 110° thanks to the large scattering angle range. However, the direct placement of LCD before the eye brings issues of insufficient resolution density and diffraction of background light.

a Schematic drawing of the working principle of pin-light display. b Pin-light display utilizing a pin-light source and a transmissive SLM. c An example of pin-mirror display with a birdbath optics. d SWD system with LBS image source and off-axis lens array. Reprinted from b ref. 176 under the Creative Commons Attribution 4.0 License and d ref. 180 with permission from OSA Publishing

To avoid these issues, architectures using pin-mirrors 177 , 178 , 179 are proposed. In these systems, the final combiner is an array of tiny mirrors 178 , 179 or gratings 177 , in contrast to their counterparts using large-area combiners. An exemplary system with birdbath design is depicted in Fig. 11c . In this case, the pin-mirrors replace the original beam-splitter in the birdbath and can thus shrink the system volume, while at the same time providing large DoF pin-light images. Nonetheless, such a system may still face the etendue conservation issue. Meanwhile, the size of pin-mirror cannot be too small in order to prevent degradation of resolution density due to diffraction. Therefore, its influence on the see-through background should also be considered in the system design.

To overcome the etendue conservation and improve see-through quality, Xiong et al. 180 proposed another type of pin-light system exploiting the etendue expansion property of waveguide, which is also referred as scanning waveguide display (SWD). As illustrated in Fig. 11d , the system uses an LBS as the image source. The collimated scanned laser rays are trapped in the waveguide and encounter an array of off-axis lenses. Upon each encounter, the lens out-couples the laser rays and forms a pin-light source. SWD has the merits of good see-through quality and large etendue. A large FoV of 100° was demonstrated with the help of an ultra-low f /# lens array based on LCHOE. However, some issues like insufficient image resolution density and image non-uniformity remain to be overcome. To further improve the system may require optimization of Gaussian beam profile and additional EPE module 180 .

Overall, pin-light systems inherit the large DoF from Maxwellian view. With adequate number of pin-light sources, the FoV and eyebox can be expanded accordingly. Nonetheless, despite different forms of implementation, a common issue of pin-light system is the image uniformity. The overlapped region of elemental views has a higher light intensity than the non-overlapped region, which becomes even more complicated considering the dynamic change of pupil size. In theory, the displayed image can be pre-processed to compensate for the optical non-uniformity. But that would require knowledge of precise pupil location (and possibly size) and therefore an accurate eye-tracking module 176 . Regarding the system performance, pin-mirror systems modified from other free-space systems generally shares similar FoV and eyebox with original systems. The combiner efficiency may be lower due to the small size of pin-mirrors. SWD, on the other hand, shares the large FoV and DoF with Maxwellian view, and large eyebox with waveguide combiners. The combiner efficiency may also be lower due to the EPE process.

Waveguide combiner

Besides free-space combiners, another common architecture in AR displays is waveguide combiner. The term ‘waveguide’ indicates the light is trapped in a substrate by the TIR process. One distinctive feature of a waveguide combiner is the EPE process that effectively enlarges the system etendue. In the EPE process, a portion of the trapped light is repeatedly coupled out of the waveguide in each TIR. The effective eyebox is therefore enlarged. According to the features of couplers, we divide the waveguide combiners into two types: diffractive and achromatic, as described in the followings.

Diffractive waveguides

As the name implies, diffractive-type waveguides use diffractive elements as couplers. The in-coupler is usually a diffractive grating and the out-coupler in most cases is also a grating with the same period as the in-coupler, but it can also be an off-axis lens with a small curvature to generate image with finite depth. Three major diffractive couplers have been developed: SRGs, photopolymer gratings (PPGs), and liquid crystal gratings (grating-type LCHOE; also known as polarization volume gratings (PVGs)). Some general protocols for coupler design are that the in-coupler should have a relatively high efficiency and the out-coupler should have a uniform light output. A uniform light output usually requires a low-efficiency coupler, with extra degrees of freedom for local modulation of coupling efficiency. Both in-coupler and out-coupler should have an adequate angular bandwidth to accommodate a reasonable FoV. In addition, the out-coupler should also be optimized to avoid undesired diffractions, including the outward diffraction of TIR light and diffraction of environment light into user’s eyes, which are referred as light leakage and rainbow. Suppression of these unwanted diffractions should also be considered in the optimization process of waveguide design, along with performance parameters like efficiency and uniformity.

The basic working principles of diffractive waveguide-based AR systems are illustrated in Fig. 12 . For the SRG-based waveguides 6 , 8 (Fig. 12a ), the in-coupler can be a transmissive-type or a reflective-type 181 , 182 . The grating geometry can be optimized for coupling efficiency with a large degree of freedom 183 . For the out-coupler, a reflective SRG with a large slant angle to suppress the transmission orders is preferred 184 . In addition, a uniform light output usually requires a gradient efficiency distribution in order to compensate for the decreased light intensity in the out-coupling process. This can be achieved by varying the local grating configurations like height and duty cycle 6 . For the PPG-based waveguides 185 (Fig. 12b ), the small angular bandwidth of a high-efficiency transmissive PPG prohibits its use as in-coupler. Therefore, both in-coupler and out-coupler are usually reflective types. The gradient efficiency can be achieved by space-variant exposure to control the local index modulation 186 or local Bragg slant angle variation through freeform exposure 19 . Due to the relatively small angular bandwidth of PPG, to achieve a decent FoV usually requires stacking two 187 or three 188 PPGs together for a single color. The PVG-based waveguides 189 (Fig. 12c ) also prefer reflective PVGs as in-couplers because the transmissive PVGs are much more difficult to fabricate due to the LC alignment issue. In addition, the angular bandwidth of transmissive PVGs in Bragg regime is also not large enough to support a decent FoV 29 . For the out-coupler, the angular bandwidth of a single reflective PVG can usually support a reasonable FoV. To obtain a uniform light output, a polarization management layer 190 consisting of a LC layer with spatially variant orientations can be utilized. It offers an additional degree of freedom to control the polarization state of the TIR light. The diffraction efficiency can therefore be locally controlled due to the strong polarization sensitivity of PVG.

Schematics of waveguide combiners based on a SRGs, b PPGs and c PVGs. Reprinted from a ref. 85 with permission from OSA Publishing, b ref. 185 with permission from John Wiley and Sons and c ref. 189 with permission from OSA Publishing

The above discussion describes the basic working principle of 1D EPE. Nonetheless, for the 1D EPE to produce a large eyebox, the exit pupil in the unexpanded direction of the original image should be large. This proposes design challenges in light engines. Therefore, a 2D EPE is favored for practical applications. To extend EPE in two dimensions, two consecutive 1D EPEs can be used 191 , as depicted in Fig. 13a . The first 1D EPE occurs in the turning grating, where the light is duplicated in y direction and then turned into x direction. Then the light rays encounter the out-coupler and are expanded in x direction. To better understand the 2D EPE process, the k -vector diagram (Fig. 13b ) can be used. For the light propagating in air with wavenumber k 0 , its possible k -values in x and y directions ( k x and k y ) fall within the circle with radius k 0 . When the light is trapped into TIR, k x and k y are outside the circle with radius k 0 and inside the circle with radius nk 0 , where n is the refractive index of the substrate. k x and k y stay unchanged in the TIR process and are only changed in each diffraction process. The central red box in Fig. 13b indicates the possible k values within the system FoV. After the in-coupler, the k values are added by the grating k -vector, shifting the k values into TIR region. The turning grating then applies another k -vector and shifts the k values to near x -axis. Finally, the k values are shifted by the out-coupler and return to the free propagation region in air. One observation is that the size of red box is mostly limited by the width of TIR band. To accommodate a larger FoV, the outer boundary of TIR band needs to be expanded, which amounts to increasing waveguide refractive index. Another important fact is that when k x and k y are near the outer boundary, the uniformity of output light becomes worse. This is because the light propagation angle is near 90° in the waveguide. The spatial distance between two consecutive TIRs becomes so large that the out-coupled beams are spatially separated to an unacceptable degree. The range of possible k values for practical applications is therefore further shrunk due to this fact.

a Schematic of 2D EPE based on two consecutive 1D EPEs. Gray/black arrows indicate light in air/TIR. Black dots denote TIRs. b k-diagram of the two-1D-EPE scheme. c Schematic of 2D EPE with a 2D hexagonal grating d k-diagram of the 2D-grating scheme

Aside from two consecutive 1D EPEs, the 2D EPE can also be directly implemented with a 2D grating 192 . An example using a hexagonal grating is depicted in Fig. 13c . The hexagonal grating can provide k -vectors in six directions. In the k -diagram (Fig. 13d ), after the in-coupling, the k values are distributed into six regions due to multiple diffractions. The out-coupling occurs simultaneously with pupil expansion. Besides a concise out-coupler configuration, the 2D EPE scheme offers more degrees of design freedom than two 1D EPEs because the local grating parameters can be adjusted in a 2D manner. The higher design freedom has the potential to reach a better output light uniformity, but at the cost of a higher computation demand for optimization. Furthermore, the unslanted grating geometry usually leads to a large light leakage and possibly low efficiency. Adding slant to the geometry helps alleviate the issue, but the associated fabrication may be more challenging.

Finally, we discuss the generation of full-color images. One important issue to clarify is that although diffractive gratings are used here, the final image generally has no color dispersion even if we use a broadband light source like LED. This can be easily understood in the 1D EPE scheme. The in-coupler and out-coupler have opposite k -vectors, which cancels the color dispersion for each other. In the 2D EPE schemes, the k -vectors always form a closed loop from in-coupled light to out-coupled light, thus, the color dispersion also vanishes likewise. The issue of using a single waveguide for full-color images actually exists in the consideration of FoV and light uniformity. The breakup of propagation angles for different colors results in varied out-coupling situations for each color. To be more specific, if the red and the blue channels use the same in-coupler, the propagating angle for the red light is larger than that of the blue light. The red light in peripheral FoV is therefore easier to face the mentioned large-angle non-uniformity issue. To acquire a decent FoV and light uniformity, usually two or three layers of waveguides with different grating pitches are adopted.

Regarding the system performance, the eyebox is generally large enough (~10 mm) to accommodate different user’s IPD and alignment shift during operation. A parameter of significant concern for a waveguide combiner is its FoV. From the k -vector analysis, we can conclude the theoretical upper limit is determined by the waveguide refractive index. But the light/color uniformity also influences the effective FoV, over which the degradation of image quality becomes unacceptable. Current diffractive waveguide combiners generally achieve a FoV of about 50°. To further increase FoV, a straightforward method is to use a higher refractive index waveguide. Another is to tile FoV through direct stacking of multiple waveguides or using polarization-sensitive couplers 79 , 193 . As to the optical efficiency, a typical value for the diffractive waveguide combiner is around 50–200 nit/lm 6 , 189 . In addition, waveguide combiners adopting grating out-couplers generate an image with fixed depth at infinity. This leads to the VAC issue. To tackle VAC in waveguide architectures, the most practical way is to generate multiple depths and use the varifocal or multifocal driving scheme, similar to those mentioned in the VR systems. But to add more depths usually means to stack multiple layers of waveguides together 194 . Considering the additional waveguide layers for RGB colors, the final waveguide thickness would undoubtedly increase.

Other parameters special to waveguide includes light leakage, see-through ghost, and rainbow. Light leakage refers to out-coupled light that goes outwards to the environment, as depicted in Fig. 14a . Aside from decreased efficiency, the leakage also brings drawback of unnatural “bright-eye” appearance of the user and privacy issue. Optimization of the grating structure like geometry of SRG may reduce the leakage. See-through ghost is formed by consecutive in-coupling and out-couplings caused by the out-coupler grating, as sketched in Fig. 14b , After the process, a real object with finite depth may produce a ghost image with shift in both FoV and depth. Generally, an out-coupler with higher efficiency suffers more see-through ghost. Rainbow is caused by the diffraction of environment light into user’s eye, as sketched in Fig. 14c . The color dispersion in this case will occur because there is no cancellation of k -vector. Using the k -diagram, we can obtain a deeper insight into the formation of rainbow. Here, we take the EPE structure in Fig. 13a as an example. As depicted in Fig. 14d , after diffractions by the turning grating and the out-coupler grating, the k values are distributed in two circles that shift from the origin by the grating k -vectors. Some diffracted light can enter the see-through FoV and form rainbow. To reduce rainbow, a straightforward way is to use a higher index substrate. With a higher refractive index, the outer boundary of k diagram is expanded, which can accommodate larger grating k -vectors. The enlarged k -vectors would therefore “push” these two circles outwards, leading to a decreased overlapping region with the see-through FoV. Alternatively, an optimized grating structure would also help reduce the rainbow effect by suppressing the unwanted diffraction.

Sketches of formations of a light leakage, b see-through ghost and c rainbow. d Analysis of rainbow formation with k-diagram

Achromatic waveguide

Achromatic waveguide combiners use achromatic elements as couplers. It has the advantage of realizing full-color image with a single waveguide. A typical example of achromatic element is a mirror. The waveguide with partial mirrors as out-coupler is often referred as geometric waveguide 6 , 195 , as depicted in Fig. 15a . The in-coupler in this case is usually a prism to avoid unnecessary color dispersion if using diffractive elements otherwise. The mirrors couple out TIR light consecutively to produce a large eyebox, similarly in a diffractive waveguide. Thanks to the excellent optical property of mirrors, the geometric waveguide usually exhibits a superior image regarding MTF and color uniformity to its diffractive counterparts. Still, the spatially discontinuous configuration of mirrors also results in gaps in eyebox, which may be alleviated by using a dual-layer structure 196 . Wang et al. designed a geometric waveguide display with five partial mirrors (Fig. 15b ). It exhibits a remarkable FoV of 50° by 30° (Fig. 15c ) and an exit pupil of 4 mm with a 1D EPE. To achieve 2D EPE, similar architectures in Fig. 13a can be used by integrating a turning mirror array as the first 1D EPE module 197 . Unfortunately, the k -vector diagrams in Fig. 13b, d cannot be used here because the k values in x-y plane no longer conserve in the in-coupling and out-coupling processes. But some general conclusions remain valid, like a higher refractive index leading to a larger FoV and gradient out-coupling efficiency improving light uniformity.

a Schematic of the system configuration. b Geometric waveguide with five partial mirrors. c Image photos demonstrating system FoV. Adapted from b , c ref. 195 with permission from OSA Publishing

The fabrication process of geometric waveguide involves coating mirrors on cut-apart pieces and integrating them back together, which may result in a high cost, especially for the 2D EPE architecture. Another way to implement an achromatic coupler is to use multiplexed PPHOE 198 , 199 to mimic the behavior of a tilted mirror (Fig. 16a ). To understand the working principle, we can use the diagram in Fig. 16b . The law of reflection states the angle of reflection equals to the angle of incidence. If we translate this behavior to k -vector language, it means the mirror can apply any length of k -vector along its surface normal direction. The k -vector length of the reflected light is always equal to that of the incident light. This puts a condition that the k -vector triangle is isosceles. With a simple geometric deduction, it can be easily observed this leads to the law of reflection. The behavior of a general grating, however, is very different. For simplicity we only consider the main diffraction order. The grating can only apply a k -vector with fixed k x due to the basic diffraction law. For the light with a different incident angle, it needs to apply different k z to produce a diffracted light with equal k -vector length as the incident light. For a grating with a broad angular bandwidth like SRG, the range of k z is wide, forming a lengthy vertical line in Fig. 16b . For a PPG with a narrow angular bandwidth, the line is short and resembles a dot. If multiple of these tiny dots are distributed along the oblique line corresponding to a mirror, then the final multiplexed PPGs can imitate the behavior of a tilted mirror. Such a PPHOE is sometimes referred as a skew-mirror 198 . In theory, to better imitate the mirror, a lot of multiplexed PPGs is preferred, while each PPG has a small index modulation δn . But this proposes a bigger challenge in device fabrication. Recently, Utsugi et al. demonstrated an impressive skew-mirror waveguide based on 54 multiplexed PPGs (Fig. 16c, d ). The display exhibits an effective FoV of 35° by 36°. In the peripheral FoV, there still exists some non-uniformity (Fig. 16e ) due to the out-coupling gap, which is an inherent feature of the flat-type out-couplers.

a System configuration. b Diagram demonstrating how multiplexed PPGs resemble the behavior of a mirror. Photos showing c the system and d image. e Picture demonstrating effective system FoV. Adapted from c – e ref. 199 with permission from ITE

Finally, it is worth mentioning that metasurfaces are also promising to deliver achromatic gratings 200 , 201 for waveguide couplers ascribed to their versatile wavefront shaping capability. The mechanism of the achromatic gratings is similar to that of the achromatic lenses as previously discussed. However, the current development of achromatic metagratings is still in its infancy. Much effort is needed to improve the optical efficiency for in-coupling, control the higher diffraction orders for eliminating ghost images, and enable a large size design for EPE.

Generally, achromatic waveguide combiners exhibit a comparable FoV and eyebox with diffractive combiners, but with a higher efficiency. For a partial-mirror combiner, its combiner efficiency is around 650 nit/lm 197 (2D EPE). For a skew-mirror combiner, although the efficiency of multiplexed PPHOE is relatively low (~1.5%) 199 , the final combiner efficiency of the 1D EPE system is still high (>3000 nit/lm) due to multiple out-couplings.

Table 2 summarizes the performance of different AR combiners. When combing the luminous efficacy in Table 1 and the combiner efficiency in Table 2 , we can have a comprehensive estimate of the total luminance efficiency (nit/W) for different types of systems. Generally, Maxwellian-type combiners with pupil steering have the highest luminance efficiency when partnered with laser-based light engines like laser-backlit LCoS/DMD or MEM-LBS. Geometric optical combiners have well-balanced image performances, but to further shrink the system size remains a challenge. Diffractive waveguides have a relatively low combiner efficiency, which can be remedied by an efficient light engine like MEMS-LBS. Further development of coupler and EPE scheme would also improve the system efficiency and FoV. Achromatic waveguides have a decent combiner efficiency. The single-layer design also enables a smaller form factor. With advances in fabrication process, it may become a strong contender to presently widely used diffractive waveguides.

Conclusions and perspectives

VR and AR are endowed with a high expectation to revolutionize the way we interact with digital world. Accompanied with the expectation are the engineering challenges to squeeze a high-performance display system into a tightly packed module for daily wearing. Although the etendue conservation constitutes a great obstacle on the path, remarkable progresses with innovative optics and photonics continue to take place. Ultra-thin optical elements like PPHOEs and LCHOEs provide alternative solutions to traditional optics. Their unique features of multiplexing capability and polarization dependency further expand the possibility of novel wavefront modulations. At the same time, nanoscale-engineered metasurfaces/SRGs provide large design freedoms to achieve novel functions beyond conventional geometric optical devices. Newly emerged micro-LEDs open an opportunity for compact microdisplays with high peak brightness and good stability. Further advances on device engineering and manufacturing process are expected to boost the performance of metasurfaces/SRGs and micro-LEDs for AR and VR applications.

Data availability

All data needed to evaluate the conclusions in the paper are present in the paper. Additional data related to this paper may be requested from the authors.

Cakmakci, O. & Rolland, J. Head-worn displays: a review. J. Disp. Technol. 2 , 199–216 (2006).

Article   ADS   Google Scholar  

Zhan, T. et al. Augmented reality and virtual reality displays: perspectives and challenges. iScience 23 , 101397 (2020).

Rendon, A. A. et al. The effect of virtual reality gaming on dynamic balance in older adults. Age Ageing 41 , 549–552 (2012).

Article   Google Scholar  

Choi, S., Jung, K. & Noh, S. D. Virtual reality applications in manufacturing industries: past research, present findings, and future directions. Concurrent Eng. 23 , 40–63 (2015).

Li, X. et al. A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Autom. Constr. 86 , 150–162 (2018).

Kress, B. C. Optical Architectures for Augmented-, Virtual-, and Mixed-Reality Headsets (Bellingham: SPIE Press, 2020).

Cholewiak, S. A. et al. A perceptual eyebox for near-eye displays. Opt. Express 28 , 38008–38028 (2020).

Lee, Y. H., Zhan, T. & Wu, S. T. Prospects and challenges in augmented reality displays. Virtual Real. Intell. Hardw. 1 , 10–20 (2019).

Kim, J. et al. Foveated AR: dynamically-foveated augmented reality display. ACM Trans. Graph. 38 , 99 (2019).

Tan, G. J. et al. Foveated imaging for near-eye displays. Opt. Express 26 , 25076–25085 (2018).

Lee, S. et al. Foveated near-eye display for mixed reality using liquid crystal photonics. Sci. Rep. 10 , 16127 (2020).

Yoo, C. et al. Foveated display system based on a doublet geometric phase lens. Opt. Express 28 , 23690–23702 (2020).

Akşit, K. et al. Manufacturing application-driven foveated near-eye displays. IEEE Trans. Vis. Computer Graph. 25 , 1928–1939 (2019).

Zhu, R. D. et al. High-ambient-contrast augmented reality with a tunable transmittance liquid crystal film and a functional reflective polarizer. J. Soc. Inf. Disp. 24 , 229–233 (2016).

Lincoln, P. et al. Scene-adaptive high dynamic range display for low latency augmented reality. In Proc. 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games . (ACM, San Francisco, CA, 2017).

Duerr, F. & Thienpont, H. Freeform imaging systems: fermat’s principle unlocks “first time right” design. Light.: Sci. Appl. 10 , 95 (2021).

Bauer, A., Schiesser, E. M. & Rolland, J. P. Starting geometry creation and design method for freeform optics. Nat. Commun. 9 , 1756 (2018).

Rolland, J. P. et al. Freeform optics for imaging. Optica 8 , 161–176 (2021).

Jang, C. et al. Design and fabrication of freeform holographic optical elements. ACM Trans. Graph. 39 , 184 (2020).

Gabor, D. A new microscopic principle. Nature 161 , 777–778 (1948).

Kostuk, R. K. Holography: Principles and Applications (Boca Raton: CRC Press, 2019).

Lawrence, J. R., O'Neill, F. T. & Sheridan, J. T. Photopolymer holographic recording material. Optik 112 , 449–463 (2001).

Guo, J. X., Gleeson, M. R. & Sheridan, J. T. A review of the optimisation of photopolymer materials for holographic data storage. Phys. Res. Int. 2012 , 803439 (2012).

Jang, C. et al. Recent progress in see-through three-dimensional displays using holographic optical elements [Invited]. Appl. Opt. 55 , A71–A85 (2016).

Xiong, J. H. et al. Holographic optical elements for augmented reality: principles, present status, and future perspectives. Adv. Photonics Res. 2 , 2000049 (2021).

Tabiryan, N. V. et al. Advances in transparent planar optics: enabling large aperture, ultrathin lenses. Adv. Optical Mater. 9 , 2001692 (2021).

Zanutta, A. et al. Photopolymeric films with highly tunable refractive index modulation for high precision diffractive optics. Optical Mater. Express 6 , 252–263 (2016).

Moharam, M. G. & Gaylord, T. K. Rigorous coupled-wave analysis of planar-grating diffraction. J. Optical Soc. Am. 71 , 811–818 (1981).

Xiong, J. H. & Wu, S. T. Rigorous coupled-wave analysis of liquid crystal polarization gratings. Opt. Express 28 , 35960–35971 (2020).

Xie, S., Natansohn, A. & Rochon, P. Recent developments in aromatic azo polymers research. Chem. Mater. 5 , 403–411 (1993).

Shishido, A. Rewritable holograms based on azobenzene-containing liquid-crystalline polymers. Polym. J. 42 , 525–533 (2010).

Bunning, T. J. et al. Holographic polymer-dispersed liquid crystals (H-PDLCs). Annu. Rev. Mater. Sci. 30 , 83–115 (2000).

Liu, Y. J. & Sun, X. W. Holographic polymer-dispersed liquid crystals: materials, formation, and applications. Adv. Optoelectron. 2008 , 684349 (2008).

Xiong, J. H. & Wu, S. T. Planar liquid crystal polarization optics for augmented reality and virtual reality: from fundamentals to applications. eLight 1 , 3 (2021).

Yaroshchuk, O. & Reznikov, Y. Photoalignment of liquid crystals: basics and current trends. J. Mater. Chem. 22 , 286–300 (2012).

Sarkissian, H. et al. Periodically aligned liquid crystal: potential application for projection displays. Mol. Cryst. Liq. Cryst. 451 , 1–19 (2006).

Komanduri, R. K. & Escuti, M. J. Elastic continuum analysis of the liquid crystal polarization grating. Phys. Rev. E 76 , 021701 (2007).

Kobashi, J., Yoshida, H. & Ozaki, M. Planar optics with patterned chiral liquid crystals. Nat. Photonics 10 , 389–392 (2016).

Lee, Y. H., Yin, K. & Wu, S. T. Reflective polarization volume gratings for high efficiency waveguide-coupling augmented reality displays. Opt. Express 25 , 27008–27014 (2017).

Lee, Y. H., He, Z. Q. & Wu, S. T. Optical properties of reflective liquid crystal polarization volume gratings. J. Optical Soc. Am. B 36 , D9–D12 (2019).

Xiong, J. H., Chen, R. & Wu, S. T. Device simulation of liquid crystal polarization gratings. Opt. Express 27 , 18102–18112 (2019).

Czapla, A. et al. Long-period fiber gratings with low-birefringence liquid crystal. Mol. Cryst. Liq. Cryst. 502 , 65–76 (2009).

Dąbrowski, R., Kula, P. & Herman, J. High birefringence liquid crystals. Crystals 3 , 443–482 (2013).

Mack, C. Fundamental Principles of Optical Lithography: The Science of Microfabrication (Chichester: John Wiley & Sons, 2007).

Genevet, P. et al. Recent advances in planar optics: from plasmonic to dielectric metasurfaces. Optica 4 , 139–152 (2017).

Guo, L. J. Nanoimprint lithography: methods and material requirements. Adv. Mater. 19 , 495–513 (2007).

Park, J. et al. Electrically driven mid-submicrometre pixelation of InGaN micro-light-emitting diode displays for augmented-reality glasses. Nat. Photonics 15 , 449–455 (2021).

Khorasaninejad, M. et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging. Science 352 , 1190–1194 (2016).

Li, S. Q. et al. Phase-only transmissive spatial light modulator based on tunable dielectric metasurface. Science 364 , 1087–1090 (2019).

Liang, K. L. et al. Advances in color-converted micro-LED arrays. Jpn. J. Appl. Phys. 60 , SA0802 (2020).

Jin, S. X. et al. GaN microdisk light emitting diodes. Appl. Phys. Lett. 76 , 631–633 (2000).

Day, J. et al. Full-scale self-emissive blue and green microdisplays based on GaN micro-LED arrays. In Proc. SPIE 8268, Quantum Sensing and Nanophotonic Devices IX (SPIE, San Francisco, California, United States, 2012).

Huang, Y. G. et al. Mini-LED, micro-LED and OLED displays: present status and future perspectives. Light.: Sci. Appl. 9 , 105 (2020).

Parbrook, P. J. et al. Micro-light emitting diode: from chips to applications. Laser Photonics Rev. 15 , 2000133 (2021).

Day, J. et al. III-Nitride full-scale high-resolution microdisplays. Appl. Phys. Lett. 99 , 031116 (2011).

Liu, Z. J. et al. 360 PPI flip-chip mounted active matrix addressable light emitting diode on silicon (LEDoS) micro-displays. J. Disp. Technol. 9 , 678–682 (2013).

Zhang, L. et al. Wafer-scale monolithic hybrid integration of Si-based IC and III–V epi-layers—A mass manufacturable approach for active matrix micro-LED micro-displays. J. Soc. Inf. Disp. 26 , 137–145 (2018).

Tian, P. F. et al. Size-dependent efficiency and efficiency droop of blue InGaN micro-light emitting diodes. Appl. Phys. Lett. 101 , 231110 (2012).

Olivier, F. et al. Shockley-Read-Hall and Auger non-radiative recombination in GaN based LEDs: a size effect study. Appl. Phys. Lett. 111 , 022104 (2017).

Konoplev, S. S., Bulashevich, K. A. & Karpov, S. Y. From large-size to micro-LEDs: scaling trends revealed by modeling. Phys. Status Solidi (A) 215 , 1700508 (2018).

Li, L. Z. et al. Transfer-printed, tandem microscale light-emitting diodes for full-color displays. Proc. Natl Acad. Sci. USA 118 , e2023436118 (2021).

Oh, J. T. et al. Light output performance of red AlGaInP-based light emitting diodes with different chip geometries and structures. Opt. Express 26 , 11194–11200 (2018).

Shen, Y. C. et al. Auger recombination in InGaN measured by photoluminescence. Appl. Phys. Lett. 91 , 141101 (2007).

Wong, M. S. et al. High efficiency of III-nitride micro-light-emitting diodes by sidewall passivation using atomic layer deposition. Opt. Express 26 , 21324–21331 (2018).

Han, S. C. et al. AlGaInP-based Micro-LED array with enhanced optoelectrical properties. Optical Mater. 114 , 110860 (2021).

Wong, M. S. et al. Size-independent peak efficiency of III-nitride micro-light-emitting-diodes using chemical treatment and sidewall passivation. Appl. Phys. Express 12 , 097004 (2019).

Ley, R. T. et al. Revealing the importance of light extraction efficiency in InGaN/GaN microLEDs via chemical treatment and dielectric passivation. Appl. Phys. Lett. 116 , 251104 (2020).

Moon, S. W. et al. Recent progress on ultrathin metalenses for flat optics. iScience 23 , 101877 (2020).

Arbabi, A. et al. Efficient dielectric metasurface collimating lenses for mid-infrared quantum cascade lasers. Opt. Express 23 , 33310–33317 (2015).

Yu, N. F. et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science 334 , 333–337 (2011).

Liang, H. W. et al. High performance metalenses: numerical aperture, aberrations, chromaticity, and trade-offs. Optica 6 , 1461–1470 (2019).

Park, J. S. et al. All-glass, large metalens at visible wavelength using deep-ultraviolet projection lithography. Nano Lett. 19 , 8673–8682 (2019).

Yoon, G. et al. Single-step manufacturing of hierarchical dielectric metalens in the visible. Nat. Commun. 11 , 2268 (2020).

Lee, G. Y. et al. Metasurface eyepiece for augmented reality. Nat. Commun. 9 , 4562 (2018).

Chen, W. T. et al. A broadband achromatic metalens for focusing and imaging in the visible. Nat. Nanotechnol. 13 , 220–226 (2018).

Wang, S. M. et al. A broadband achromatic metalens in the visible. Nat. Nanotechnol. 13 , 227–232 (2018).

Lan, S. F. et al. Metasurfaces for near-eye augmented reality. ACS Photonics 6 , 864–870 (2019).

Fan, Z. B. et al. A broadband achromatic metalens array for integral imaging in the visible. Light.: Sci. Appl. 8 , 67 (2019).

Shi, Z. J., Chen, W. T. & Capasso, F. Wide field-of-view waveguide displays enabled by polarization-dependent metagratings. In Proc. SPIE 10676, Digital Optics for Immersive Displays (SPIE, Strasbourg, France, 2018).

Hong, C. C., Colburn, S. & Majumdar, A. Flat metaform near-eye visor. Appl. Opt. 56 , 8822–8827 (2017).

Bayati, E. et al. Design of achromatic augmented reality visors based on composite metasurfaces. Appl. Opt. 60 , 844–850 (2021).

Nikolov, D. K. et al. Metaform optics: bridging nanophotonics and freeform optics. Sci. Adv. 7 , eabe5112 (2021).

Tamir, T. & Peng, S. T. Analysis and design of grating couplers. Appl. Phys. 14 , 235–254 (1977).

Miller, J. M. et al. Design and fabrication of binary slanted surface-relief gratings for a planar optical interconnection. Appl. Opt. 36 , 5717–5727 (1997).

Levola, T. & Laakkonen, P. Replicated slanted gratings with a high refractive index material for in and outcoupling of light. Opt. Express 15 , 2067–2074 (2007).

Shrestha, S. et al. Broadband achromatic dielectric metalenses. Light.: Sci. Appl. 7 , 85 (2018).

Li, Z. Y. et al. Meta-optics achieves RGB-achromatic focusing for virtual reality. Sci. Adv. 7 , eabe4458 (2021).

Ratcliff, J. et al. ThinVR: heterogeneous microlens arrays for compact, 180 degree FOV VR near-eye displays. IEEE Trans. Vis. Computer Graph. 26 , 1981–1990 (2020).

Wong, T. L. et al. Folded optics with birefringent reflective polarizers. In Proc. SPIE 10335, Digital Optical Technologies 2017 (SPIE, Munich, Germany, 2017).

Li, Y. N. Q. et al. Broadband cholesteric liquid crystal lens for chromatic aberration correction in catadioptric virtual reality optics. Opt. Express 29 , 6011–6020 (2021).

Bang, K. et al. Lenslet VR: thin, flat and wide-FOV virtual reality display using fresnel lens and lenslet array. IEEE Trans. Vis. Computer Graph. 27 , 2545–2554 (2021).

Maimone, A. & Wang, J. R. Holographic optics for thin and lightweight virtual reality. ACM Trans. Graph. 39 , 67 (2020).

Kramida, G. Resolving the vergence-accommodation conflict in head-mounted displays. IEEE Trans. Vis. Computer Graph. 22 , 1912–1931 (2016).

Zhan, T. et al. Multifocal displays: review and prospect. PhotoniX 1 , 10 (2020).

Shimobaba, T., Kakue, T. & Ito, T. Review of fast algorithms and hardware implementations on computer holography. IEEE Trans. Ind. Inform. 12 , 1611–1622 (2016).

Xiao, X. et al. Advances in three-dimensional integral imaging: sensing, display, and applications [Invited]. Appl. Opt. 52 , 546–560 (2013).

Kuiper, S. & Hendriks, B. H. W. Variable-focus liquid lens for miniature cameras. Appl. Phys. Lett. 85 , 1128–1130 (2004).

Liu, S. & Hua, H. Time-multiplexed dual-focal plane head-mounted display with a liquid lens. Opt. Lett. 34 , 1642–1644 (2009).

Wilson, A. & Hua, H. Design and demonstration of a vari-focal optical see-through head-mounted display using freeform Alvarez lenses. Opt. Express 27 , 15627–15637 (2019).

Zhan, T. et al. Pancharatnam-Berry optical elements for head-up and near-eye displays [Invited]. J. Optical Soc. Am. B 36 , D52–D65 (2019).

Oh, C. & Escuti, M. J. Achromatic diffraction from polarization gratings with high efficiency. Opt. Lett. 33 , 2287–2289 (2008).

Zou, J. Y. et al. Broadband wide-view Pancharatnam-Berry phase deflector. Opt. Express 28 , 4921–4927 (2020).

Zhan, T., Lee, Y. H. & Wu, S. T. High-resolution additive light field near-eye display by switchable Pancharatnam–Berry phase lenses. Opt. Express 26 , 4863–4872 (2018).

Tan, G. J. et al. Polarization-multiplexed multiplane display. Opt. Lett. 43 , 5651–5654 (2018).

Lanman, D. R. Display systems research at facebook reality labs (conference presentation). In Proc. SPIE 11310, Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) (SPIE, San Francisco, California, United States, 2020).

Liu, Z. J. et al. A novel BLU-free full-color LED projector using LED on silicon micro-displays. IEEE Photonics Technol. Lett. 25 , 2267–2270 (2013).

Han, H. V. et al. Resonant-enhanced full-color emission of quantum-dot-based micro LED display technology. Opt. Express 23 , 32504–32515 (2015).

Lin, H. Y. et al. Optical cross-talk reduction in a quantum-dot-based full-color micro-light-emitting-diode display by a lithographic-fabricated photoresist mold. Photonics Res. 5 , 411–416 (2017).

Liu, Z. J. et al. Micro-light-emitting diodes with quantum dots in display technology. Light.: Sci. Appl. 9 , 83 (2020).

Kim, H. M. et al. Ten micrometer pixel, quantum dots color conversion layer for high resolution and full color active matrix micro-LED display. J. Soc. Inf. Disp. 27 , 347–353 (2019).

Xuan, T. T. et al. Inkjet-printed quantum dot color conversion films for high-resolution and full-color micro light-emitting diode displays. J. Phys. Chem. Lett. 11 , 5184–5191 (2020).

Chen, S. W. H. et al. Full-color monolithic hybrid quantum dot nanoring micro light-emitting diodes with improved efficiency using atomic layer deposition and nonradiative resonant energy transfer. Photonics Res. 7 , 416–422 (2019).

Krishnan, C. et al. Hybrid photonic crystal light-emitting diode renders 123% color conversion effective quantum yield. Optica 3 , 503–509 (2016).

Kang, J. H. et al. RGB arrays for micro-light-emitting diode applications using nanoporous GaN embedded with quantum dots. ACS Applied Mater. Interfaces 12 , 30890–30895 (2020).

Chen, G. S. et al. Monolithic red/green/blue micro-LEDs with HBR and DBR structures. IEEE Photonics Technol. Lett. 30 , 262–265 (2018).

Hsiang, E. L. et al. Enhancing the efficiency of color conversion micro-LED display with a patterned cholesteric liquid crystal polymer film. Nanomaterials 10 , 2430 (2020).

Kang, C. M. et al. Hybrid full-color inorganic light-emitting diodes integrated on a single wafer using selective area growth and adhesive bonding. ACS Photonics 5 , 4413–4422 (2018).

Geum, D. M. et al. Strategy toward the fabrication of ultrahigh-resolution micro-LED displays by bonding-interface-engineered vertical stacking and surface passivation. Nanoscale 11 , 23139–23148 (2019).

Ra, Y. H. et al. Full-color single nanowire pixels for projection displays. Nano Lett. 16 , 4608–4615 (2016).

Motoyama, Y. et al. High-efficiency OLED microdisplay with microlens array. J. Soc. Inf. Disp. 27 , 354–360 (2019).

Fujii, T. et al. 4032 ppi High-resolution OLED microdisplay. J. Soc. Inf. Disp. 26 , 178–186 (2018).

Hamer, J. et al. High-performance OLED microdisplays made with multi-stack OLED formulations on CMOS backplanes. In Proc. SPIE 11473, Organic and Hybrid Light Emitting Materials and Devices XXIV . Online Only (SPIE, 2020).

Joo, W. J. et al. Metasurface-driven OLED displays beyond 10,000 pixels per inch. Science 370 , 459–463 (2020).

Vettese, D. Liquid crystal on silicon. Nat. Photonics 4 , 752–754 (2010).

Zhang, Z. C., You, Z. & Chu, D. P. Fundamentals of phase-only liquid crystal on silicon (LCOS) devices. Light.: Sci. Appl. 3 , e213 (2014).

Hornbeck, L. J. The DMD TM projection display chip: a MEMS-based technology. MRS Bull. 26 , 325–327 (2001).

Zhang, Q. et al. Polarization recycling method for light-pipe-based optical engine. Appl. Opt. 52 , 8827–8833 (2013).

Hofmann, U., Janes, J. & Quenzer, H. J. High-Q MEMS resonators for laser beam scanning displays. Micromachines 3 , 509–528 (2012).

Holmström, S. T. S., Baran, U. & Urey, H. MEMS laser scanners: a review. J. Microelectromechanical Syst. 23 , 259–275 (2014).

Bao, X. Z. et al. Design and fabrication of AlGaInP-based micro-light-emitting-diode array devices. Opt. Laser Technol. 78 , 34–41 (2016).

Olivier, F. et al. Influence of size-reduction on the performances of GaN-based micro-LEDs for display application. J. Lumin. 191 , 112–116 (2017).

Liu, Y. B. et al. High-brightness InGaN/GaN Micro-LEDs with secondary peak effect for displays. IEEE Electron Device Lett. 41 , 1380–1383 (2020).

Qi, L. H. et al. 848 ppi high-brightness active-matrix micro-LED micro-display using GaN-on-Si epi-wafers towards mass production. Opt. Express 29 , 10580–10591 (2021).

Chen, E. G. & Yu, F. H. Design of an elliptic spot illumination system in LED-based color filter-liquid-crystal-on-silicon pico projectors for mobile embedded projection. Appl. Opt. 51 , 3162–3170 (2012).

Darmon, D., McNeil, J. R. & Handschy, M. A. 70.1: LED-illuminated pico projector architectures. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 39 , 1070–1073 (2008).

Essaian, S. & Khaydarov, J. State of the art of compact green lasers for mobile projectors. Optical Rev. 19 , 400–404 (2012).

Sun, W. S. et al. Compact LED projector design with high uniformity and efficiency. Appl. Opt. 53 , H227–H232 (2014).

Sun, W. S., Chiang, Y. C. & Tsuei, C. H. Optical design for the DLP pocket projector using LED light source. Phys. Procedia 19 , 301–307 (2011).

Chen, S. W. H. et al. High-bandwidth green semipolar (20–21) InGaN/GaN micro light-emitting diodes for visible light communication. ACS Photonics 7 , 2228–2235 (2020).

Yoshida, K. et al. 245 MHz bandwidth organic light-emitting diodes used in a gigabit optical wireless data link. Nat. Commun. 11 , 1171 (2020).

Park, D. W. et al. 53.5: High-speed AMOLED pixel circuit and driving scheme. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 41 , 806–809 (2010).

Tan, L., Huang, H. C. & Kwok, H. S. 78.1: Ultra compact polarization recycling system for white light LED based pico-projection system. Soc. Inf. Disp. Int. Symp. Dig. Tech. Pap. 41 , 1159–1161 (2010).

Maimone, A., Georgiou, A. & Kollin, J. S. Holographic near-eye displays for virtual and augmented reality. ACM Trans. Graph. 36 , 85 (2017).

Pan, J. W. et al. Portable digital micromirror device projector using a prism. Appl. Opt. 46 , 5097–5102 (2007).

Huang, Y. et al. Liquid-crystal-on-silicon for augmented reality displays. Appl. Sci. 8 , 2366 (2018).

Peng, F. L. et al. Analytical equation for the motion picture response time of display devices. J. Appl. Phys. 121 , 023108 (2017).

Pulli, K. 11-2: invited paper: meta 2: immersive optical-see-through augmented reality. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 48 , 132–133 (2017).

Lee, B. & Jo, Y. in Advanced Display Technology: Next Generation Self-Emitting Displays (eds Kang, B., Han, C. W. & Jeong, J. K.) 307–328 (Springer, 2021).

Cheng, D. W. et al. Design of an optical see-through head-mounted display with a low f -number and large field of view using a freeform prism. Appl. Opt. 48 , 2655–2668 (2009).

Zheng, Z. R. et al. Design and fabrication of an off-axis see-through head-mounted display with an x–y polynomial surface. Appl. Opt. 49 , 3661–3668 (2010).

Wei, L. D. et al. Design and fabrication of a compact off-axis see-through head-mounted display using a freeform surface. Opt. Express 26 , 8550–8565 (2018).

Liu, S., Hua, H. & Cheng, D. W. A novel prototype for an optical see-through head-mounted display with addressable focus cues. IEEE Trans. Vis. Computer Graph. 16 , 381–393 (2010).

Hua, H. & Javidi, B. A 3D integral imaging optical see-through head-mounted display. Opt. Express 22 , 13484–13491 (2014).

Song, W. T. et al. Design of a light-field near-eye display using random pinholes. Opt. Express 27 , 23763–23774 (2019).

Wang, X. & Hua, H. Depth-enhanced head-mounted light field displays based on integral imaging. Opt. Lett. 46 , 985–988 (2021).

Huang, H. K. & Hua, H. Generalized methods and strategies for modeling and optimizing the optics of 3D head-mounted light field displays. Opt. Express 27 , 25154–25171 (2019).

Huang, H. K. & Hua, H. High-performance integral-imaging-based light field augmented reality display using freeform optics. Opt. Express 26 , 17578–17590 (2018).

Cheng, D. W. et al. Design and manufacture AR head-mounted displays: a review and outlook. Light.: Adv. Manuf. 2 , 24 (2021).

Google Scholar  

Westheimer, G. The Maxwellian view. Vis. Res. 6 , 669–682 (1966).

Do, H., Kim, Y. M. & Min, S. W. Focus-free head-mounted display based on Maxwellian view using retroreflector film. Appl. Opt. 58 , 2882–2889 (2019).

Park, J. H. & Kim, S. B. Optical see-through holographic near-eye-display with eyebox steering and depth of field control. Opt. Express 26 , 27076–27088 (2018).

Chang, C. L. et al. Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica 7 , 1563–1578 (2020).

Hsueh, C. K. & Sawchuk, A. A. Computer-generated double-phase holograms. Appl. Opt. 17 , 3874–3883 (1978).

Chakravarthula, P. et al. Wirtinger holography for near-eye displays. ACM Trans. Graph. 38 , 213 (2019).

Peng, Y. F. et al. Neural holography with camera-in-the-loop training. ACM Trans. Graph. 39 , 185 (2020).

Shi, L. et al. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591 , 234–239 (2021).

Jang, C. et al. Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina. ACM Trans. Graph. 36 , 190 (2017).

Jang, C. et al. Holographic near-eye display with expanded eye-box. ACM Trans. Graph. 37 , 195 (2018).

Kim, S. B. & Park, J. H. Optical see-through Maxwellian near-to-eye display with an enlarged eyebox. Opt. Lett. 43 , 767–770 (2018).

Shrestha, P. K. et al. Accommodation-free head mounted display with comfortable 3D perception and an enlarged eye-box. Research 2019 , 9273723 (2019).

Lin, T. G. et al. Maxwellian near-eye display with an expanded eyebox. Opt. Express 28 , 38616–38625 (2020).

Jo, Y. et al. Eye-box extended retinal projection type near-eye display with multiple independent viewpoints [Invited]. Appl. Opt. 60 , A268–A276 (2021).

Xiong, J. H. et al. Aberration-free pupil steerable Maxwellian display for augmented reality with cholesteric liquid crystal holographic lenses. Opt. Lett. 46 , 1760–1763 (2021).

Viirre, E. et al. Laser safety analysis of a retinal scanning display system. J. Laser Appl. 9 , 253–260 (1997).

Ratnam, K. et al. Retinal image quality in near-eye pupil-steered systems. Opt. Express 27 , 38289–38311 (2019).

Maimone, A. et al. Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources. In Proc. ACM SIGGRAPH 2014 Emerging Technologies (ACM, Vancouver, Canada, 2014).

Jeong, J. et al. Holographically printed freeform mirror array for augmented reality near-eye display. IEEE Photonics Technol. Lett. 32 , 991–994 (2020).

Ha, J. & Kim, J. Augmented reality optics system with pin mirror. US Patent 10,989,922 (2021).

Park, S. G. Augmented and mixed reality optical see-through combiners based on plastic optics. Inf. Disp. 37 , 6–11 (2021).

Xiong, J. H. et al. Breaking the field-of-view limit in augmented reality with a scanning waveguide display. OSA Contin. 3 , 2730–2740 (2020).

Levola, T. 7.1: invited paper: novel diffractive optical components for near to eye displays. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 37 , 64–67 (2006).

Laakkonen, P. et al. High efficiency diffractive incouplers for light guides. In Proc. SPIE 6896, Integrated Optics: Devices, Materials, and Technologies XII . (SPIE, San Jose, California, United States, 2008).

Bai, B. F. et al. Optimization of nonbinary slanted surface-relief gratings as high-efficiency broadband couplers for light guides. Appl. Opt. 49 , 5454–5464 (2010).

Äyräs, P., Saarikko, P. & Levola, T. Exit pupil expander with a large field of view based on diffractive optics. J. Soc. Inf. Disp. 17 , 659–664 (2009).

Yoshida, T. et al. A plastic holographic waveguide combiner for light-weight and highly-transparent augmented reality glasses. J. Soc. Inf. Disp. 26 , 280–286 (2018).

Yu, C. et al. Highly efficient waveguide display with space-variant volume holographic gratings. Appl. Opt. 56 , 9390–9397 (2017).

Shi, X. L. et al. Design of a compact waveguide eyeglass with high efficiency by joining freeform surfaces and volume holographic gratings. J. Optical Soc. Am. A 38 , A19–A26 (2021).

Han, J. et al. Portable waveguide display system with a large field of view by integrating freeform elements and volume holograms. Opt. Express 23 , 3534–3549 (2015).

Weng, Y. S. et al. Liquid-crystal-based polarization volume grating applied for full-color waveguide displays. Opt. Lett. 43 , 5773–5776 (2018).

Lee, Y. H. et al. Compact see-through near-eye display with depth adaption. J. Soc. Inf. Disp. 26 , 64–70 (2018).

Tekolste, R. D. & Liu, V. K. Outcoupling grating for augmented reality system. US Patent 10,073,267 (2018).

Grey, D. & Talukdar, S. Exit pupil expanding diffractive optical waveguiding device. US Patent 10,073, 267 (2019).

Yoo, C. et al. Extended-viewing-angle waveguide near-eye display with a polarization-dependent steering combiner. Opt. Lett. 45 , 2870–2873 (2020).

Schowengerdt, B. T., Lin, D. & St. Hilaire, P. Multi-layer diffractive eyepiece with wavelength-selective reflector. US Patent 10,725,223 (2020).

Wang, Q. W. et al. Stray light and tolerance analysis of an ultrathin waveguide display. Appl. Opt. 54 , 8354–8362 (2015).

Wang, Q. W. et al. Design of an ultra-thin, wide-angle, stray-light-free near-eye display with a dual-layer geometrical waveguide. Opt. Express 28 , 35376–35394 (2020).

Frommer, A. Lumus: maximus: large FoV near to eye display for consumer AR glasses. In Proc. SPIE 11764, AVR21 Industry Talks II . Online Only (SPIE, 2021).

Ayres, M. R. et al. Skew mirrors, methods of use, and methods of manufacture. US Patent 10,180,520 (2019).

Utsugi, T. et al. Volume holographic waveguide using multiplex recording for head-mounted display. ITE Trans. Media Technol. Appl. 8 , 238–244 (2020).

Aieta, F. et al. Multiwavelength achromatic metasurfaces by dispersive phase compensation. Science 347 , 1342–1345 (2015).

Arbabi, E. et al. Controlling the sign of chromatic dispersion in diffractive optics with dielectric metasurfaces. Optica 4 , 625–632 (2017).

Download references

Acknowledgements

The authors are indebted to Goertek Electronics for the financial support and Guanjun Tan for helpful discussions.

Author information

Authors and affiliations.

College of Optics and Photonics, University of Central Florida, Orlando, FL, 32816, USA

Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan & Shin-Tson Wu

You can also search for this author in PubMed   Google Scholar

Contributions

J.X. conceived the idea and initiated the project. J.X. mainly wrote the manuscript and produced the figures. E.-L.H., Z.H., and T.Z. contributed to parts of the manuscript. S.W. supervised the project and edited the manuscript.

Corresponding author

Correspondence to Shin-Tson Wu .

Ethics declarations

Conflict of interest.

The authors declare no competing interests.

Supplementary information

Supplementary information, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and Permissions

About this article

Cite this article.

Xiong, J., Hsiang, EL., He, Z. et al. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light Sci Appl 10 , 216 (2021). https://doi.org/10.1038/s41377-021-00658-8

Download citation

Received : 06 June 2021

Revised : 26 September 2021

Accepted : 04 October 2021

Published : 25 October 2021

DOI : https://doi.org/10.1038/s41377-021-00658-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Application progress of artificial intelligence and augmented reality in orthopaedic arthroscopy surgery.

• Haojie Chen

Journal of Orthopaedic Surgery and Research (2023)

Wafer-scale organic-on-III-V monolithic heterogeneous integration for active-matrix micro-LED displays

• Simon Ogier
• Xiaojun Guo

Nature Communications (2023)

Achromatic diffractive liquid-crystal optics for virtual reality displays

• Yannanqi Li
• Shin-Tson Wu

Light: Science & Applications (2023)

Laser nanoprinting of 3D nonlinear holograms beyond 25000 pixels-per-inch for inter-wavelength-band information processing

• Pengcheng Chen

Solid-body trajectoids shaped to roll along desired pathways

• Yaroslav I. Sobolev
• Bartosz A. Grzybowski

Nature (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Virtual Reality

We are pleased to announce that Virtual Reality will become a fully open access (OA) on 1 January 2024 . As a result, all submissions received from  1 August 2023    are subject to an article publication charge (APC) if accepted and published in the journal (unless a waiver is applied). Please see our FAQs which can be found at the bottom of this page for more information on APCs, funding options, waivers and the journal's transition to fully open access. The journal, established in 1995, publishes original research in Virtual Reality, Augmented and Mixed Reality that shapes and informs the community.  The multidisciplinary nature of the field means that submissions are welcomed on a wide range of topics including, but not limited to:

• Original research studies of Virtual Reality, Augmented Reality, Mixed Reality and real-time visualization applications
• Display technologies, including Head Mounted Displays, simulators and immersive displays. 
• Haptic technologies, including novel devices, interaction and rendering
• Interaction management, including gesture control, eye gaze, biosensors and wearables 
• Tracking technologies
• VR/AR/MR in medicine, including training, surgical simulation, rehabilitation, and tissue/organ modelling.
• Impactful and original applications and studies of VR/AR/MR’s utility in areas such as manufacturing, business, telecommunications, arts, education, design, entertainment and defence
• Research demonstrating new techniques and approaches to designing, building and evaluating virtual and augmented reality systems
• Original research studies assessing the social, ethical, data or legal aspects of VR/AR/MR

To complement original research articles, the journal also welcomes papers that provide state of the art reviews of specific aspects of VR/MR/AR.  Such papers must be rigorous and represent a valuable resource for researcher and practitioners. The journal’s audience includes those undertaking research in academia and industry, and developers and users of applications.  Since readers may therefore be unfamiliar with all aspects of a specific topic being discussed, emphasis will be placed on clear, well-written, and accessible articles which will appeal to a multidisciplinary audience. 

• Covers a wide range of topics, to fully capture the multidisciplinary nature of the field. From business and commerce to telecommunications, entertainment and gaming to medicine, Virtual Reality covers a wide range of specific application areas.
• Features clear, well-written, and accessible articles that will appeal to a multidisciplinary audience.

Journal information

• Robert D. Macredie,
• Daniel Ballin

Journal metrics

Latest issue.

Issue 3, September 2023

Latest articles

User behavior modeling for ar personalized recommendations in spatial transitions, authors (first, second and last of 5).

• Maryam Shakeri
• Hyerim Park
• Woontack Woo
• Content type: Original Article
• Published: 24 October 2023

Utilization of virtual reality for operating room fire safety training: a randomized trial

Authors (first, second and last of 7).

• Daniel Katz
• Benjamin Hyers
• Garrett Burnett
• Published: 17 October 2023

An innovative mixed reality approach for maxillofacial osteotomies and repositioning

Authors (first, second and last of 4).

• Agnese Brunzini
• Alida Mazzoli
• Marco Mandolini
• Open Access

A systematic evaluation of an RTK-GPS device for wearable augmented reality

• Francesco De Pace
• Hannes Kaufmann
• Published: 16 October 2023

Shaping the auditory peripersonal space with motor planning in immersive virtual reality

• Michele Geronazzo
• Roberto Barumerli
• Paola Cesari
• Published: 10 October 2023

Journal updates

Virtual reality is transitioning to full open access.

We are excited to announce that  Virtual Reality  will become a fully open access (OA) journal in 2024. This means that as of January 2024, we will only be publishing open access articles, meaning content will be freely available to readers worldwide, enabling the widest possible dissemination and reuse.

Springer Nature is committed to ensuring our portfolios serve and meet the evolving needs of the research community and that quality research is read and disseminated as widely as possible. Transforming Virtual Reality to full OA reflects our ongoing commitment to driving the transition to open research, and providing researchers with OA publication venues that can amplify their research and enable compliance with funder and institutional OA requirements, while continuing to deliver a high level of author service. See our FAQs  for further information.

Transition to fully OA FAQ

Frequently asked questions regarding the transition to fully open access and what this means for  Virtual Reality .

Follow Virtual Reality on Twitter

Find forthcoming articles and issues and many other interesting news on Twitter

Guidelines for Special Issue Proposals

Instructions for submitting a Special Issue proposal to  Virtual Reality

For authors

Working on a manuscript.

Avoid the most common mistakes and prepare your manuscript for journal editors.

About this journal

• ACM Digital Library
• Current Contents/Engineering, Computing and Technology
• EBSCO Applied Science & Technology Source
• EBSCO Computer Science Index
• EBSCO Computers & Applied Sciences Complete
• EBSCO Discovery Service
• EBSCO Engineering Source
• EBSCO Ergonomics Abstracts
• EBSCO STM Source
• EI Compendex
• Google Scholar
• Japanese Science and Technology Agency (JST)
• Journal Citation Reports/Science Edition
• OCLC WorldCat Discovery Service
• ProQuest Advanced Technologies & Aerospace Database
• ProQuest-ExLibris Primo
• ProQuest-ExLibris Summon
• Science Citation Index Expanded (SCIE)
• TD Net Discovery Service
• UGC-CARE List (India)

Rights and permissions

Springer policies

© Springer-Verlag London Ltd., part of Springer Nature

• Reference Manager
• Simple TEXT file

People also looked at

Original research article, research into improved distance learning using vr technology.

• 1 Art College of Jinan University, Guangzhou, China
• 2 Guangzhou Nidu Information Technology Co., Ltd., Guangzhou, China
• 3 School of Fine Art, South China Normal University, Guangzhou, China

Practical skill-based education requires exemplary face-to-face operational teaching, and VR can enhance online distance learning, facilitating an alternative form of “face-to-face” teaching, which results in better teacher–student communication and learner self-efficacy. It also constitutes as a useful substitute for in-person teaching, and it also has a positive impact on learning effectiveness. In this study, a mixed-method approach was used, which utilized the following methodologies: a combination of quantitative and qualitative measures, document collection, case and comparative analysis, and VR teaching that utilizes “You, Calligrapher” as a survey tool. Teachers and students of art were selected, who then used an educational VR-based calligraphy game application for teaching activities. We investigated the impact of virtual time, space, and technical availability on learners’ understanding, imagination, and interactivity in VR education, and then we evaluated the positive impact via learner feedback. Research tools that we utilized consist of comprehension, imagination, and how feedback motivation scales with effective learning; we have also used Chinese calligraphy performance tests. The SPSS statistical analysis software was used for related statistical processing, and α was set to 0.05. The results of this study indicated that Chinese calligraphy studies in VR time and space affect students’ understanding and imagination but not their operational abilities. According to our research, a fundamental difference between traditional and modern teaching methods is a shift toward the use of VR (and the internet) in education. Therefore, the focus of this study is on understanding the impact on practical skills during distance learning and investing the impacts in order to form an effective approach to the use of VR in education.

1 Introduction

With the development of digital media technology, educational contexts continue to push the boundaries of what “can” be done: what can be virtualized, monitored in real time, and simulated ( Hu et al., 2021 ). The space for education and technical means of communication, and these interactions in operational skills education have been redefined. Given the influence of the current COVID-19 pandemic in particular, there is now an even greater demand for online distance education, whereby students and teachers can conduct educational learning without having to leave their homes. However, the limitations of online education have been a subject of criticism as practical courses cannot be taught in the usual way, and with distance learning, students may have to rely more on their imagination in certain aspects of the process. They cannot obtain in-person guidance from teachers, and teachers cannot obtain real-time feedback from students. The quality of knowledge transferred may also be limited by camera angles, screen size, and clarity. The conveying of information will be affected by abstract features of language and by feedback factors (which may not be timely), impinging on learners’ ability to obtain key knowledge and grasp key concepts. This therefore highlights the importance of VR education, which can help teachers and students to overcome time and space restrictions, and conduct remote “face-to-face” communication; teachers can borrow the artificial intelligence-based generative adversarial network (GAN) technology to translate semantics for real-time dynamic images, practical courses can be “hands-on,” and practical exercises can be conducted to obtain teacher guidance and feedback ( Huang G et al., 2019 ; Huang Y. C et al., 2019 ).

Virtual reality (VR) technology consists of a fusion of multichannel forms of information, with the ability to create interactive, three-dimensional dynamic scenes and behavioral simulations, and it enables users to enjoy immersive experiences and benefit from other advantages, such as engaging at a higher cognitive level, experiencing the richness of a highly realistic simulation, and experiencing a diverse range of sensory stimuli ( He et al., 2019 ). VR can simulate a real panoramic experimental environment and dynamic simulation of experimental teaching methods and conditions, and it can also facilitate behavior-based interactions, thus having great potential for enhancing learners’ cognitive and practical skills (Burdea et al., 2003).

Artificial intelligence (AI) technology has gradually matured in terms of image translation and dynamic image generation. It can also show important changes in voice visualization, which is self-evident for the role of education. 6DOF free-angle video collection technology can record dynamic 3-D stereo data in real time; as a result, it has great potential in operational skills training. Moreover, the lightweight, low cost, and versatility of motion-capture technology have extended the limits of people’s participation in the virtual world and have made it impressively realistic. Cloud computing and the popularization of 5G technology have enabled distance education to become more real time, and with the application of VR technology, realistic distance learning has become more and more of a reality.

Therefore, when the combination of VR technology, artificial intelligence, and other related technologies is compared with the traditional education method, it presents the unique characteristics of non-face-to-face physical space, non-linear time, 360-degree real-time interaction, and artificial intelligence-based personalized teaching, replacing traditional in-person teaching and focusing on teaching and learning. Typically, teachers pass on knowledge via language and the written word ( Kyrlitsias et al., 2020 ). During the listening process, students process information through their auditory organs, developing their own understanding of what they have been taught based on their very own interpretation of the knowledge they were taught. When they encounter problems that they do not understand, students can engage in language interaction, communicating with one another until a better level of understanding is achieved, and completing the teaching and learning process at the same time and in the same space as the teacher. However, because of the subjective nature of language and writing, there is often a discrepancy between what students understand and the knowledge taught by teachers during the communication. Teachers are required to demonstrate key aspects of learning or to use visual aids to assist teaching so that students can understand and remember. This is especially the case with skill-based teaching, such as Chinese calligraphy. Students are generally required to demonstrate their proficiency and articulate their understanding face to face, and students need teachers’ guidance and constructive feedback when practicing. In traditional education, text, images, and videos can help students enhance their understanding of knowledge, but for operational teaching, intuitive teaching assistance cannot be obtained through abstract symbols (text, images, and videos).

The purpose of this study is to use the effectiveness of VR to teach Chinese books and paintings, analyze if VR is able to enable distance education, and explore the impacts of VR practical teaching. The advent of the VR era will change education further. It can enable students to communicate with teachers “face to face” in teaching scenarios in remote and unusual situations, and it can help students overcome learning obstacles via the Internet. It is also possible to reproduce abstract knowledge in real time in virtual contexts without being subjected to physical constraints or screen display limitations. VR can help teachers and apprentices practice skills, with 360-degree interactive observations and learning, and it can simulate the teaching methods of real-world technology majors ( Chen and Deng, 2015 ).

This study started with the three characteristics of space–time (derived from immersion, focusing on the teachers and students in different spaces teaching in the same virtual environment), imagination, and interactive VR ( Chen et al., 2019a ; Lee et al., 2020 ). We investigated the effect of VR technology and artificial intelligence technology on VR-based skills distance learning and the effectiveness of learning, using the “You, Calligrapher” VR-based painting application as an experimental tool, utilizing a combination of quantitative and qualitative methods. The results of the study were then used to consider the implications for other applications in VR education.

The theoretical model of this study contains three characteristics of VR: immersion (virtual time and space) (TS), imagination (IM), and interactivity (IN). Impact on the learning effect (LE) of distance practical teaching; and the direct impact of the technical availability (TA) factor on the learning effect (LE) of distance practical teaching; then the technical availability has improved understanding of students (SC),-as shown in Figure 1 .

FIGURE 1 . Theoretical model of VR education methods to improve learning effectiveness.

Time and space are the rules of the real world that differ from the virtual world. Therefore, the space–time factors of the virtual world will result in changes to human lifestyle and behavior, break through the constraints of time and space, and be more conducive to promoting the realism and enhancing the immersion of distance education (TS to LE). The teaching of the real world has undergone formal changes in the virtual world through language communication, supplemented by visualized images, text, and other symbolic forms ( Setti and Csapo, 2021 ). The visual display of virtual spaces will not be restricted by physical screens. As a result, it is easier to spread information; realize the three-dimensional integration of information, symbolic imaging, and dynamic characteristics; and change the understanding of learners (IM to LE). Practical teaching in the real world relies more on physical operational teaching, emphasizing one-on-one or one-to-many same-physical space teaching. However, the virtual space can achieve long-range synchronized operation teaching without physical materialization so that learners can observe the operation process of the lecturer with multiple angles and perspectives without physical objects and conduct synchronous operation exercises to obtain real-time feedback from teachers’ guidance (IN to LE).

Technical availability is the decisive factor for the popularization of virtual reality. The continuous interactive update of technologies such as artificial intelligence, VR/AR, big data, 5G, cloud service, and marginal computing has prompted virtual reality distance education to become more simulated. As a result, human-to-human connectivity in the virtual world is more and more real time, diverse, and convenient, creating a seamless connection between the virtual world and the real world, and achieving more efficient teaching practices than that in the real world (TA to LE). Artificial intelligence teaching can also simultaneously translate teaching knowledge, transform voice information into visualized information in an instant, and then obtain the learner’s knowledge mastery data through big data, giving learners more modest learning information quantities and methods (TA to SC). Finally, judging the improvements of the learners’ learning effect, there is a positive impact on the learning efficiency, and it might also have a negative impact on the learners’ independent learning motivation; so this article will conduct research and exploration (SC to LE).

2 Literature Review

2.1 the status of vr technology applied to practical teaching.

Virtual activities can help students master practical skills and can promote the development of students’ cognitive skills through simulations of key knowledge and processes relevant to certain fields in the real world, or they can be substituted for real environments to a certain extent ( Liu and Wang, 2011 ) (Liou et al., 2018). With the combination of VR technology, VR equipment, and three-dimensional interactive virtual digital resources, we can provide students with an integrated virtual learning environment to help them improve their academic achievement and learning motivation (Bogusevschi et al., 2020). It taught junior high-school students the concept of the water cycle and precipitation formation by combining VR technology and VR laboratories to enhance their interest in physical learning. As VR technology becomes ever more sophisticated, with the characteristics of virtual reality immersion, interactivity, and promotion of students’ powers of imagination, researchers will try to increase the effectiveness of students’ learning, ensuring safety and maximizing interest levels through game-based learning. However, few scholars have been able to effectively monitor the actual effectiveness of students’ learning in the case of current online teaching and students’ learning outcomes cannot be fed back; the time and space of teaching and learning are not uniform, and the interactive effect of online education is not ideal ( Chiang, 2021 ). Although online education can incorporate the most advanced technology in the professional field, it is not intuitive enough to reach the caliber required for it to be as effective as education in the real world, nor can it participate in activities or propose a solution to ensure students’ complete satisfaction.

2.2 The Impact of VR Technology on Learning Effectiveness

The study of the impact of VR technology on learning effectiveness is divided into two categories. One particularly useful approach is that of Wang (2020) , who has proposed a method for improving reading of educational publications in VR environment through dual construction of content systems and interactive situations. Gao Haibo (2019) believed that VR teaching applications can enable practitioners to break free from the narrow view of traditional teaching that only focuses on knowledge transfer and change passive learning into active exploration, thereby enhancing learners’ immersive experience and participation in learning ( He. et al., 2019 ). They assert that gaming-based learning and the use of VR in education can enhance learners’ motivation. Experimental data confirm that game-based VR education can effectively enhance the learning experience and improve teaching quality ( Liu et al., 2019 ). They argue that artificial intelligence, VR, and AR technology can achieve better personalized teaching, relying on Internet-based technology to change educational visualization from a flat world to one that is more three-dimensional and clearer. Sitterding (2019) added VR and artificial intelligence methods to the training of new nurses in American hospitals. In experiments comparing traditional training and VR training, it was found that VR training may be more effective than traditional teaching methods ( Horváth et al., 2021 , Hsiao et al., 2021 ).

On the other hand, it has been argued that the application of VR technology to education will bring a higher cognitive load and will not significantly promote the learning effect. For example, Parong et al. (2018) found that the cornerstone learning effect for learners using PowerPoint (PPT) was clearly better than that for learners using VR. The reason for this may be related to the immersive nature of the VR environment, which can produce greater external cognitive loads for learners, manifested in a low level of cognitive participation (reflected in learners’ brainwave electrocardiograms) and poor migration test results. Makransky et al. (2017) used EEG equipment to track the brainwave status of desktop VR and headset-wearing VR learners, and found that learners were overloaded for nearly half of the time during the learning process, which may suggest that VR technology cannot improve the learning effect. Few scholars have conducted analysis of experimental data of the impact of VR on students’ learning effectiveness from the perspective of the three characteristics of the VR technology described earlier, and then later used mature VR application software to analyze data relating to the experiences of users ( Parong and Mayer, 2018 ; Paul and Jefferson, 2019) .

2.3 The Impact of AI Technology Applications on Student Learning

An algorithm framework for AI GAN semantic translation generational learning technology was developed with the goal of facilitating unsupervised learning. This technology can be used for mutual conversion of texts and images in VR education, translation of voice images, image over-resolution, and dynamic image generation, and it can enable other machines to learn independently ( Ninaus and Nebel, 2012 ). GAN is inspired by the two-person zero-sum game theory ( Huang et al., 2012 ). Its unique adversarial training ideas can generate high-quality samples with more powerful learning and expression characteristics than traditional machine learning algorithms ( Liang et al., 2013 ). The learning-generation process can help learners to select important information (images, text, sound, etc.) and organize this into new, continuous psychological representations in their working memory; finally, the new knowledge is integrated with existing knowledge and stored in long-term memory (Mayer, 2009). Few scholars have carried out relevant research on improving learners’ understanding in a virtual environment from the dynamic translation of artificial intelligence semantics.

2.4 Literature Summary

In general, previous studies have actively explored and studied VR technology and AI technology in practical educational contexts from multiple angles, and improvements in students’ learning and personalized education are particularly prominent. It has also been pointed out that current VR and AI technology are not yet mature, and there are many deficiencies.

Scholars often discuss the effectiveness of VR education from the perspectives of simulation safety of VR teaching and increased interest in games, from the three characteristics of VR: immersion (TS), imagination (IM), and interactivity (IN), as well as artificial intelligence. From a technical perspective, however, there are few scholars analyzing the effects of distance education and traditional online education, practical teaching in the VR environment, and the actual learning effect (LE) produced by the technical availability (TA) of artificial intelligence and VR technology on the understanding of students’ acceptance of knowledge (SC).

In summary, this study will look at the development of virtual reality practical teaching and explore the impact VR technology and AI technology have on teaching and learning. Can remote face-to-face communication in VR time and space be achieved, and can teachers obtain student feedback in real time? When students have the opportunity to view teachers’ demonstrations interactively in the VR environment, will it improve their learning effect on interactive operation exercises, and can dynamic knowledge translation be seen to occur?

3 Research Methods

3.1 research assumptions and framework.

VR technology has the three characteristics of immersion, imagination, and interaction. A VR-based immersive learning experience provides learners with an environment in which problems can be solved and knowledge can be acquired, and it can enable teachers and students in remote and foreign places to enjoy the benefits of face-to-face interaction at the same time (of particular relevance in the context of the COVID-19 pandemic). Therefore, in VR education, it will have a positive impact on the learning effect of learners. The artificial intelligence presentation of VR education will make a greater contribution to the performance of technical availability. Virtual reality technology has the three characteristics: imagination (IM), interactivity (IN), and immersion (virtual time and space) (TS). VR education provides learners with an environment to solve problems and acquire knowledge through immersive learning experience. Imagination and interactivity have a positive effect on learning effect (LE). The direct impact of the technical availability (TA) factor was generated by artificial intelligence and VR technology on the learning effect (LE) of remote practical teaching. Virtual real-life artificial intelligence will help the content of the lecturer to be translated from language to text, image translation, and image generation, and three-dimensional dynamic information generation transformation will have a positive impact on students’ ability to understand (SC). Comprehension (SC) is the basic ability that affects the effectiveness of learning (LE), the basic ability of students to develop critical thinking and innovation skills, and an important ability that affects the long-term development of students ( Chen et al., 2015 ).

It can be seen that space–time, technical availability, imagination, comprehension, and interactivity are crucial factors in students’ learning. Therefore, this study adopted a theoretical model to explain the structural approach, and this enabled us to investigate the effectiveness of skill-based teaching using VR spatiotemporal characteristics, as shown in Figure 2 .

FIGURE 2 . Diagram of hypothetical model used in the study.

In order to further explore the relationship between various influencing factors, nine possible relationships were hypothesized around these factors, and the nine assumptions in the theoretical model were as follows:

People can obtain more intuitive comprehension in VR time and space than in traditional imaging space ( Feng 2006 ). Thus, hypothesis 1 (H1) is suggested: VR time and space have a positive impact on students’ understanding.

Learning without understanding is not real learning, and students’ level of understanding will be reflected in learning outcomes. Therefore, evaluations of learning effects are essentially judgments about students’ level of understanding ( Chen et al., 2015 ). Hence, hypothesis 2 (H2) is proposed: Students’ understanding has a positive impact on learning effectiveness.

Teachers and students in VR time and space can access the same virtual teaching situation, with students receiving a valid and realistic (substitutive) learning experience (Wu, 2017). A well-designed VR program will integrate the teaching content and strategies into the application context. It is necessary to consider VR space–time characteristics and link teachers’ demonstration operations with learners’ practice. VR space–time energy stimulates the human brain’s ability to think imaginatively ( Huang et al., 2010 ). Learners are more likely to learn abstract concepts in the VR space–time environment ( He et al., 2019 ). As a result, hypothesis 3 (H3) is formed: VR time and space have a positive impact on students’ ability to use their imagination when learning.

Thinking power is the motor of intelligence. The power of thinking is weak. It directly affects the rationality and style of language, and language affects the intellectual development of learning and the geological formation of good ideas (Wei, 2017). Accordingly, hypothesis 4 (H4) is proposed: Students’ imagination has a positive effect on learning.

McLuhan believes that the extension of human consciousness is constantly designed by electrons as a holistic world environment ( Chen et al., 2015 ). A well-designed VR application product integrates the teaching content and strategies into the application scene. It is necessary to consider the VR space–time characteristics and to link the demonstration operation of the lecturer with the practice of the learner. Subsequently, hypothesis 5 (H5) is proposed: VR time and space have a positive effect on interactivity.

VR technology can track learners’ physical movements, activate time in the virtual world, and give learners a sensory experience that is so vivid and realistic as to seem physically real, akin to real-world activities but in a VR environment ( He et al., 2019 ). Thus, hypothesis 6 (H6) is formed: Technology availability has a positive impact on VR interactivity.

Using VR technology, behavioral data relating to lecturers and learners via peripheral equipment (such as handles, helmets, and motion captures), interactive sensory means (such as audiovisual devices), and through forms of informational interaction between learners and VR education can be obtained, in order to promote learning. Interactive feedback helps learners construct meaningful knowledge ( Huang et al., 2010 ). Hence, hypothesis 7 (H7) is suggested: The interactivity of VR has a positive impact on learning.

Virtual reality AI can help lecturers’ language translation, image translation, image generation, and 3-D dynamic information generation, which will have a positive impact on students’ ability to understand abstract information, enabling them to transform this into knowledge while enhancing interest levels and learning effects ( Li et al., 2017 ). As a result, hypothesis 8 (H8) is proposed: Technology availability has a positive impact on learners’ understanding.

With AI technology, voice or text can be converted into multiple languages, which can be displayed through images, 3-D static and dynamic scenes, or objects. With VR technology, learners can use external equipment such as helmets, handles, headphones, microphones, and Wi-Fi in different scenarios to interact, collaborate, and learn. VR technology can display things in multiple ways, helping learners to better understand virtual scenarios, boost their imagination, and reveal the essential characteristics of things ( Li et al., 2017 ). Hence, hypothesis 9 (H9) is suggested: The availability of technology has a positive impact on learners’ imagination.

3.2 VR Application Design and Realization

In this study, Oculus Quest2 VR equipment was used, a Chinese calligraphy and painting teaching application, and a virtual calligraphy painting application was developed using Unity 3D. “You, Calligrapher” is an educational application that integrates teaching and practice within a virtual environment,-as shown in Figure 3 . This application allows lecturers to demonstrate the teaching content in simulated ancient study rooms, emulating real-life teaching in the real world. Learners can experience an (simulated) in-person viewing process, like with traditional teaching (without being limited by position), observe the instructor’s demonstration from any angle, and repeat the viewing at any point in time. Learners can also use helmets to access the virtual environment and experience contexts with traditional Chinese cultural characteristics, such as ancient calligraphy and Zen rooms. Learners use a handle to simulate an ink brush during the writing and painting process.

FIGURE 3 . “You, Calligrapher” VR application.

The “You, Calligrapher” software enables calligraphy learners to use distance learning methods and to practice in a virtual environment. There are two ways to watch video teaching materials, and users have a 360-degree view of the teaching content. Teaching demonstrations and activities are completed simultaneously, and the AI system scores learners’ performance. The simulation process has three components: teaching, practice, and application. Learners complete the process of practicing writing during the game session.

The goal of the application is to train students to learn and improve their skills in calligraphy and Chinese painting techniques, with technical modeling and learning interaction. Learners’ progress records are stored in real time in the database, and ( via a manual intelligence algorithm evaluation) they receive instructional learning feedback, content frames, and feedback on the learning process.

The “You, Calligrapher” learning games include three components: teaching demonstrations, interactive practice, and breaking through the game level. Learners use voice and video teaching by “virtual teachers” in the game and do practice tasks, with a clear 360-degree view, enabling them to copy the teachers simultaneously. The second step involves learners using a game handle to simulate a brush, to obtain a realistic interactive writing experience and practice Chinese painting. They can also select relevant text or inscriptions from the database to copy. The third step is to complete the checkpoint task objectives (based on the present game) breaking through the game level. The AI algorithm evaluates feedback based on the learner’s writing results and completes the process.

3.3 Participants

The study participants were educated to a higher level and were selected from students at the College of Art of Jinan University. This college runs a degree course on Chinese calligraphy and painting appreciation, and has an excellent reputation throughout the country. The study sample comprises 160 teachers and students of this subject (not just limited to teachers and students of calligraphy). The participants used the VR equipment at various times to experience the “You, Calligrapher” software (at least 30 min of painting study). An electronic questionnaire was drawn up and distributed to the research subjects, for them to report on their experience of using the VR equipment/software, using real-time communication tools such as micro credit and the QQ social application. In total, 160 questionnaires were distributed, and 152 valid questionnaires were recovered (a response rate of 95%).

3.4 Measurement Tools

Learning effectiveness was assessed by the test questions, jointly prepared by the teacher and the researcher, including a “knowledge point understanding level” test and “technical ability” test. Participants took the tests as soon as they had finished using the application. In total, 152 effective test results were collected in this study. The “Education Questionnaire on the Effectiveness of Improving Practical Skills Teaching” included six dimensions: VR time and space, technical availability, interactivity, imagination, comprehension, and learning effects. Each dimension consisted of five questions (30 questions in total). The questionnaire was based on Huang et al.’s (2010) study. The original questionnaire used the Likert seven-point scale, which contains 16 topics. As this study was specifically focused on technical objectivity, a five-point Likert scale was used (1: “completely disagree,” 2: “uncertain,” 3: “not sure,” 4: “agree,” and 5: “fully agree”). Taking into account the particularity of distance learning practice technical courses, the content of three dimensions (VR time and space, technical availability, and comprehension) was replaced, -as shown in Table 1 .

TABLE 1 . Participants’ sociodemographic features [sample demographics ( N = 152)].

3.5 Experimental Steps

The first step entailed participants listening to an explanation and watching a demonstration (before the commencement of the experiment) to ensure that they understood the methods and precautions involved with the use of Oculus Quest2 and PICO NEO3, and that they understood the “You, Calligrapher” application. Participants could only proceed to the VR experience stage after the introduction by the research team. In the second step, the participants started the VR application, checked the position, adjusted the equipment to ensure that it was comfortable (e.g., the helmet and height of the virtual desktop), and started the experience.

With this application, the user needed to complete the teaching steps and freely write 16 Chinese characters. The game provided teaching feedback, and the same 16 Chinese characters could then be written again in three stages, with a completion time of 15–20 min. In the third step, the user was presented with the initial set of calligraphy results and the second set of results (after training and completing the VR experience), and then completed the questionnaire. In the fourth step, three professional teachers were asked to compare the calligraphy test results relating to the 16 Chinese characters, and a judgment was made as to the learning effect and the learning experience. The fifth step entailed data collection. IBM SPSS was used to describe participants’ opinions of the VR application experience/environment to perform descriptive statistical analysis and to analyze the impact of understanding, imagination, and interactivity on learning (through linear regression analysis). IBM SPSS was also used to verify the structural equations, and for factor analysis and path analysis.

4 Results Analysis

4.1 credit effectiveness analysis.

All participants completed the questionnaire online via the Internet, and 152 valid questionnaires were recovered. After revision, experts were invited to revise and conduct a letter-effectiveness analysis. In the letter-level analysis of the six variables, the Cronbach α coefficient value was found to be greater than 0.6, thus indicating that the quality of the letter analysis was acceptable ( Table 2 ). The Keyser–Meyer–Olkin (KMO) value was greater than 0.7 (0.710), indicating that the effect was good, and the common value was greater than 0.4. The absolute value of the factor load coefficient of the variable corresponding to each factor was greater than 0.4, and the vast majority of these were greater than 0.6, indicating that the variable and the factor had a better correspondence ( Table 3 ).

TABLE 2 . Cronbach credit analysis.

TABLE 3 . Results of effectiveness analysis.

4.2 Results Analysis

4.2.1 descriptive analysis of variables based on vr time and space in the teaching application.

The descriptive statistics on learners’ attitudes toward the teaching application in terms of VR time and space were given in relation to six dimensions: VR time and space, technical visibility, imagination, understanding, interactivity, and the learning effect. The statistical results are listed in Table 4 . Learners’ overall attitude toward the VR environment was high (mean = 4.08). From the perspective of the cognitive level of each dimension, from high to low, the order was found to be as follows: understanding, technical visibility, learning effect, VR time and space, interaction, and imagination.

TABLE 4 . Correspondence between variables and factors.

The average score for understanding was the highest (in the VR environment) and had a significant effect on understanding and skill improvement. The questionnaire item “You think VR education can help teachers and students to communicate face to face without space restrictions,” in relation to the VR time and space variables, was divided into an average of 4.6 points (total score = 5 points). The questionnaire item “You think VR6DOF teaching content is a more intuitive teaching experience than video format teaching” also scored 4.2 points (total possible score = 5 points). Testers used this function repeatedly during the learning experience. The function enabling learners to closely watch the teacher’s demonstration may fulfill an aspect of teaching that regular online teaching cannot achieve, and the virtual live experience is even higher than the actual live demonstration experience. This discovery validates the findings of previous studies, that is, that the VR space–time environment can provide an immersive teaching and learning experience, which helps improve learning effectiveness, as shown in Table 5 .

TABLE 5 . Statistical table showing descriptions of learning effect variables.

4.2.2 Structural Equations for Factors Influencing Learning Motivation in the VR Environment

4.2.2.1 establishment and evaluation of structural equation models.

IBM SPSS was used for modeling and test fitting, and to produce a structural equation model for the theoretical model of teaching effectiveness in relation to VR spatiotemporal characteristics. The model’s proposed index was as follows: χ2 = 101.673, df = 80, IFI = 0.958, CFI = 0.956, TLI = 0.942, and p = 0.051 > 0.05. The overall model adaptation met the standard, and the proposed fit was good. It can be seen that VR time and space interactivity, technical availability and comprehension, interactivity, and learning effect coefficients are negative, and the path coefficients between other variables were positive values, as shown in Table 6 .

TABLE 6 . Model fitting indicators.

Through statistical inspection and analysis of the structural equations, the standard path coefficient and distinctive results were obtained, as shown in Table 7 . The p value of H1 and H4 reached a significant level of 0.01, indicating that VR time and space have a positive and significant impact on learners’ understanding, and imagination has a positive and significant impact on learning effects. The p values of H2, H6, and H9 reached a significant level of 0.05, indicating that learners’ understanding has a positive and significant impact on the learning effect, technical availability has a positive and significant impact on interactivity, and technical availability has a positive impact on learners’ imagination. The p value of H3 reached a significant level of 0.1, indicating that VR time and space have a positive and significant impact on learners’ imagination. The p values of H5, H7, and H8 did not reach a significant level, indicating that VR time and space do not have a positive and significant impact on interactivity. Interactivity has no positive and significant impact on learning effects, and technical availability has no positive and significant impact on understanding. Judging from the size of the standard path coefficient value, the variable effect of H3 was weak.

TABLE 7 . Standardized regression coefficient and its distinctiveness.

4.2.2.2 Path Analysis Model of Factors Influencing Learning in the VR Environment

The relationship model of factors influencing learning is shown in Figure 4 (based on the hypothesis test analysis of the model). This indicated that learners’ understanding and imagination can significantly positively affect learning. VR time and space and technical availability have a positive impact on learners’ understanding and imagination. It can be seen from the effect value between the paths in the figure that in the VR learning environment, the factor exerting the greatest impact on learning was imagination, and the direct effect value of this was found to be 0.274. The second factor was that of comprehension, and the direct effect value of this was 0.073. The effect value of both in terms of learning in the VR environment was 0.347. The impact of VR time and space on understanding was greater than the impact of VR time and space on imagination. The effect value of VR time and space in terms of learners’ understanding was 0.317, and the effect value of VR time and space for learners’ imagination was 0.065. The effect value of technical availability in terms of interactive effects was 0.363,-as shown in Figure 5 .

FIGURE 4 . “You, Calligrapher” operating interface.

FIGURE 5 . Relationship model of VR time and space characteristics influencing skill-based learning.

5 Discussion

Our results indicated that in terms of VR-based educational games, VR time and space and technical availability are the factors that have the greatest effect on learning, and they have a direct impact on the learners’ understanding, imagination, and interactivity. From the perspective of learning effects, learners’ understanding and imagination have a direct effect, but interactivity does not significantly impact it.

The results indicated that overall, our expectations were met. VR space–time variables have a significant effect on learners’ understanding and imagination, indicating that VR’s immersive and trans -time characteristics have a positive impact on teachers and students. The technical availability of VR technology and artificial intelligence also have a positive impact on learners’ imagination and the interactive experience, indicating that technology plays an important role in facilitating learners’ perceptions of the VR environment in a multidimensional way. The 6DOF interactive application can enable learners to engage with learning in a very lifelike way, as they would in the physical world, with VR creating a well-matched projection of real-time and space sensory feedback within a virtual world. Improvements in both learners’ understanding and imagination have a positive impact on learning, indicating that in information processing and in-depth learning, cognitive understanding and imaginative thinking promote the effectiveness of learning. Dizziness and the controller affected the user experience to a certain extent; consequently, the learning effect was impacted negatively.

5.1 Theoretical Implication

Immersive VR technology creates a learning environment that simulates the physical world, allowing learners to obtain sensory immersion ( Hu et al., 2021 ). VR time and space blur the boundary between real-time and space and virtual time and space, creating a sense of presence that is not limited by such physical constraints, enabling teachers and students to communicate face to face and enhancing the possibility for teaching supervision and feedback. With the enhancement of the body’s perception function, multimode characteristics can enhance the level of stimulation, providing lifelike learning contexts that amplify users’ emotional response and sensory experience, making it easier for them to understand key information. Related EEG studies have further confirmed that in a VR environment, the human brain is more likely to present a neural pattern similar to that of the real thing (Petukhov et al., 2020). On a theoretical level, this provides strong evidence that VR’s space–time nature creates a sense of learning “ in situ ” and improves learners’ understanding. Such findings also indicate that the technical availability of VR and artificial intelligence has a positive effect on learners’ imagination and knowledge acquisition.

However, it has also been found that the interactivity of VR technology can pose major problems, and its high-dimensional characteristics can have a negative impact on the learning process. For example, a VR environment has been known to cause higher cognitive loads and physiological discomfort for learners (such as dizziness and unrealistic controller interaction). Therefore, we should be cautious about VR education and be alert to the impact of redundant information with multidimensional perception, which interferes with learners’ information acquisition.

5.2 Practical Implication

As of now, the world is still being affected by the COVID-19 pandemic, and social gatherings are severely restricted. Education has been significantly affected by lockdowns and restriction of in-person teaching, especially where practical skills are concerned. Although knowledge can be imparted via distance learning, it is almost impossible to know whether learners have actually acquired such knowledge. In the case of practical skills, students accessing online learning (e.g., in fields such as art, experimental disciplines, and vocational skills) must draw heavily on their imagination. Teachers can only use cameras to shoot the demonstration process, and learners watch teaching videos in order to learn how to carry out the practices in question for themselves.

Some believe that VR will revolutionize education. Teaching processes that can be completed by VR and AI technology for “teacher and apprentice” teaching (including those that require considerable teacher input, as with Chinese calligraphy) can be applied to distance learning using VR technology, and the addition of AI can effectively improve learners’ understanding and powers of imagination. But at the same time, we must also realize that VR and AI still have a long way to go before popularizing education in general. The technical barriers associated with VR equipment (e.g., VR equipment’s stunning, simulation of interactive handles, tactile simulation, and other technologies) are obstacles that practitioners need to overcome.

The author proposes the following as potential applications for improving the effectiveness of learning through the use of VR technology:

5.3 Improve the Role of VR Education

Given the particularity of distance education, different users clearly need to feel confident about the identity of the other party involved in the VR process. Traditional education has a clear identity authentication relationship in a fixed environment scenario, and it is easier for those involved to establish identity relationships with one another. The identity of the teaching and receiving parties is fixed. The relationship between the two is stable, with a guaranteed sense of trust and authority, and given these conditions, the teaching and learning process can progress and produce results.

5.4 Improve Interaction Through Use of VR Education

The function that Internet-based education cannot achieve for traditional skills teaching is the operability of students. Whether lesson materials are accessed via a computer or using mobile data, the process is limited by the user’s input method, that is, a mouse, keyboard, or touch screen, and these modes of operation cannot achieve the interactive functionality of VR technology, with its ability to simulate reality. However, VR can break this deadlock, combining operational interaction with teaching. This technology not only makes it possible for students and teachers to communicate face to face, with teachers modeling key operations in real time, but it also means that students can actually practice the skills involved and obtain feedback on their performance and progress in real time.

5.5 Improve Understanding of Teaching Content and Performance Using VR

In the age of artificial intelligence, the conversion power of language has greatly increased. Real-time translation of language can be achieved through AI, and real-time translation (or conversion) of language to text and text to language, along with real-time translation of language to images, text to images, and even translation of images themselves, can be achieved. Fully intelligent real-time interpretation will be the basic feature of the AI age. In the VR space, the screen is no longer limited by factors such as size and cost. An image of a teacher, real-time translation of language, and text and images can be displayed in the virtual space, which will greatly enhance students’ understanding. With the popularization of the 5G network (and with the help of cloud storage servers), lightweight VR equipment is on its way to becoming a feature of daily life, and it is likely to have a huge transformative effect on education.

5.6 Improve the Design of VR Education and Learning Initiatives

It can be difficult to know how to improve learning initiatives in any educational form, but VR in education is likely to become more prominent. Virtual environments make it impossible for students to stop listening and to disengage. This is very different from normal online programs, where there may be no supervision mechanism, making it possible for students with poor autonomy to lose interest and learn bad habits (especially in the case of younger students). This problem has become very prominent in the Internet age. During COVID-19 lockdowns, teachers have found it difficult to sustain students’ attention between screens, and the systems used may not have a feedback mechanism. Although video conferencing can effectively alleviate this problem with network technology, it is still impossible to achieve the same effect as classroom teaching in the case of distance learning. Teacher supervision and face-to-face contact with students may be almost zero, and a classroom learning atmosphere is difficult to be established. Therefore, in the era of VR education, there is a need to solve this problem in order to achieve true distance education and provide an alternative form of face-to-face teaching.

6 Conclusion

Through our experimental test with the Chinese “You, Calligrapher” VR software, it was found that VR education has a positive effect on learners and teachers in distance education, and this technology can help teachers and students establish an effective “face-to-face” teaching environment and solve the difficulties associated with supervision and providing teacher feedback. Practical skills education in particular can greatly benefit from this approach, meeting the needs of both teachers and learners. The results show that VR time and space can significantly enhance the learning of practical skills. VR space–time is the most important difference between online teaching and traditional education. Technical availability is key to content presentation and, in combination with AI, is a great way to improve the communicative efficiency of teachers and students. Interaction helps promote more intuitive teaching and learning, and synchronization of understanding and imagination is key to students’ knowledge formation. These are some of the core elements of education and have been shown to have a positive impact on learning effects.

6.1 Limitation and Future Work of Study

The research on VR education in this subject is not deep enough, and there is a lack of more diverse experimental samples of educational content. The scope of the experiments is not wide enough, and further observations and experiments are needed to obtain richer data. In the future, we hope to obtain more samples of VR education applications and expand the experimental data population, such as the elderly and school-age children, to obtain a larger amount of data to test the accuracy of research conclusions.

6.2 Academic Contribution

Research into educational psychology training mainly focuses on educators and future developments. Existing research belongs to the fields of pedagogy and computer science. According to previous studies, research has focused on the linearization of skill-based education, with artificial intelligence and virtual reality technology as variable factors, trying to improve the traditional education model, and the purpose is to improve the efficiency of distance operational skill teaching. We used software to test a contemporary Chinese age-group and obtained relevant data for analysis, to clarify the effectiveness of the method, and to make relevant recommendations (to the government and industry) based on the research results.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author Contributions

PL designed the study, participated in all steps of the research process, and wrote the first draft. TJ made a substantial and direct intellectual contribution to this work, reviewed the literature, revised the manuscript, and participated in the interpretation of the results. ZF has contributed to data collection, statistical analysis, and writing, and is responsible for communication. All authors approved the manuscript and agreed to be responsible for all aspects of the work.

This research was supported by the new engineering construction project of the Chinese Ministry of Education’s collaborative education project (VR virtual simulation collaborative education platform construction) (2018), the “13th Five-Year Plan” project of the development of philosophy and social sciences of Guangzhou City, Research on International Economic Cooperation and Competition in the Bay Area” (2019GZGJ33).

Conflict of Interest

Author ZF was employed by the company Guangzhou Nidu Information Technology Co., Ltd.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Chen, M. X., and Deng, Z. (2015). The Transformation of Learning Evaluation in the Process of Educational Informatization-A Understanding-Based Perspective. Int. J. E-education Res. 36 (10), 12–19. doi:10.13811/j.cnki.eer.2015.10.002

CrossRef Full Text | Google Scholar

Chen, X., Chen, S., Ma, Y., Liu, B., Zhang, Y., and Huang, G. (2019a). An Adaptive Offloading Framework for Android Applications in Mobile Edge Computing. Sci. China Inf. Sci. 62, 82102. doi:10.1007/s11432-018-9749-8

Chen, X., Li, A., Zeng, X. e., Guo, W., and Huang, G. (2015). Runtime Model Based Approach to IoT Application Development. Front. Comput. Sci. 9, 540–553. doi:10.1007/s11704-015-4362-0

Chiang, T. H. (2021). Investigating Effects of Interactive Virtual Reality Games and Gender on Immersion, Empathy and Behavior into Environmental Education. Front. Psychol. 12, 608407. doi:10.3389/fpsyg.2021.6084010.3389/fpsyg.2021.608407

PubMed Abstract | CrossRef Full Text | Google Scholar

Feng, D. (2006). VR's Ontology Exploration. J. Hefei Ind. Univ. (Social Sci. Edition) (01), 167–170. doi:10.3969/j.issn.1008-3634.2006.01.039

He, J. H., Huang, X. L., Han, G. X., Liang, Y. S., and He, X. Q. (2019). Empirical Research on the Influence Factors of VR Educational Game Learning Motivation. Int. J. Electr. Educ. Res. 40 (08), 70–77. doi:10.13811/j.cnki.eer.2019.08.009

Horváth, I. (2021). An Analysis of Personalized Learning Opportunities in 3D VR. Front. Comput. Sci. 3, 673826. doi:10.3389/fcomp.2021.673826

Hsiao, S. C. (2021). Effects of the Application of Virtual Reality to Experiential Education on Self-Efficacy and Learning Motivation of Social Workers. Front. Psychol. 12, 770481. doi:10.3389/fpsyg.2021.770481

Hu, Y. L., Nie, J., Zhang, T. Q., and Wu, T. (2021). The Effectiveness of VR Technology Empowerment Experimental Teaching under the Field of Personal Cognitive Vision. Int. J. Mod. distance Educ. Res. 33 (05), 94–102. doi:10.3969/j.issn.1009-5195.2021.05.010

Huang G, G., Liu, X., Ma, Y., Lu, X., Zhang, Y., and Xiong, Y. (2019). Programming Situational Mobile Web Applications with Cloud-Mobile Convergence: an Internetware-Oriented Approach. IEEE Trans. Serv. Comput. 12, 6–19. doi:10.1109/TSC.2016.2587260

Huang, G., Chen, X., Zhang, Y., and Zhang, X. (2012). Towards Architecture-Based Management of Platforms in the Cloud. Front. Comput. Sci. 6 (4), 388–397. doi:10.1007/s11704-012-2100-4

Huang, G., Xu, M., Lin, F. X., Liu, Y., Ma, Y., Pushp, S., et al. (2017). Shuffledog: Characterizing and Adapting User-Perceived Latency of Android Apps. IEEE Trans. Mobile Comput. 16, 2913–2926. doi:10.1109/TMC.2017.2651823

Huang, H.-M., Rauch, U., and Liaw, S.-S. (2010). Investigating Learners' Attitudes toward Virtual Reality Learning Environments: Based on a Constructivist Approach. Comput. Educ. 55 (3), 1171–1182. doi:10.1016/j.compedu.2010.05.014

Huang, Y. C, Y.-C., Backman, S. J., Backman, K. F., McGuire, F. A., and Moore, D. (2019). An Investigation of Motivation and Experience in Virtual Learning Environments: a Self-Determination Theory. Educ. Inf. Technol. 24 (1), 591–611. doi:10.1007/s10639-018-9784-5

Kyrlitsias, C., Christofi, M., Michael-Grigoriou, D., Banakou, D., and Ioannou, A. (2020). A Virtual Tour of a Hardly Accessible Archaeological Site: The Effect of Immersive Virtual Reality on User Experience, Learning and Attitude Change. Front. Comput. Sci. 2, 23. doi:10.3389/fcomp.2020.00023

Lee, C. C., Hsiao, K. L., and Chen, C. C. (2020). Exploring the Benefit and Sacrifice Factors of Virtual Reality Gameplay. Front. Psychol. 11, 251. doi:10.3389/fpsyg.2020.00251

Li, X. P., Chen, J. Z., Zhao, F. N., Zhang, L., and Zhang, G. G. (2017). AR/VR Research on the Design of Learning Situations. J. Mod. Educ. Technol. 27 (8), 12–17. doi:10.3969/j.issn.1009-8097.2017.08.002

Liang, C., Hsu, Y., Chang, C.-C., and Lin, L.-J. 2013). In Search of an index of Imagination for Virtual Experience Designers. Int. J. Technol. Des. Educ. , 23(4):1037–1046. doi:10.1007/s10798-012-9224-6

Liu, X. B., and Wang, G. X. (2011). Desktop Virtual Experiments Promote the Verification and Analysis of Student Operation Skills’ Migration. J. Distance Education China (11), 4296–4346. doi:10.3969/j.issn.1009-458X.2011.11.009

Liu, X. X., Zhu, N., and Wang, L. F. (2019). Become “Stereotyped”-talk about How Artificial Intelligence, VR and AR Will Change the Future Development of Education. J. Occup. (07), 38–39. doi:10.3969/j.issn.1009-9573.2019.07.016

Makransky, G., Terkildsen, T. S., and Mayer, R. E. (2019). Adding Immersive Virtual Reality to a Science Lab Simulation Causes More Presence but Less Learning. Learn. Instruction 60, 225–236. doi:10.1016/j.learninstruc.2017.12.007

Ninaus, M., and Nebel, S. (2021). A Systematic Literature Review of Analytics for Adaptivity within Educational Video Games. Front. Educ. 5, 611072. doi:10.3389/feduc.2020.611072

Parong, J., and Mayer, R. E. (2018). , Cognitive and affective processes for learning, science in immersive virtual reality. J. Educ. Psychol. 110 (6), 785–797. doi:10.1111/JCAL.12482

Paul, J., and Jefferson, F. (2019). A Comparative Analysis of Student Performance in an Online vs. Face-To-Face Environmental Science Course from 2009 to 2016. Front. Comput. Sci. 1, 7. doi:10.3389/fcomp.2019.00007

Petukhov, I. V., Glazyrin, A. E., Gorokhov, A. V., Steshina, L. A., and Tanryverdiev, I. O. (2020). Being Present in a Real or Virtual World: A EEG Study. Int. J. Med. Inform. 136, 103977. doi:10.1016/j.ijmedinf.2019.103977

Setti, T., and Csapo, A. B. (2021). A Canonical Set of Operations for Editing Dashboard Layouts in Virtual Reality. Front. Comput. Sci. 3, 659600. doi:10.3389/fcomp.2021.659600

Sitterding, M. C., Raab, D. L., Saupe, J. L., and Israel, K. J. (2019). Using Artificial Intelligence and Gaming to Improve New Nurse Transition. Nurse Leader 17 (2), 125–130. doi:10.1016/j.mnl.2018.12.013

Wang, M. (2020). The Optimization Path of VR Educational Publications-The Dual Construction of "content + Situation. J. View Publishing (16), 44–46. doi:10.16491/j.cnki.cn45-1216/g2.2020.16.011

Keywords: VR time and space, artificial intelligence, education strategy, distance education, Chinese calligraphy

Citation: Li P, Fang Z and Jiang T (2022) Research Into improved Distance Learning Using VR Technology. Front. Educ. 7:757874. doi: 10.3389/feduc.2022.757874

Received: 12 August 2021; Accepted: 14 January 2022; Published: 11 February 2022.

Reviewed by:

Copyright © 2022 Li, Fang and Jiang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Tan Jiang, [email protected]

This article is part of the Research Topic

Deep Learning in Adaptive Learning: Educational Behavior and Strategy

International Conference on Interactive Collaborative Learning

ICL 2019: The Impact of the 4th Industrial Revolution on Engineering Education pp 193–205 Cite as

Virtual and Augmented Reality in Science Teaching and Learning

• Charilaos Tsichouridis 17 ,
• Marianthi Batsila 18 ,
• Dennis Vavougios 17 &
• George Ioannidis 19  
• Conference paper
• First Online: 18 March 2020

925 Accesses

1 Citations

Part of the Advances in Intelligent Systems and Computing book series (AISC,volume 1134)

The present research investigates the state-of-the-art concerning virtual and augmented reality lab environments, in science teaching and learning. Both environments are suggested as most appropriate for science education. The study explores the extent to which research has progressed concerning these new tools, while focusing on the extent to which actual educational trials in science classrooms have been completed. To this effect, 19 research papers were identified and reviewed. Conclusions are drawn as regards the effectiveness of these two lab environment types as a function of the age of students, today. The creation of educational VR and AR that resembles reality so closely that it is hard to differentiate between virtual, augmented, and real, thus creating a unified continuum, seems to finally be within our grasp, yet further research is needed as to their optimum use.

• Science education
• Virtual Reality
• Augmented Reality
• Mixed Reality
• Meta-analysis

This is a preview of subscription content, access via your institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Tsihouridis, Ch., Batsila, M., Vavougios, D., Ioannidis, G.S.: The timeless controversy between virtual and real laboratories in science education-“And the winner is…”. In: Auer, M.E., Guralnick, D., Simonics, I. (eds.) Teaching and Learning in a Digital World. Advances in Intelligent Systems and Computing, vol. 2, pp. 1539–1550. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-732014-6_20 . ISBN 978-3-319-73203-9

Jonassen, D.H.: Computers as Mindtools for Schools: Engaging Critical Thinking, 2nd edn. Prentice Hall, New Jersey (2000). 297 pages

Google Scholar  

Han, I., Black, J.B.: Incorporating haptic feedback in simulation for learning physics. Comput. Educ. 57 , 2281–2290 (2011)

CrossRef   Google Scholar  

Freitas, R., Campos, P.: SMART: a SysteM of augmented reality for teaching 2nd grade students. In: Proceedings of the 22nd British HCI Group Annual Conference on HCI 2008: People and Computers XXII: Culture, Creativity, Interaction - Volume 2, BCS HCI, Liverpool, pp. 27–30 (2008)

Cai, S., Chiang, F.K., Wang, X.: Using the augmented reality 3-D technique for a convex imaging experiment in a physics course. Int. J. Eng. Educ. 29 (4), 856–865 (2013)

Chiang, T.H.C., Yang, S.J.H., Hwang, G.J.: An Augmented reality-based mobile learning system to improve students’ learning achievements and motivations in natural science inquiry activities. Educ. Technol. Soc. 17 (4), 352–365 (2014)

Tsihouridis, Ch., et al.: The optimum equilibrium when using experiments in teaching. Submitted to International Journal of Engineering Pedagogy (iJEP)

Ahn, S., Ames, A.J., Myers, N.D.: A review of meta-analysis in education: methodological strengths and weaknesses. Rev. Educ. Res. 82 (4), 436–476 (2016)

Shelby, L.B., Vaske, J.J.: Understanding meta-analysis: a review of the methodological literature. Leis. Sci. 30 , 96–110 (2008)

Sapounidis, Th., Demetriadis, S.: Tangible versus graphical user interfaces for robot programming: exploring cross-age children’s preferences. Pers. Ubiquit. Comput. 17 , 1775–1786 (2013)

Bakas, Ch., Katsikis, A., Mikropoulos, T.: Virtual reality as a mental representations creation tool: a science example. In: Dimitrakopoulou, A. (ed.) Proceedings of the 3rd Conference ETPE (ICT in Education), Volume B, Aegean University, Rhodes, pp. 37–46 (2002)

Patsalou, Th., Charitos, D., Martakos, D.: Evaluating the use of virtual environment to support chemistry teaching. In: Dimitrakopoulou, A. (ed.) Proceedings of the 3rd Conference ETPE (ICT in Education), Volume B, Aegean University, Rhodes, pp. 491–496 (2002)

Ibanez-Espiga, B.M.: Experimenting with electromagnetism using augmented reality: impact on flow student experience and educational effectiveness. Comput. Educ. 71 , 1–13 (2014)

Hsu, Y.-S., Lin, Y.-H., Yang, B.: Impact of augmented reality lessons on students’ STEM interest. Res. Pract. Technol. Enhanc. Learn. 12 , 2–14 (2017)

Pirker, J., Lesjak, I.: An educational physics laboratory in mobile versus room scale virtual reality – a comparative study. i-JOE 13 (8), 106–120 (2017)

Trindade, J., Fiolhais, C., Almeida, L.: Science learning in virtual environments: a descriptive study. Br. J. Educ. Technol. 33 (4), 2–18 (2002)

Akai, C.: Depth perception in real and virtual environments. An exploration of individual differences, Thesis for the Degree of Master of Science, School of Interactive Arts & Technology, University of British Columbia (1998). 109 pages

Crăciun, D., Bunoiu, M.: Boosting physics education through mobile augmented reality. In: TIM17 Physics Conference, AIP Conference Proceedings, vol. 1916, pp. 050003-1–050003-6. AIP Publishing (2017). https://doi.org/10.1063/1.5017456 . ISBN 978-0-7354-1608-6, pp. 1–6

Strzys, M.P., Kapp, S., Thees, M., Klein, P., Lukowicz, P., Knierim, P., Schmidt, A., Kuhn, J.: Physics holo.lab learning experience: using smartglasses for augmented reality labwork to foster the concepts of heat conduction. Eur. J. Phys., 1–14 (2018)

Huang, K.-T., Ball, C., Francis, J., Ratan, R., Boumis, J., Fordham, J.: Augmented versus virtual reality in education: an exploratory study examining science knowledge retention when using augmented reality/virtual reality mobile applications. Cyberpsychol. Behav. Soc. Netw. 22 (2), 105–110 (2018)

Tibola, L.R., Herpich, F., Fernanda da Silva, P., Tarouco, L.M.R.: Experience in teaching science in virtual environment. Int. J. Innov. Educ. Res. 7 (4), 23–43 (2019)

Hussein, M., Natterdal, C.: The benefits of virtual reality in education – a comparison study, Bachelor of Science Thesis in Software Engineering and Management, University of Gothenburg Sweden, pp. 1–15 (2015)

Techakosit, S., Nilsook, P.: Using augmented reality for teaching physics. In: Proceedings of International e-Learning Conference, IEC, Global Trends in Digital Leaning, BITEC Banga, Bangkok, Thailand, pp. 282–285 (2015)

Nielsen, B.L., Brandt, H., Swensen, H.: Augmented reality in science education - affordances for student learning. NORDINA 12 (2), 152–174 (2016)

Download references

Author information

Authors and affiliations.

University of Thessaly, Volos, Greece

Charilaos Tsichouridis & Dennis Vavougios

Directorate of Secondary Education, Ministry of Education, Larissa, Greece

Marianthi Batsila

University of Patras, Patras, Greece

George Ioannidis

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Charilaos Tsichouridis , Marianthi Batsila , Dennis Vavougios or George Ioannidis .

Editor information

Editors and affiliations.

Carinthia University of Applied Sciences, Villach, Austria

Prof. Dr. mult. Michael E. Auer

Technische Universität Dresden, Dresden, Germany

Prof. Dr. Hanno Hortsch

King Mongkut's University of Technology North Bangkok, Bangkok, Thailand

Prof. Dr. Panarit Sethakul

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Cite this paper.

Tsichouridis, C., Batsila, M., Vavougios, D., Ioannidis, G. (2020). Virtual and Augmented Reality in Science Teaching and Learning. In: Auer, M., Hortsch, H., Sethakul, P. (eds) The Impact of the 4th Industrial Revolution on Engineering Education. ICL 2019. Advances in Intelligent Systems and Computing, vol 1134. Springer, Cham. https://doi.org/10.1007/978-3-030-40274-7_20

Download citation

DOI : https://doi.org/10.1007/978-3-030-40274-7_20

Published : 18 March 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-40273-0

Online ISBN : 978-3-030-40274-7

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us

Virtual reality: A brief survey

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2023 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

This paper is in the following e-collection/theme issue:

Published on 6.11.2023 in Vol 25 (2023)

Evaluation of a Virtual Reality Platform to Train Stress Management Skills for a Defense Workforce: Multisite, Mixed Methods Feasibility Study

Authors of this article:

Original Paper

• Murielle G Kluge 1, 2 , PhD   ; 
• Steven Maltby 1, 2 , PhD   ; 
• Caroline Kuhne 1, 3 , BSc   ; 
• Nicole Walker 4 , MPsych ; 
• Neanne Bennett 5 , MPsych   ; 
• Eugene Aidman 2, 6 , PhD   ; 
• Eugene Nalivaiko 1, 2 , PhD   ; 
• Frederick Rohan Walker 1, 2 , PhD  

1 Centre for Advanced Training Systems, Faculty of Health, Medicine & Wellbeing, The University of Newcastle, Callaghan, Australia

2 School of Biomedical Sciences & Pharmacy, Faculty of Health, Medicine & Wellbeing, The University of Newcastle, Callaghan, Australia

3 School of Psychological Sciences, College of Engineering, Science and Environment, The University of Newcastle, Callaghan, Australia

4 Army School of Health, Australian Defence Force, Canberra, Australia

5 Joint Health Command, Department of Defence, Canberra, Australia

6 Human and Decision Sciences Division, Defence Science & Technology Group, Edinburgh, Australia

Corresponding Author:

Frederick Rohan Walker, PhD

Centre for Advanced Training Systems

Faculty of Health, Medicine & Wellbeing

The University of Newcastle

University Drive

Callaghan, 2300

Phone: 61 024921 5012

Email: [email protected]

Background: Psychological stress-related injuries within first-responder organizations have created a need for the implementation of effective stress management training. Most stress management training solutions have limitations associated with scaled adoption within the workforce. For instance, those that are effective in civilian populations often do not align with the human performance culture embedded within first-responder organizations. Programs involving expert-led instructions that are high in quality are often expensive.

Objective: This study sought to evaluate a tailored stress management training platform within the existing training schedule of the Australian Defense Force (ADF). The platform, known as Performance Edge (PE), is a novel virtual reality (VR) and biofeedback-enabled stress management skills training platform. Focusing on practical training of well-established skills and strategies, the platform was designed to take advantage of VR technology to generate an immersive and private training environment. This study aimed to assess the feasibility of delivering the VR platform within the existing group-based training context and intended training population. In this setting, the study further aimed to collect data on critical predictors of user acceptance and technology adoption in education, including perceived usability, usefulness, and engagement, while also assessing training impacts.

Methods: This study used a mixed methods, multisite approach to collect observational, self-reported, and biometric data from both training staff and trainers within a real-world “on-base” training context in the ADF. Validated scales include the Presence Questionnaire and User Engagement Scale for perceived usefulness, usability, and engagement, as well as the State Mindfulness Scale and Relaxation Inventory, to gain insights into immediate training impacts for specific training modules. Additional surveys were specifically developed to assess implementation feedback, intention to use skills, and perceived training impact and value.

Results: PE training was delivered to 189 ADF trainees over 372 training sessions. The platform was easy to use at an individual level and was feasible to deliver in a classroom setting. Trainee feedback consistently showed high levels of engagement and a sense of presence with the training content and environment. PE is overall perceived as an effective and useful training tool. Self-report and objective indices confirmed knowledge improvement, increased skill confidence, and increased competency after training. Specific training elements resulted in increased state mindfulness, increased physical relaxation, and reduced breathing rate. The ability to practice cognitive strategies in a diverse, private, and immersive training environment while in a group setting was highlighted as particularly valuable.

Conclusions: This study found the VR-based platform (PE) to be a feasible stress management training solution for group-based training delivery in a defense population. Furthermore, the intended end users, both trainers and trainees, perceive the platform to be usable, useful, engaging, and effective for training, suggesting end-user acceptance and potential for technology adoption.

Introduction

Stress management training in the workforce.

The negative impacts of unmanaged stress exposure are well documented in first-responder populations [ 1 , 2 ]. Consequences of prolonged exposure to unmanaged psychological stress can include, but are not limited to, changes in cognition, judgment, motivation, and mood [ 3 ]. With prolonged exposure to stress, disruptions in mood and cognition can transition into diagnosable pathologies such as burnout, anxiety, depression, and trauma [ 4 - 6 ].

To protect their workforce and those under their care, many first-responder organizations have sought to deliver scalable and structured stress management training to their staff. Stress management training includes several different forms and components. A useful viewpoint is the definition of stress management training as “the application of any set of techniques ( e.g. , exposure training, relaxation, biofeedback, and cognitive behavioural therapy) with the intent to improve the way people cope with stress” [ 7 ]. Furthermore, existing stress management training programs can be broadly classified into two types: (1) stress-inoculation training—repeated exposure to a stressor to develop tolerance (eg, outdoor adventure, live fire, and mission rehearsal) and (2) cognitive and psychological skills training conducted in a nonstressful setting. Stress inoculation training programs have been popular for military and first-responder training organizations when the stressor is predictable and likely (eg, physical and verbal altercation or combat [ 8 , 9 ]). Cognitive and psychological skills training, also often referred to as resilience training, can include mindfulness-based, cognitive-behavioral strategies and relaxation techniques [ 10 - 15 ]. Breathwork and mindfulness-based training interventions have documented efficacy in both clinical and nonclinical settings [ 16 , 17 ]. These strategies are being increasingly assessed in workforce and workplace contexts [ 18 - 21 ]. Cognitive strategies, including goal setting and emotional and attentional control, have also been shown to positively impact psychological well-being in military personnel [ 22 - 25 ]. Although there is growing evidence on the benefit of training cognitive stress management skills in first responders and similar occupational cohorts [ 26 - 30 ], further research is required to inform best practice strategies on how to effectively implement and scale training within and across organizations. A major challenge in this context is that many workplace and training organizations deliver training as scheduled activities and in groups.

The Australian Defence Force (ADF) has a well-established stress-management training platform (BattleSMART) based on cognitive behavioral therapy tailored to ADF members [ 31 ]. Consistent with other stress management programs, BattleSMART relies on the provision of instructional and theoretical materials on using cognitive and psychological skills [ 31 ]. Although well accepted, BattleSMART has been constrained, with respect to skill establishment, by the number of expert facilitators and the time required to deliver the program. Robustly establishing psychological skills, as is the case with other complex skills, requires extensive time for skill rehearsal and expert-led facilitation [ 18 ]. A major challenge connected to the delivery of practical stress management skills is that this type of training benefits from a private and focused training environment. Hence, it is typically facilitated in one-on-one sessions. However, like many training organizations, ADF typically organize and operate their activities via a group-based structure. A change from group-based to one-on-one training for stress management skills instruction would represent a significant burden to the organization.

The Use of Virtual Reality for Stress Management Training

Virtual reality (VR) is an interesting solution for stress management training. VR provides a technical platform to place skill development and expert-led instruction “into the headset.” Moving from a human delivered to digitally delivered instruction can mitigate many issues associated with specialist workforce limitations and allow for flexibility in when and where training can occur. Although similar things can be said about many digital software solutions, the VR headset can create a private and immersive environment that can be particularly beneficial for both group-based delivery and the nature of the subject matter. Additional benefits of VR over other conventional 2D-based platforms include increased immersion, interaction through handheld controllers, a strong sense of presence, engagement, and student motivation [ 32 - 34 ]. Thus, VR-delivered training may provide a viable solution to circumvent the major challenges associated with group-delivered implementation of stress management training for large organizations, including defense.

Several VR and biofeedback-integrated training applications have already been trialed in military and police populations, including those targeting stress inoculation, passive relaxation, and breath control [ 9 , 35 - 38 ]. To our knowledge, however, there have been no comprehensive stress management training programs that teach a diverse range of stress management skills that are appropriate for first-responder organizations, such as the Australian military.

The Performance Edge Stress Management Training Platform

To address the existing unmet needs for practical and scalable training of stress management skills for first-responder populations, we developed a new and comprehensive VR-based training platform called Performance Edge (PE) [ 39 ]. In collaboration with the ADF, PE was specifically developed to target an early career training population and be aligned with ADF values. Evaluation of the first PE module, which focused on training controlled breathing skills, demonstrated the in-principal suitability of the technology and training approach [ 39 ]. Building on the initial work, the modular PE platform was expanded to include 5 modules, each of which provided fundamental skill training for evidence-based stress management strategies adapted from cognitive-behavioral therapy and acceptance and commitment therapy. More details on the PE training concept, the framework and specific module content are provided in Multimedia Appendix 1 .

This study has the following objectives:

• Evaluate the feasibility and implementation of the 5-module PE prototype training package, delivered on-site at multiple military locations to its intended ADF target population, and within existing training activities and group-based, classroom settings.
• Assess critical predictors of technology adoption (eg, perceived usability, and usefulness), engagement, and training impact.

PE Training Platform and Framework

The PE software was delivered on the Oculus Quest (Meta), a freestanding VR headset with 2 handheld controllers and an inside-out tracking system. A total of 20 headsets using the Oculus for Business Enterprise platform were used, which supported fleet management, content uploading, and battery monitoring. Respiratory signals were collected and integrated into VR training using a GoDirect respiratory belt (Vernier). A custom software application was developed and installed to translate and display the breathing traces and rates (breaths per minute) from the belt into the headset in real time to facilitate biofeedback for training.

The PE software application used in this study contained a full menu management system and 5 individual training modules. Each module focused on a specific skill or cognitive strategy adapted from cognitive-behavioral therapy and acceptance and commitment therapy principles ( Table 1 and Multimedia Appendix 1 ). All PE modules included an introductory video, guided narration, personalized learning, practical training, user interactions, feedback, performance measures, and the opportunity for repetition.

The PE interface adopts a futuristic design that pays homage to space-themed computer games ( Figures 1 A-1D). Although the target audience was military, visual design features were created using a neutral pallet without specific reference to military design, situations, or terminology. This was in part due to military triservice-specific branding but also emphasized that the skills can be relevant for any stress-provoking situation, either work related or in everyday life. Alignment with ADF values was generated using a clear and directive tone for instructions, feedback, and explanations, with an emphasis on practical elements. All the exercises were designed to take advantage of the immersive nature and interactive capability of VR technology to create an engaging learning and training environment. Examples can be found in Multimedia Appendices 2 and 3 , containing walk-through videos from the user's view within the VR headset.

Study Participants and Recruitment

Participants were recruited from the initial employment training (IET) programs at 3 military bases in Australia. The IET program is the first service-specific training that military personnel undergo after basic training. As both the existing stress management training program (BattleSMART) and the PE platform were developed for military personnel early in their careers, this population represents the target training population. Participants were informed of the VR training and research study that took part as part of their regular training schedule 1 to 2 weeks prior by their commanding officers, who also provided them with Participant Information Statements. Participants were invited to attend multiple PE training sessions over a week in May 2020, June 2021, November 2021, and March 2022. On-base training staff members were also invited to attend open exploration sessions to use PE. A total of 189 participants were involved in the study across all locations (156 ADF trainees and 33 training staff members). Participant numbers for each module differed, as not all trainees were available across all delivery days at each site, and not all modules were tested at all locations (eg, trial week 2 at trial site 2 was shortened due to the COVID-19 lockdown). Furthermore, 12 trainees attended 2 separate trial weeks, and their data were only included in the analysis for their first attendance.

Ethical Considerations

Research activities were reviewed and approved by the Australian Defense Science and Technology Low Risk Ethics Panel (Protocol Land Division 17-19) and coregistered with The University of Newcastle Human Research Ethics Committee (H-2020-0020). All participants provided written informed consent to participate in this study. Participants were not reimbursed and did not receive compensation for participating in the study. The study was conducted as a scheduled activity within the IET program; however, there was clear communication that participation in the research study was voluntary and outside of their training requirements. An alternative work- or training-related activity, as specified by their commanding officer, was provided to trainees who chose not to participate. A participant ID number was used to match responses where necessary, and no identifiable information (name, rank, or ID number) was collected. Any identifiable data or information provided by a participant within the survey responses were redacted to ensure the anonymity of the participants.

Study Design

A mixed methods approach was applied to evaluate the delivery of PE as a multiday classroom-based training program across the 3 training sites. The trainees completed one to five PE modules on consecutive days within a training week. Training and subsequent assessment of each module occurred within a single 1-hour session in a group setting with 8 to 15 participants simultaneously ( Figures 1 E and 1F and Figure 2 ).

The trainees completed pre- and posttraining surveys specific to each training module. Each training session was concluded with a 10-minute group discussion focused on providing an opportunity for group feedback.

The on-base training staff attended a single unstructured exploration session, in which any module or component could be explored, and completed the posttraining trainer survey only.

Measures and Data Collection

This trial employed observational and objective biometric data, as well as self-report and focus group data related to multiple aspects of PE training delivery.

Self-Report Instruments

Training gains measures.

Each pre- and posttraining survey contained validated scales and general questions on relevant constructs and domains, informed by the technology acceptance model. This included questions to assess technology acceptance (perceived usability, perceived usefulness, and implementation feedback), relevance of content and framework, previous work experience, virtual reality, stress management skills, and perceived training impact. A full list of questions is provided in Multimedia Appendix 1 , Table S1.

The study-specific questions were adapted from those previously developed and administered in studies assessing VR training and the first PE pilot study [ 39 ]. Questions were constructed using a 3-step process (defining construct, content domains, item generation, and determining the format) from VR experts, teaching, training experts, and ADF members. Questions were drafted by investigator MK and finalized using an iterative approach, with feedback from all study investigators using multiple choice, 5-point Likert scale, or an open-ended format. A list of surveys, validated scales, validity, and relevant references are provided in Tables 2 and 3 . All surveys were distributed using QR codes and Office Forms (Microsoft Office).

a VR: virtual reality.

b ADF: Australian Defense Force.

Trainer Surveys

Training staff received a tailored trainer survey after exploring PE content. Questions were adapted from trainee surveys and included additional questions on the suitability of the training framework and technology delivery within the broader organization.

Observational Data

The research team documented quantitative data, including costs, time frames, hardware charging, and set-up requirements.

Objective Biometric Data

Respiratory signals were sampled every 0.05 seconds (20 Hz) and translated into respiratory rate (breaths per minute), calculated after every inhalation. Rates are calculated via a peak detection algorithm using the derivate of data points after smoothing low-pass filtering and are updated after each peak inhalation detection. The average respiratory rate for each 2-minute exercise period was collected and saved, time-stamped, and sent using Wi-Fi to an external cloud-based server with data linked to the serial number of the corresponding headset. Respiratory data integration and collection occurred in modules 2 and 3, respectively. All the modules collected and recorded the selection choices, interactions, and time spent in each exercise.

Data Analysis

Data from all locations were pooled for analysis unless specifically mentioned in the text, whereas data from trainees and training staff were analyzed and reported separately. Self-reported and objective data were analyzed using Prism (version 8; GraphPad) and JASP (version 0.16.3).

Self-reported data were summarized and presented as mean (SD) or absolute number of responses, and all responses were included. Validated scales were calculated per protocol and analyzed using a 1-tailed parametric test (paired t test) or nonparametric equivalent (Wilcoxon signed-rank test), where assumptions were violated.

Respiratory rate data were analyzed using one-way repeated measures ANOVA adjusted for multiple comparisons and a paired-sample t test to compare average respiratory rates during training in M2 and average respiratory rates before and after training in M3.

Both P values and Bayes factor (BF) have been reported. Values of P <.05 are reported as indicators of significance and a BF 10 >3 is considered decisive evidence in favor of a difference or effect.

Training Population and Existing Skill Level

Overall, 40% (50/126) reported previous experience with VR technology, largely in the recreational gaming context with 50% (25/50) reporting less than1 hour, 28% (14/50) less than 10 hours, and 22% (11/50) reporting over 10 hours of total exposure. Overall confidence in the set-up and use of VR technology was moderate (mean 3.4, SD 1.2; 1=not at all confident, 5=extremely confident). The remaining 60% (114/126) of trainees had not previously used VR.

Trainees indicated general awareness and theoretical knowledge of stress management, but the perceived utility of specific skills at the outset of training was modest. When questioned about day-to-day attention to their thoughts, emotions, and initial behaviors (“inner world”), trainees reported being very aware (33/64, 52%) or somewhat aware (31/64, 48%) of the degree to which their thoughts and emotions influenced each other and their behaviors. Nearly all participants (62/63, 98%) indicated that they were either very (33/63, 52%) or somewhat (29/63, 46%) aware of the impact that stress can have on their inner world. Of the specific skills taught in PE, controlled breathing was the most highly reported stress management skill used in the cohort pretraining, with 80% of trainees (64/80, 80%) previously engaging in controlled breathing. Although awareness of controlled breathing as an effective stress management skill was high (80/80, 100%; very or somewhat aware), it was primarily applied in a physical and exercise context (eg, target shooting and combat training) rather than for stress management per se. Few other skills and stress management concepts were understood to be effective strategies to reduce stress, and their use was limited. Only 15% (8/54), 23% (23/98), and 36% (15/42) of trainees reported having used progressive muscle relaxation, grounding, or an acceptance strategy to manage challenging emotions, respectively.

Engagement and Sense of Presence With PE Training

Engagement with PE training was assessed for each module using the validated User-Engagement Scale–Short Form (UES-SF; [ 40 ]) for digital domains, which includes the subdimensions of esthetic appeal, focused attention, perceived usability, and reward. All 5 modules were positively rated by the trainees ( Figure 3 A), with the highest level of user engagement reported for M2—Controlled Breathing (3.9, SD 0.54) and lowest for M5—Managing Emotions (3.5, SD 0.60; 5-point Likert scale, >3=positive), with no modules scoring below 3.0 (which would indicate a negative assessment) in any UES-SF subdimensions. Across all modules, perceived usability was consistently the highest scoring subdimension.

To assess presence within the experience, the Presence Questionnaire was administered after each training module ( Figure 3 B). All the modules were perceived as having an overall feeling of presence. The sense of presence ranged from 4.7 (SD 0.94) in M5—Managing Emotions, to 5.1 in M2—Controlled Breathing, and M3—Progressive Muscle Relaxation (PMR) (SD M2 0.67; SD M3 0.73; 7-point scale). The single item assessing perceived privacy was rated above 3.6 for each module (5-point scale, Figure 3 C).

By design, all training modules and most exercises within PE contain interactive components, including actions and response options for trainees. Trainees responded to interactions within 6 to 31 s, with response times varying by the number of response options, complexity, and familiarity with the exercise. Although answer options were never considered “correct” or “incorrect,” certain exercises and scenarios aimed to invoke a particular emotion or thought response. For example, 87% (32/38) of trainees selected either relaxed or happy as the main emotion felt during an intentionally relaxing 360° video beach environment in M5—Exercise 1.

In an open-ended response section, immersion and privacy of VR technology were named the most beneficial components. This observation was supported by additional question responses, in which both immersion and privacy were rated to be useful or extremely useful in supporting training and attention focus (immersion=3.9, SD 0.90 and privacy=3.7, SD 0.86, respectively; 1=not at all useful and 5=extremely useful; Figure 3 D). Biofeedback functionality, interactive elements, and concept visualization were also named as particularly positive features of PE training across all modules in open-ended questions. Survey responses consistently provided positive feedback on individual modules while endorsing the overall length of individual exercises.

When asked which elements of the training could be improved, open-ended responses primarily included the expansion of existing elements, escalation of provocative content and scenarios, and inclusion of ADF-specific content.

Module-Specific PE Training Outcomes

Module-specific training perceptions were assessed using the objective and qualitative outcomes relevant to the specific training objectives of each module. The central training objective within M2—Controlled Breathing, was to gain awareness and control over breath cadence, specifically to maintain a slow and steady breathing rate across escalating training exercises ( Figure 4 A). Consequently, changes in breathing rates were used to gain insights into training outcomes with the hypothesis that exercises within PE would result in a reduction in breaths per minute (BPM) compared with the initial, noninstructed breathing rates recorded at the start of the module. Initial baseline respiratory rates, without prompts, were 16.3 (SD 3.9) breaths BPM for trainees and 16.8 (SD 5.6) BPM for training staff. When prompted to actively reduce their breathing rate, initial mean breathing rates reduced to 10.3 (SD 3.4) BPM (trainees) and 10.7 (SD 2.1) BPM (staff). Live biofeedback with a respiratory trace (termed “controlled breathing assisted”) supported further reductions to 8.5 (SD 3.2) BPM (trainees) and 8.5 (SD 3.1) BPM (staff). Participants subsequently sustained reduced controlled breathing rates without live biofeedback (“controlled breathing non-assisted”), and in subsequent exercises incorporating a distracting environment (“controlled breathing concert”) and shooting tasks. Training effects for M3—Progressive Muscle Relaxation were assessed using changes in breathing rates and state relaxation before and after the 15-minute guided relaxation exercise. Mean respiratory rates were significantly reduced compared with baseline at the start of the exercise, from 19 (SD 3.8) to 6.9 (SD 2.3) BPM (mean difference −12.74, SD 3.4; t 26 =19.198, P <.001; BF 10 =1.413e+14).

The Physical Assessment Scale (PAS) and Cognitive Tension Scale (CTS) subscales of the Relaxation Inventory were administered before and after both M2 and M3 to quantify relaxation states ( Figure 4 B). Owing to the common use of controlled breathing and PMR for relaxation, we hypothesized that training would result in an increased self-report rating on the PAS and reduced ratings on the CTS. As hypothesized, physical relaxation (PAS) significantly increased after both M2—Controlled Breathing (mean increase 7.78, SD 9.9; t 30 =−4.354, P <.001; BF 10 =374.435) and M3—PMR training (mean increase 11.27, SD 14.49; w=224, Z =−5.087; P <.001, BF 10 =891.313). Similarly, cognitive tension scores (reverse scored to indicate relaxation) increased after both modules (M2 mean increase 2.53, SD 6.04; w=87, Z =−2.451 , P =.007; BF 10 =22.523; M3 mean increase 3.4, SD 5.10; w=193, Z =−4.406, P <.001; BF 10 =3543.625).

Given the training objectives for M4 and M5, we explored the assumption that training may increase mindfulness, as measured using the State Mindfulness Scale ( Figure 4 C). M4—Grounding increased mindfulness scores from 63.42 (SD 17.26) to 69.98 (SD 16.57) , a significant change (mean increase 6.14, SD 15.22; w=906.5, Z =−4.036, P <.001; BF 10 =5067.078). A trend toward increased State Mindfulness Scale scores following M5—Managing Emotions training was not statistically meaningful (mean increase 2.97, SD 12.45; w=155, Z =−1.351, P =.09, BF 10 =0.743). This was true for both the Body (mean increase 0.50, SD 5.195; w=120.000, Z =−1.410, P =.08; BF 10 =0.830), and Mind (mean increase 2.47, SD 8.97; w=186.5, Z =−1.449, P =.08; BF 10 =1.452) subscales. However, the Bayes factor for the Mind subscale (BF 10 =1.452) provided anecdotal evidence of an increase in mental-state mindfulness.

In addition to objective outcomes, self-reported data on training impact, efficacy, and perceived value were collected after each module and final training session. Following training, trainees reported a deeper understanding of the theoretical concepts taught within PE, specifically for underlying practical skills (M1: 3.9, SD 0.7; M2: 3.8, SD 0.7; M3: 3.8, SD 0.71; M4: 3.8, SD 0.7; M5: 3.8, SD 0.7; 1=strongly disagree; 5=strongly agree; Figure 5 A). Perceived skill competency improved after training for M2—Controlled Breathing, M3—PMR, M4—Grounding and M5—Managing Emotions (M2: 3.7, SD 0.8; M3: 3.8, SD 0.7; M4: 3.7, SD 0.6; M5: 3.3, SD 0.9; 1=strongly disagree; 5=strongly agree; Figure 5 C). Statistical analysis was conducted on self-report items asking about the likelihood of engaging in these skills. We hypothesized that PE training would increase the likelihood of engaging in the respective skill in a stressful context. Trainees indicated they were more likely to actively consider their thoughts and emotions before reacting to a stressful event (mean 3.8, SD 0.8 on a 5-point Likert scale; mean difference [post-pre]: 0.46, SD 0.7; Wilcoxon signed-rank test Z =−2.548, P =.002; BF 10 =27.94). Similarly, trainees were more likely to use grounding skills after PE training compared with pretraining (mean increase 0.55, SD 0.9; Wilcoxon signed-rank test Z =−2.353, P =.008; BF 10 =9.118). Although the likelihood of using other skills did not change pre-post training, trainees indicated they were likely to engage in controlled breathing (4.0, SD 0.8) and use an acceptance strategy to manage their emotions (3.5, SD 0.9) the next time they encountered a stressful or challenging event (1=not at all likely, 5=extremely likely; Figure 5 B), indicating that pretraining levels were already favorable toward these skills. The overall intention to use PMR and grounding skills after completing M3 and M4 training was only modest (mean 3.1, SD 1.2 and 3.1, SD 1.1, respectively). Overall, 66% (75/113) of participants were confident or extremely confident that PE represented a useful and effective platform to train and practice stress management skills (3.9 SD, 0.8, 1=not at all confident; 5=extremely confident, Figure 5 D), whereas only 2 of 113 trainees stated they were not confident that the platform was a useful stress management training tool.

Trainer Feedback

Most trainers agreed or strongly agreed that trainees engaged with the training content (24/29, 83%) and that the platform delivered effective practical training on stress management skills (18/28, 64%; only 2 trainers disagreed). Furthermore, 50% (15/30) of the training staff agreed that the platform provided valuable knowledge transfer, whereas 5 disagreed with that statement ( Figure 6 A). Importantly, 79% (23/29) of the trainers stated that they were confident or extremely confident in their ability to deliver VR training in the classroom. Verbal feedback and responses to an open-ended question indicated that staff members were positively surprised by the platform and how the technology supported fundamental skill development. With expected stress inoculation training, many saw great benefits in approaching cognitive and emotional skills training within the immersive and private environment of the VR headset. Engaging and interactive components were named particularly beneficial and contrasted with the traditional delivery approach using PowerPoint-based materials.

Trainer quotes from survey:

[Performance Edge modules are] really good at helping people identify their responses to situations.

It is very beneficial to include practical training related to a soldier mindset.

Although trainers praised the use of nonmilitary design, language, and introduction segments, both trainees and trainers suggested that the final exercises in each module would benefit from being placed within a relevant military context.

Alignment of PE With the Existing ADF Training Framework

The PE platform was developed as an extension of the ADF stress management framework. Compared with the existing approach, most staff and trainees indicated training added value or was superior ( Figure 6 B). Specifically, for staff, 41% (11/27) believed PE to be a valuable addition, which supports existing stress management training, and 19% indicated that PE was superior (a combined total of 60%). Of the remaining staff, 26% indicated that they were unable to recall the existing stress management training and, therefore, were unable to make a comparison, 11% felt it neither supported nor detracted from existing training, and only 1 trainer (out of 27) indicated that PE was inferior. For trainees, 57% (86/150) believed PE to be a valuable addition, supporting existing stress management training, and 28% indicated that PE was superior (combined total of 85%). Moreover, 12% felt that it neither supported nor detracted, and only 1 trainee (of 150) indicated that it was inferior to the existing approach.

Delivery of PE Training

Data on trial logistics and requirements were also collected to inform the implementation strategies and determine their feasibility. A total of 372 PE training sessions were delivered over 9 days across 4 training weeks. Average “in headset” training time (in minutes) for each module was M1=22.9 (SD 4.2), M2=17.4 (SD 2.8), M3=18.9 (SD 1.8), M4=33.9 (SD 1.3), and M5=32.6 (SD 6.2).

Content was loaded on headsets off-site before each trial week (duration approximately 20 minutes) and the initial on-site hardware set-up of 20 headsets required approximately 1 hour by a study team member. A full headset charge from 0% to 100% required approximately 3 hours. After three consecutive training sessions (approximately 1.5-hour active runtime plus standby time of approximately 5 h), the average headset battery charge was 40%. The hardware was charged overnight between training days. A silicone cover was used over the headset face foam. The silicone cover, outside the headset, controllers, and respiratory belt were wiped down before and after each session using a skin-friendly VR head mounted display cleaning wipe provided to the trainees (a typical classroom set-up is shown in Figure 1 E).

Trial weeks in 2021 occurred under COVID-19 social distancing requirements (1.5 m distancing between individuals; Figure 1 F). Although the first session each week required specific instructions and clarifications on how to set-up, use, and navigate the hardware, trainees required limited input in subsequent sessions. Additional instructions were provided for the biofeedback-enabled modules to ensure that the respiratory belt was plugged into the headset. Trainees required repeated and specific instructions on how to re-enter the VR field of view. Attendance by two instructional staff members to support the delivery was useful, but only required during the first session each week to support information and consent processes for the research component of the trial. For standard operational training delivery outside the research context, one team member would be sufficient.

In addition, 98% (363/372) of the training sessions were delivered without any issues or difficulties identified. During the initial March 2020 trial (n=42 total participants), a total of 8 technical issues were reported. All but two technical issues were resolved with assistance during the session. The remaining two issues related to automated data capture were not apparent until the completion of the training. Five technical issues related to the connection between the respiratory belt and the VR headset (specifically, the USB-C wired connection). The USB-C connection issue was resolved for subsequent trial dates by replacing the USB-C adapter with a robust model. The remaining issues included freezing of the VR headset screen, operator difficulties, and set-up of the VR “guardian area.”

During the November 2021 trial (n=37 participants), 7 trainees were unable to complete the final exercise of M1 owing to a loading issue in the software, which was subsequently addressed and resolved for the March 2022 trial. Two additional trainees accidentally exited the training module. No VR-induced motion sickness was reported directly or in the survey responses during or after training. The research team was informed of a single unanticipated response to one exercise in the M4—Grounding module, in which a trainee reported to the ADF chaplain that the memory recall component of the training triggered emotional distress. As a result, a warning was included in subsequent versions, noting the potential for an emotional response.

Principal Findings

This trial describes the delivery of the PE prototype as a VR-based practical training platform for fundamental stress management skills within a workplace setting. The outcomes demonstrated the perceived usefulness, feasibility, usability, and positive training outcomes of the technology platform, training concept, and specific training modules within the intended real-world context and training population.

PE was delivered to 189 military trainees during consecutive 1 hour in-classroom training sessions of up to 20 trainees at a time and 5 modules in total. The distribution and utility of the biofeedback-integrated VR system were portable, easy to set-up, and suitable for the needs and requirements of the training organization and the target training population. Both the intended training population and their training staff perceived the platform to be useful, easy to use, engaging, immersive, and aligned with the existing stress management training framework ( Figures 4 and 6 ). Based on the technology acceptance model, our results for perceived usefulness, immersion, and engagement suggest that future adoption of the platform is highly feasible [ 44 , 45 ]. The ability to practice cognitive strategies in a diverse, private, and immersive training environment, while in a group setting, was highlighted as particularly valuable and supported the training objectives. Training benefits were observed in both physiological and mindfulness outcomes for specific training modules ( Figure 4 ). Consistent positive feedback and self-report responses from both the training staff and trainees indicated increased knowledge, skill competency, and intention to use certain skills in the future (controlled breathing, awareness, and emotional acceptance; Figure 5 ).

PE Is a Feasible VR Training Solution for Group-Based Training

VR technology is an emerging field of research in military contexts and is predicted to improve training effectiveness in multiple domains, including combat command and decision making [ 35 , 46 , 47 ]. The use of VR as a training modality is relatively new, and very few training organizations have sustainably adopted, scaled, or integrated technology within their training continuum. This is particularly true for VR-based cognitive and psychological stress management interventions as their integration into the workforce has been challenging [ 14 , 48 , 49 ]. The relative novelty of VR-based stress management training has resulted in research focusing largely on the efficacy of applications under controlled experimental conditions [ 50 ]. However, these types of controlled studies cannot address questions about feasibility and implementation in real-world contexts, and there are limited reports on implementation challenges in the literature [ 51 ]. Factors that may impair the uptake of an otherwise effective digital tool include technology acceptance, practical usability, proficiency in use, and the existing structures required to support the technology [ 52 ]. This was due to the paucity of information on implementation issues, which we placed a particular emphasis on in this study.

After our original study evaluating PE in a controlled research trial, we found that the initial set-up and hardware was impractical, complex, and unstable outside of a research setting, particularly the research grade biometric data collection system. The platform underwent significant redesign to address challenges related to practical implementation within a group-based classroom setting [ 39 ]. We demonstrated the effective delivery of training to groups within their existing unit size, using a trainer to trainee ratio of 1:20. Seamless delivery was made possible using a freestanding headset solution and an enterprise hardware version, allowing remote fleet management, software upload, and updates. Familiarization with the technology occurred quickly despite limited previous trainee experience with VR technology. Notably, no occurrence of motion sickness, cybersickness or dizziness, a common concern among first-time VR users, has been reported [ 53 , 54 ]. Taken together, the results from this trial suggest that PE represents a VR training solution that is suitable for group-delivered training in the workplace context.

Engagement and User Acceptance of a Novel Training Solution

PE represents a novel training approach, not because of the intrinsic strategies included within the platform but rather by delivering practical training in a diverse, engaging, and immersive training environment that is private despite delivery in a group setting.

An important feature of PE is the emphasis placed on the practical development of stress management skills [ 55 ]. Consistent with BattleSMART, PE adopts the perspective that stress-management skills are central to optimal human performance. Throughout the modules, language with an overt mental health tone has been avoided. Instead, the perspective is taken that stress management skills should be considered like any other skill that contributes to healthy rounded performance in the workplace. Throughout the modules, the trainee is encouraged to consider stress management skills as general life skills that can be usefully applied not just to major stressors but also in response to small or mundane day-to-day challenges. The effort to recast validated skills within the platform was broadly received by the training audience.

In addition to the practical challenges of implementation, critical predictors for the future use and acceptance of new technologies in education are perceived usability and usefulness [ 44 ]. User engagement is particularly relevant for the transition of digital mental health interventions into real-world practice is user engagement [ 56 , 57 ]. However, transitioning from an expert-led to a digital-training format is often associated with low engagement, shallow learning, and potential frustration [ 51 , 58 ]. Given the technological and framework novelty of PE as a training solution for stress management soft skills, critical elements related to technology acceptance were investigated in this case study.

User engagement was validated for the target training population, specifically for military staff undergoing initial employment training. Feedback from both trainees and trainers suggests a general level of enjoyment and high levels of engagement for all modules ( Figures 3 A and 6A). Engagement is further supported by self-report ratings on the UES-SF and its subscales (engagement, esthetic appeal, focused attention, and perceived usability) as well as data collection and response times within the headset, which indicate that users participated in the activities, including personal reflections, as intended. Although the UES-SF is not intended to compare scores across applications or empirically classify high and low ratings, positive ratings and general training participation provide evidence of end-user acceptance of the training platform [ 40 ].

Immersion and sense of presence within PE are important to validate, as 360° video and computer-generated interface activities were intentionally designed to generate relaxing (beach and forest scenes), interesting (at an aquarium, in a sports gym), distracting (at a rock concert), or confrontational (angry men) training environments. The terms “immersion” and “presence” are often used interchangeably. However, in the scientific literature, presence refers specifically to the subjective psychological response of being within the environment, whereas immersion describes objective inputs within the digital environment (eg, interactions with the surroundings and selection items) [ 59 , 60 ]. The cognitive skills and reflections practiced within PE benefit from a sense of presence, as research suggests that presence can prompt emotional responses and interactions with digital avatars and environments despite users being fully aware of the fictitious nature of the setting [ 61 - 63 ]. A sense of presence has also been linked to improved training efficacy in digital-training applications [ 64 , 65 ]. All modules of PE generated a sense of presence (average rating of 5 on a 7-point scale; 7=high, 0=low), and the training environments triggered emotional states as intended ( Figure 3 ). In support of VR technology, immersion and privacy were rated as valuable and specifically mentioned as useful elements of the platform ( Figure 3 ). An important element connected to trainee engagement with the subject matter (specifically, emotional, thought awareness, and memory recall activities) was the affordance of privacy in a group setting provided by the VR headset. Existing research into the use of VR in education and training has shown the benefits of immersive VR over 2D screen delivery in areas of increased relaxation and arousal, motivation, engagement, and interest [ 34 , 66 - 68 ]. Although future research is required to validate the outcomes for PE platform in direct comparison to a screen-delivered training tool, it seems unlikely that trainees would feel an equivalent sense of presence, privacy, and engagement in a room and training setting of 20 trainees.

Training Outcomes

This study demonstrates the perceived usefulness of PE training, directly and indirectly, through self-report of improved skill competency and enhanced knowledge for each module ( Figure 5 ). Both training staff and trainees reported that PE provided useful and effective practical training and represented a valuable addition to the existing program, particularly via the provision of biofeedback, as well as the use of immersive and interactive training components. In pretraining surveys, ADF members indicated a relatively high level of understanding of the benefits and familiarity with concepts related to stress management skills. These findings are consistent with reports of military personnel who respond positively to the use of stress management techniques [ 69 ]. This background knowledge is likely to be, at least in part, due to previous exposure to the ADF BattleSMART program, which provides comprehensive education on optimal emotional and behavioral outcomes, resiliency, and arousal reduction skills [ 31 ]. However, despite this existing awareness, ADF staff reported minimal application of stress management skills outside of the training. Trainees had heard of grounding and progressive muscle relaxation but had very little practical experience in using them.

Modules 2 and 3 (controlled breathing and progressive muscle relaxation) aimed to develop skills that reduce the physiological effects of stress. The training outcomes included improved relaxation states and reduced respiratory rates ( Figures 4 A and 4B). Trainees were able to effectively reduce their respiration during the controlled breathing module, which is comparable with the results of our previous pilot trial [ 39 ]. Gaining conscious control, reducing breathing rates, and increasing relaxation are elements associated with effective stress management and reduced stress [ 70 , 71 ].

Modules 4 and 5 (grounded and emotional acceptance) provide training on cognitive skills; thus, skill competency and training outcomes for these modules were difficult to objectively quantify and compare with existing training approaches. There is growing literature on the relationship between mindfulness, psychological health, and stress reduction [ 72 ]. As elements of grounding overlap with mindfulness, the State Mindfulness Scale was administered, and an increase was observed following module 4 training. Although no differences between pre- and posttraining mindfulness states were observed for module 5, this may be due to the specific items used within the State Mindfulness Scale. Items within the body subdomain may not be relevant to training in emotion identification and acceptance. Although increased state mindfulness suggests a beneficial impact on stress, further research would be useful in assessing the efficacy of these training modules (M4 and M5).

Overall, the data suggest a positive immediate impact of PE training across the four stress management skill areas. Further studies will be useful to assess whether these short-term effects translate effectively to skills consolidation, the application of skills to real-world contexts, and the long-term effects of behavior changes and stress outcomes.

Study Limitations

The current trial was intended to assess the usability, feasibility, and suitability of PE training within its target training population in a real-world context. It should be noted that this is a case study within the ADF, and thus the findings may not be generalizable to other training settings, user populations, or training tools. Given the limited amount of investigation into the effective implementation of VR technology in the workplace, the results of this study provide valuable information on how to effectively integrate VR training into the workplace. Although specific to the ADF, we would view this as a useful starting point for any large organization interested in using VR within their training continuum. This work is intended to be the first step, with future studies required to document training efficacy. In particular, future research should investigate the number of iterations of VR exposure to develop skill mastery, the rate of skill degradation, how effectively these skills can be incorporated into real-world performance, and any effects on long-term mental health outcomes and workplace performance. Software and hardware were updated iteratively in response to the identified issues across the project and differed slightly between trial locations. To mitigate self-report bias, a conscious effort was made to brief participants that their input and responses were completely anonymous and that their input was being sought to improve the development of the application. Unfortunately, not all of the modules could be tested at all trial locations owing to staff availability and last-minute changes resulting from COVID-19 restrictions. As a result, trainee response numbers varied across modules (all numbers are reported). Despite this limitation, the multilocation trial approach resulted in participant numbers that far exceeded numbers generally seen within the VR literature, provided consistent findings for ADF service members across varying branches and locations, and provided balanced feedback for this level of evaluation.

Conclusions

This study found that the PE platform was feasible, implementable, and acceptable for stress management skills training within the ADF. Although many other studies have only assessed training solutions in controlled study environments, our work shows that virtual-reality and biofeedback technology can support training in real-world workplace settings. The ability of the PE platform to generate a private and immersive environment within a group setting provides a valuable proposition for the use of VR for this type of cognitive training. Engagement with the training platform is likely connected to the use of a targeted training framework as well as an approach and philosophy that is aligned with the overarching organizational values and practical requirements.

Acknowledgments

This study was funded by the Australian Department of Defense through the Defense Innovation Hub open-submission process. We would like to acknowledge all members of the Australian Defense Force who contributed to and participated in the studies, university staff members, and software developers at JumpGate who assisted with study support and implementation.

Data Availability

The datasets generated and analyzed in this study are available from the corresponding author upon reasonable request.

Conflicts of Interest

None declared.

Additional information and self-report items.

Promotional video for the Performance Edge software recorded within the headset (it is advised that the recording may result in motion sickness).

Module 1 walk-through start. This video was recorded as a screen capture within the virtual reality headset and shows the beginning of module 1 (thoughts, emotions, and behaviors). It is advised that the recording may result in motion sickness.

• Waller M, Treloar SA, Sim MR, McFarlane AC, McGuire AC, Bleier J, et al. Traumatic events, other operational stressors and physical and mental health reported by Australian Defence Force personnel following peacekeeping and war-like deployments. BMC Psychiatry 2012 Jul 26;12:88 [ https://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-12-88 ] [ CrossRef ] [ Medline ]
• Garbarino S, Cuomo G, Chiorri C, Magnavita N. Association of work-related stress with mental health problems in a special police force unit. BMJ Open 2013 Jul 19;3(7):e002791 [ https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=23872288 ] [ CrossRef ] [ Medline ]
• Hemi A, Sopp MR, Schäfer SK, Michael T, Levy-Gigi E. Adaptive responding to prolonged stress exposure: a binational study on the impact of flexibility on latent profiles of cognitive, emotional and behavioural responses to the COVID-19 pandemic. Health Soc Care Community 2022 Nov;30(6):e6163-e6174 [ https://europepmc.org/abstract/MED/36184793 ] [ CrossRef ] [ Medline ]
• Nisar SK, Rasheed MI. Stress and performance: investigating relationship between occupational stress, career satisfaction, and job performance of police employees. J Public Aff 2019 Jun 24;20(1):e1986 [ CrossRef ]
• Motowidlo SJ, Packard JS, Manning MR. Occupational stress: its causes and consequences for job performance. J Appl Psychol 1986;71(4):618-629 [ CrossRef ]
• Jones S. Describing the mental health profile of first responders: a systematic review [formula: see text]. J Am Psychiatr Nurses Assoc 2017 May;23(3):200-214 [ CrossRef ] [ Medline ]
• Lehrer PM, Woolfolk RL. Principles and Practice of Stress Management. New York, NY: Guilford Publications; 2021.
• Bouchard S, Baus O, Bernier F, McCreary DR. Selection of key stressors to develop virtual environments for practicing stress management skills with military personnel prior to deployment. Cyberpsychol Behav Soc Netw 2010 Feb;13(1):83-94 [ CrossRef ] [ Medline ]
• Rizzo A, Shilling R. Clinical virtual reality tools to advance the prevention, assessment, and treatment of PTSD. Eur J Psychotraumatol 2017 Jan 16;8(sup5):1414560 [ https://europepmc.org/abstract/MED/29372007 ] [ CrossRef ] [ Medline ]
• Worthen M, Cash E. Stress management. In: StatPearls. Treasure Island, FL: StatPearls Publishing; Aug 14, 2023.
• Murphy LR. Stress management in work settings: a critical review of the health effects. Am J Health Promot 1996;11(2):112-135 [ CrossRef ] [ Medline ]
• Tabibnia G, Radecki D. Resilience training that can change the brain. Consult Psychol J Pract Res 2018;70(1):59-88 [ CrossRef ]
• Jha AP, Morrison AB, Parker SC, Stanley EA. Practice is protective: mindfulness training promotes cognitive resilience in high-stress cohorts. Mindfulness 2016 Jan 21;8:46-58 [ CrossRef ]
• Precious D, Lindsay A. Mental resilience training. J R Army Med Corps 2019 Apr;165(2):106-108 [ CrossRef ] [ Medline ]
• Mattingly V, Kraiger K. Can emotional intelligence be trained? A meta-analytical investigation. Hum Resour Manag Rev 2019 Jun;29(2):140-155 [ CrossRef ]
• Khoury B, Sharma M, Rush SE, Fournier C. Mindfulness-based stress reduction for healthy individuals: a meta-analysis. J Psychosom Res 2015 Jun;78(6):519-528 [ CrossRef ] [ Medline ]
• Ma X, Yue ZQ, Gong ZQ, Zhang H, Duan NY, Shi YT, et al. The effect of diaphragmatic breathing on attention, negative affect and stress in healthy adults. Front Psychol 2017 Jun 06;8:874 [ https://europepmc.org/abstract/MED/28626434 ] [ CrossRef ] [ Medline ]
• Crawford C, Wallerstedt DB, Khorsan R, Clausen SS, Jonas WB, Walter JAG. A systematic review of biopsychosocial training programs for the self-management of emotional stress: potential applications for the military. Evid Based Complement Alternat Med 2013;2013:747694 [ https://doi.org/10.1155/2013/747694 ] [ CrossRef ] [ Medline ]
• Birrer D, Morgan G. Psychological skills training as a way to enhance an athlete's performance in high-intensity sports. Scand J Med Sci Sports 2010 Oct 14;20 Suppl 2:78-87 [ CrossRef ] [ Medline ]
• Zaccaro A, Piarulli A, Laurino M, Garbella E, Menicucci D, Neri B, et al. How breath-control can change your life: a systematic review on psycho-physiological correlates of slow breathing. Front Hum Neurosci 2018 Sep 7;12:353 [ https://europepmc.org/abstract/MED/30245619 ] [ CrossRef ] [ Medline ]
• Holman D, Johnson S, O'Connor E. Stress management interventions: improving subjective psychological well-being in the workplace. In: Diener E, Oishi S, Tay L, editors. Handbook of Well-Being. Salt Lake City, UT: DEF Publishers; 2018.
• Adler AB, Williams J, McGurk D, Moss A, Bliese PD. Resilience training with soldiers during basic combat training: randomisation by platoon. Appl Psychol Health Well Being 2015 Mar;7(1):85-107 [ CrossRef ] [ Medline ]
• Cornum R, Matthews MD, Seligman ME. Comprehensive soldier fitness: building resilience in a challenging institutional context. Am Psychol 2011 Jan;66(1):4-9 [ CrossRef ] [ Medline ]
• Guo D, Sun L, Yu X, Liu T, Wu L, Sun Z, et al. Mindfulness-based stress reduction improves the general health and stress of Chinese military recruits: a pilot study. Psychiatry Res 2019 Nov;281:112571 [ CrossRef ] [ Medline ]
• Taylor MK, Rolo C, Stump J, Mayo J, Hernandez LM, Gould DR. Psychological strategies during military training are linked to resilience in US navy explosive ordnance disposal operators. J Spec Oper Med 2019;19(1):61-65 [ CrossRef ] [ Medline ]
• Obuobi-Donkor G, Eboreime E, Bond J, Phung N, Eyben S, Hayward J, et al. An e-mental health solution to prevent and manage posttraumatic stress injuries among first responders in Alberta: protocol for the implementation and evaluation of text messaging services (Text4PTSI and Text4Wellbeing). JMIR Res Protoc 2022 Apr 25;11(4):e30680 [ https://www.researchprotocols.org/2022/4/e30680/ ] [ CrossRef ] [ Medline ]
• Aidman E, Fogarty GJ, Crampton J, Bond J, Taylor P, Heathcote A, et al. An app-enhanced cognitive fitness training program for athletes: the rationale and validation protocol. Front Psychol 2022 Aug 30;13:957551 [ https://europepmc.org/abstract/MED/36110271 ] [ CrossRef ] [ Medline ]
• Aidman E. Cognitive fitness framework: towards assessing, training and augmenting individual-difference factors underpinning high-performance cognition. Front Hum Neurosci 2019 Jan 14;13:466 [ https://europepmc.org/abstract/MED/32009919 ] [ CrossRef ] [ Medline ]
• Albertella L, Kirkham R, Adler AB, Crampton J, Drummond SPA, Fogarty GJ, et al. Building a transdisciplinary expert consensus on the cognitive drivers of performance under pressure: An international multi-panel Delphi study. Front Psychol 2022;13:1017675 [ https://europepmc.org/abstract/MED/36755983 ] [ CrossRef ] [ Medline ]
• Fogarty GJ, Crampton J, Bond J, Zaichkowsky LD, Taylor P, Aidman E. Neurobiological foundations of cognitive fitness in high-performance applications. In: Boyle GJ, Northoff G, Barbey AK, Fregni F, Jahanshahi M, Pascual-Leone A, et al, editors. The Sage Handbook of Cognitive and Systems Neuroscience Cognitive Systems, Development and Applications. Thousand Oaks, CA: SAGE Publications; 2023.
• Cohn A, Hodson S, Crane M. Resilience training in the Australian Defence Force. InPsych 2010;32(2)
• Garzón J, Acevedo J, Pavón J, Baldiris S. Promoting eco-agritourism using an augmented reality-based educational resource: a case study of aquaponics. Interact Learn Environ 2020 Jan 13;30(7):1200-1214 [ CrossRef ]
• Mikropoulos TA, Chalkidis A, Katsikis A, Emvalotis A. Students' attitudes towards educational virtual environments. Educ Inf Technol 1998;3:137-148 [ https://link.springer.com/article/10.1023/A:1009687025419#citeas ] [ CrossRef ]
• Jensen L, Konradsen F. A review of the use of virtual reality head-mounted displays in education and training. Educ Inf Technol 2017 Nov 25;23:1515-1529 [ CrossRef ]
• Pallavicini F, Argenton L, Toniazzi N, Aceti L, Mantovani F. Virtual reality applications for stress management training in the military. Aerosp Med Hum Perform 2016 Dec 01;87(12):1021-1030 [ CrossRef ] [ Medline ]
• Prinsloo GE, Derman WE, Lambert MI, Laurie Rauch HG. The effect of a single session of short duration biofeedback-induced deep breathing on measures of heart rate variability during laboratory-induced cognitive stress: a pilot study. Appl Psychophysiol Biofeedback 2013 Jun;38(2):81-90 [ CrossRef ] [ Medline ]
• McCraty R, Atkinson M. Resilience training program reduces physiological and psychological stress in police officers. Glob Adv Health Med 2012 Nov;1(5):44-66 [ https://europepmc.org/abstract/MED/27257532 ] [ CrossRef ] [ Medline ]
• Hourani L, Tueller S, Kizakevich P, Lewis G, Strange L, Weimer B, et al. Toward preventing post-traumatic stress disorder: development and testing of a pilot predeployment stress inoculation training program. Mil Med 2016 Sep;181(9):1151-1160 [ CrossRef ] [ Medline ]
• Kluge MG, Maltby S, Walker N, Bennett N, Aidman E, Nalivaiko E, et al. Development of a modular stress management platform (Performance Edge VR) and a pilot efficacy trial of a bio-feedback enhanced training module for controlled breathing. PLoS One 2021 Feb 2;16(2):e0245068 [ https://dx.plos.org/10.1371/journal.pone.0245068 ] [ CrossRef ] [ Medline ]
• O’Brien HL, Cairns P, Hall M. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. Int J Hum Comput Stud 2018 Apr;112:28-39 [ CrossRef ]
• Witmer BG, Singer MJ. Measuring presence in virtual environments: a presence questionnaire. Presence Teleoperators Virtual Environ 1998;7(3):225-240 [ https://direct.mit.edu/pvar/article-abstract/7/3/225/92643/Measuring-Presence-in-Virtual-Environments-A?redirectedFrom=fulltext ]
• Tanay G, Bernstein A. State Mindfulness Scale (SMS): development and initial validation. Psychol Assess 2013 Dec;25(4):1286-1299 [ CrossRef ] [ Medline ]
• Crist DA, Rickard HC, Prentice-Dunn S, Barker HR. The Relaxation Inventory: self-report scales of relaxation training effects. J Pers Assess 1989;53(4):716-726 [ CrossRef ] [ Medline ]
• Granić A, Marangunić N. Technology acceptance model in educational context: a systematic literature review. Br J Educ Technol 2019 Jul 09;50(5):2572-2593 [ CrossRef ]
• Marangunić N, Granić A. Technology acceptance model: a literature review from 1986 to 2013. Univ Access Inf Soc 2014 Feb 16;14:81-95 [ CrossRef ]
• Ahir K, Govani K, Gajera R, Shah M. Application on virtual reality for enhanced education learning, military training and sports. Augment Hum Res 2019 Nov 29;5:7 [ CrossRef ]
• Liu X, Zhang J, Hou G, Wang Z. Virtual reality and its application in military. IOP Conf Ser Earth Environ Sci 2018 Jul 17;170:032155 [ CrossRef ]
• Thompson SR, Dobbins S. The applicability of resilience training to the mitigation of trauma-related mental illness in military personnel. J Am Psychiatr Nurses Assoc 2018 Nov 15;24(1):23-34 [ CrossRef ] [ Medline ]
• Hourani LL, Council CL, Hubal RC, Strange LB. Approaches to the primary prevention of posttraumatic stress disorder in the military: a review of the stress control literature. Mil Med 2011 Jul;176(7):721-730 [ CrossRef ] [ Medline ]
• Gaggioli A, Pallavicini F, Morganti L, Serino S, Scaratti C, Briguglio M, et al. Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: a block randomized controlled trial. J Med Internet Res 2014 Jul 08;16(7):e167 [ https://www.jmir.org/2014/7/e167/ ] [ CrossRef ] [ Medline ]
• Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res 2018 Jun 06;20(6):e199 [ https://www.jmir.org/2018/6/e199/ ] [ CrossRef ] [ Medline ]
• Butler DL, Sellbom M. Barriers to adopting technology for teaching and learning. Educause Q 2002;25(2):22-28 [ https://www.learntechlib.org/p/92849/ ]
• Howard MC, Van Zandt EC. A meta-analysis of the virtual reality problem: unequal effects of virtual reality sickness across individual differences. Virtual Real 2021 May 11;25(4):1221-1246 [ CrossRef ]
• Rebenitsch L, Owen C. Review on cybersickness in applications and visual displays. Virtual Real 2016 Apr 26;20(2):101-125 [ CrossRef ]
• Varvogli L, Darviri C. Stress management techniques: evidence-based procedures that reduce stress and promote health. Health Sci J 2011 Apr;5(2):74-89 [ https://www.researchgate.net/publication/236685932_Stress_Management_Techniques_evidence-based_procedures_that_reduce_stress_and_promote_health_Varvogli_L_Darviri_C ]
• Baumel A, Edan S, Kane JM. Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs. Transl Behav Med 2019 Nov 25;9(6):1020-1033 [ CrossRef ] [ Medline ]
• Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res 2021 Mar 24;23(3):e24387 [ https://www.jmir.org/2021/3/e24387/ ] [ CrossRef ] [ Medline ]
• Mrazek AJ, Mrazek MD, Cherolini CM, Cloughesy JN, Cynman DJ, Gougis LJ, et al. The future of mindfulness training is digital, and the future is now. Curr Opin Psychol 2019 Aug;28:81-86 [ CrossRef ] [ Medline ]
• Bowman DA, McMahan RP. Virtual reality: how much immersion is enough? Computer 2007 Jul;40(7):36-43 [ CrossRef ]
• Harris DJ, Bird JM, Smart PA, Wilson MR, Vine SJ. A framework for the testing and validation of simulated environments in experimentation and training. Front Psychol 2020 Mar 31;11:605 [ https://europepmc.org/abstract/MED/32296379 ] [ CrossRef ] [ Medline ]
• Sanz FA, Olivier AH, Bruder G, Pettré J, Lécuyer A. Virtual proxemics: locomotion in the presence of obstacles in large immersive projection environments. In: Proceedings of the 2015 IEEE Virtual Reality (VR). 2015 Presented at: 2015 IEEE Virtual Reality (VR); March 23-27, 2015; Arles, France [ CrossRef ]
• Meehan M, Razzaque S, Insko B, Whitton M, Brooks FPJ. Review of four studies on the use of physiological reaction as a measure of presence in stressful virtual environments. Appl Psychophysiol Biofeedback 2005 Sep;30(3):239-258 [ CrossRef ] [ Medline ]
• Slater M, Antley A, Davison A, Swapp D, Guger C, Barker C, et al. A virtual reprise of the Stanley Milgram obedience experiments. PLoS One 2006 Dec 20;1(1):e39 [ https://dx.plos.org/10.1371/journal.pone.0000039 ] [ CrossRef ] [ Medline ]
• Grassini S, Laumann K, Rasmussen Skogstad M. The use of virtual reality alone does not promote training performance (but sense of presence does). Front Psychol 2020 Jul 17;11:1743 [ https://europepmc.org/abstract/MED/32765384 ] [ CrossRef ] [ Medline ]
• Lerner D, Mohr S, Schild J, Göring M, Luiz T. An immersive multi-user virtual reality for emergency simulation training: usability study. JMIR Serious Games 2020 Jul 31;8(3):e18822 [ https://games.jmir.org/2020/3/e18822/ ] [ CrossRef ] [ Medline ]
• Cao L, Peng C, Hansberger JT. Usability and engagement study for a serious virtual reality game of lunar exploration missions. Informatics 2019 Oct 03;6(4):44 [ CrossRef ]
• Riches S, Azevedo L, Bird L, Pisani S, Valmaggia L. Virtual reality relaxation for the general population: a systematic review. Soc Psychiatry Psychiatr Epidemiol 2021 Oct 13;56(10):1707-1727 [ https://europepmc.org/abstract/MED/34120220 ] [ CrossRef ] [ Medline ]
• Andersen JP, Gustafsberg H. A training method to improve police use of force decision making: a randomized controlled trial. SAGE Open 2016 Apr 07;6(2) [ CrossRef ]
• Stetz MC, Bouchard S, Wiederhold BK, Riva G, Folen RA. The receptiveness of stress management techniques by military personnel. Stud Health Technol Inform 2009;144:125-127 [ Medline ]
• Kaspereen D. Relaxation intervention for stress reduction among teachers and staff. Int J Stress Manag 2012 Aug;19(3):238-250 [ CrossRef ]
• Bouchard S, Bernier F, Boivin E, Morin B, Robillard G. Using biofeedback while immersed in a stressful videogame increases the effectiveness of stress management skills in soldiers. PLoS One 2012 Apr 27;7(4):e36169 [ https://dx.plos.org/10.1371/journal.pone.0036169 ] [ CrossRef ] [ Medline ]
• Grossman P, Niemann L, Schmidt S, Walach H. Mindfulness-based stress reduction and health benefits: a meta-analysis. J Psychosomatic Res 2004 Jul;57(1):35-43 [ CrossRef ]

Abbreviations

Edited by A Mavragani; submitted 08.02.23; peer-reviewed by T Ong, T Cahill, S Cargill-Fealy; comments to author 31.07.23; revised version received 24.08.23; accepted 18.09.23; published 06.11.23

©Murielle G Kluge, Steven Maltby, Caroline Kuhne, Nicole Walker, Neanne Bennett, Eugene Aidman, Eugene Nalivaiko, Frederick Rohan Walker. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 06.11.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Advanced Search
• Journal List
• Front Psychol

The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature

Pietro cipresso.

1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy

2 Department of Psychology, Catholic University of the Sacred Heart, Milan, Italy

Irene Alice Chicchi Giglioli

3 Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, Valencia, Spain

Mariano Alcañiz Raya

Giuseppe riva, associated data.

The recent appearance of low cost virtual reality (VR) technologies – like the Oculus Rift, the HTC Vive and the Sony PlayStation VR – and Mixed Reality Interfaces (MRITF) – like the Hololens – is attracting the attention of users and researchers suggesting it may be the next largest stepping stone in technological innovation. However, the history of VR technology is longer than it may seem: the concept of VR was formulated in the 1960s and the first commercial VR tools appeared in the late 1980s. For this reason, during the last 20 years, 100s of researchers explored the processes, effects, and applications of this technology producing 1000s of scientific papers. What is the outcome of this significant research work? This paper wants to provide an answer to this question by exploring, using advanced scientometric techniques, the existing research corpus in the field. We collected all the existent articles about VR in the Web of Science Core Collection scientific database, and the resultant dataset contained 21,667 records for VR and 9,944 for augmented reality (AR). The bibliographic record contained various fields, such as author, title, abstract, country, and all the references (needed for the citation analysis). The network and cluster analysis of the literature showed a composite panorama characterized by changes and evolutions over the time. Indeed, whether until 5 years ago, the main publication media on VR concerned both conference proceeding and journals, more recently journals constitute the main medium of communication. Similarly, if at first computer science was the leading research field, nowadays clinical areas have increased, as well as the number of countries involved in VR research. The present work discusses the evolution and changes over the time of the use of VR in the main areas of application with an emphasis on the future expected VR’s capacities, increases and challenges. We conclude considering the disruptive contribution that VR/AR/MRITF will be able to get in scientific fields, as well in human communication and interaction, as already happened with the advent of mobile phones by increasing the use and the development of scientific applications (e.g., in clinical areas) and by modifying the social communication and interaction among people.

Introduction

In the last 5 years, virtual reality (VR) and augmented reality (AR) have attracted the interest of investors and the general public, especially after Mark Zuckerberg bought Oculus for two billion dollars ( Luckerson, 2014 ; Castelvecchi, 2016 ). Currently, many other companies, such as Sony, Samsung, HTC, and Google are making huge investments in VR and AR ( Korolov, 2014 ; Ebert, 2015 ; Castelvecchi, 2016 ). However, if VR has been used in research for more than 25 years, and now there are 1000s of papers and many researchers in the field, comprising a strong, interdisciplinary community, AR has a more recent application history ( Burdea and Coiffet, 2003 ; Kim, 2005 ; Bohil et al., 2011 ; Cipresso and Serino, 2014 ; Wexelblat, 2014 ). The study of VR was initiated in the computer graphics field and has been extended to several disciplines ( Sutherland, 1965 , 1968 ; Mazuryk and Gervautz, 1996 ; Choi et al., 2015 ). Currently, videogames supported by VR tools are more popular than the past, and they represent valuables, work-related tools for neuroscientists, psychologists, biologists, and other researchers as well. Indeed, for example, one of the main research purposes lies from navigation studies that include complex experiments that could be done in a laboratory by using VR, whereas, without VR, the researchers would have to go directly into the field, possibly with limited use of intervention. The importance of navigation studies for the functional understanding of human memory in dementia has been a topic of significant interest for a long time, and, in 2014, the Nobel Prize in “Physiology or Medicine” was awarded to John M. O’Keefe, May-Britt Moser, and Edvard I. Moser for their discoveries of nerve cells in the brain that enable a sense of place and navigation. Journals and magazines have extended this knowledge by writing about “the brain GPS,” which gives a clear idea of the mechanism. A huge number of studies have been conducted in clinical settings by using VR ( Bohil et al., 2011 ; Serino et al., 2014 ), and Nobel Prize winner, Edvard I. Moser commented about the use of VR ( Minderer et al., 2016 ), highlighting its importance for research and clinical practice. Moreover, the availability of free tools for VR experimental and computational use has made it easy to access any field ( Riva et al., 2011 ; Cipresso, 2015 ; Brown and Green, 2016 ; Cipresso et al., 2016 ).

Augmented reality is a more recent technology than VR and shows an interdisciplinary application framework, in which, nowadays, education and learning seem to be the most field of research. Indeed, AR allows supporting learning, for example increasing-on content understanding and memory preservation, as well as on learning motivation. However, if VR benefits from clear and more definite fields of application and research areas, AR is still emerging in the scientific scenarios.

In this article, we present a systematic and computational analysis of the emerging interdisciplinary VR and AR fields in terms of various co-citation networks in order to explore the evolution of the intellectual structure of this knowledge domain over time.

Virtual Reality Concepts and Features

The concept of VR could be traced at the mid of 1960 when Ivan Sutherland in a pivotal manuscript attempted to describe VR as a window through which a user perceives the virtual world as if looked, felt, sounded real and in which the user could act realistically ( Sutherland, 1965 ).

Since that time and in accordance with the application area, several definitions have been formulated: for example, Fuchs and Bishop (1992) defined VR as “real-time interactive graphics with 3D models, combined with a display technology that gives the user the immersion in the model world and direct manipulation” ( Fuchs and Bishop, 1992 ); Gigante (1993) described VR as “The illusion of participation in a synthetic environment rather than external observation of such an environment. VR relies on a 3D, stereoscopic head-tracker displays, hand/body tracking and binaural sound. VR is an immersive, multi-sensory experience” ( Gigante, 1993 ); and “Virtual reality refers to immersive, interactive, multi-sensory, viewer-centered, 3D computer generated environments and the combination of technologies required building environments” ( Cruz-Neira, 1993 ).

As we can notice, these definitions, although different, highlight three common features of VR systems: immersion, perception to be present in an environment, and interaction with that environment ( Biocca, 1997 ; Lombard and Ditton, 1997 ; Loomis et al., 1999 ; Heeter, 2000 ; Biocca et al., 2001 ; Bailenson et al., 2006 ; Skalski and Tamborini, 2007 ; Andersen and Thorpe, 2009 ; Slater, 2009 ; Sundar et al., 2010 ). Specifically, immersion concerns the amount of senses stimulated, interactions, and the reality’s similarity of the stimuli used to simulate environments. This feature can depend on the properties of the technological system used to isolate user from reality ( Slater, 2009 ).

Higher or lower degrees of immersion can depend by three types of VR systems provided to the user:

• simple • Non-immersive systems are the simplest and cheapest type of VR applications that use desktops to reproduce images of the world.
• simple • Immersive systems provide a complete simulated experience due to the support of several sensory outputs devices such as head mounted displays (HMDs) for enhancing the stereoscopic view of the environment through the movement of the user’s head, as well as audio and haptic devices.
• simple • Semi-immersive systems such as Fish Tank VR are between the two above. They provide a stereo image of a three dimensional (3D) scene viewed on a monitor using a perspective projection coupled to the head position of the observer ( Ware et al., 1993 ). Higher technological immersive systems have showed a closest experience to reality, giving to the user the illusion of technological non-mediation and feeling him or her of “being in” or present in the virtual environment ( Lombard and Ditton, 1997 ). Furthermore, higher immersive systems, than the other two systems, can give the possibility to add several sensory outputs allowing that the interaction and actions were perceived as real ( Loomis et al., 1999 ; Heeter, 2000 ; Biocca et al., 2001 ).

Finally, the user’s VR experience could be disclosed by measuring presence, realism, and reality’s levels. Presence is a complex psychological feeling of “being there” in VR that involves the sensation and perception of physical presence, as well as the possibility to interact and react as if the user was in the real world ( Heeter, 1992 ). Similarly, the realism’s level corresponds to the degree of expectation that the user has about of the stimuli and experience ( Baños et al., 2000 , 2009 ). If the presented stimuli are similar to reality, VR user’s expectation will be congruent with reality expectation, enhancing VR experience. In the same way, higher is the degree of reality in interaction with the virtual stimuli, higher would be the level of realism of the user’s behaviors ( Baños et al., 2000 , 2009 ).

From Virtual to Augmented Reality

Looking chronologically on VR and AR developments, we can trace the first 3D immersive simulator in 1962, when Morton Heilig created Sensorama, a simulated experience of a motorcycle running through Brooklyn characterized by several sensory impressions, such as audio, olfactory, and haptic stimuli, including also wind to provide a realist experience ( Heilig, 1962 ). In the same years, Ivan Sutherland developed The Ultimate Display that, more than sound, smell, and haptic feedback, included interactive graphics that Sensorama didn’t provide. Furthermore, Philco developed the first HMD that together with The Sword of Damocles of Sutherland was able to update the virtual images by tracking user’s head position and orientation ( Sutherland, 1965 ). In the 70s, the University of North Carolina realized GROPE, the first system of force-feedback and Myron Krueger created VIDEOPLACE an Artificial Reality in which the users’ body figures were captured by cameras and projected on a screen ( Krueger et al., 1985 ). In this way two or more users could interact in the 2D-virtual space. In 1982, the US’ Air Force created the first flight simulator [Visually Coupled Airbone System Simulator (VCASS)] in which the pilot through an HMD could control the pathway and the targets. Generally, the 80’s were the years in which the first commercial devices began to emerge: for example, in 1985 the VPL company commercialized the DataGlove, glove sensors’ equipped able to measure the flexion of fingers, orientation and position, and identify hand gestures. Another example is the Eyephone, created in 1988 by the VPL Company, an HMD system for completely immerging the user in a virtual world. At the end of 80’s, Fake Space Labs created a Binocular-Omni-Orientational Monitor (BOOM), a complex system composed by a stereoscopic-displaying device, providing a moving and broad virtual environment, and a mechanical arm tracking. Furthermore, BOOM offered a more stable image and giving more quickly responses to movements than the HMD devices. Thanks to BOOM and DataGlove, the NASA Ames Research Center developed the Virtual Wind Tunnel in order to research and manipulate airflow in a virtual airplane or space ship. In 1992, the Electronic Visualization Laboratory of the University of Illinois created the CAVE Automatic Virtual Environment, an immersive VR system composed by projectors directed on three or more walls of a room.

More recently, many videogames companies have improved the development and quality of VR devices, like Oculus Rift, or HTC Vive that provide a wider field of view and lower latency. In addition, the actual HMD’s devices can be now combined with other tracker system as eye-tracking systems (FOVE), and motion and orientation sensors (e.g., Razer Hydra, Oculus Touch, or HTC Vive).

Simultaneously, at the beginning of 90’, the Boing Corporation created the first prototype of AR system for showing to employees how set up a wiring tool ( Carmigniani et al., 2011 ). At the same time, Rosenberg and Feiner developed an AR fixture for maintenance assistance, showing that the operator performance enhanced by added virtual information on the fixture to repair ( Rosenberg, 1993 ). In 1993 Loomis and colleagues produced an AR GPS-based system for helping the blind in the assisted navigation through adding spatial audio information ( Loomis et al., 1998 ). Always in the 1993 Julie Martin developed “Dancing in Cyberspace,” an AR theater in which actors interacted with virtual object in real time ( Cathy, 2011 ). Few years later, Feiner et al. (1997) developed the first Mobile AR System (MARS) able to add virtual information about touristic buildings ( Feiner et al., 1997 ). Since then, several applications have been developed: in Thomas et al. (2000) , created ARQuake, a mobile AR video game; in 2008 was created Wikitude that through the mobile camera, internet, and GPS could add information about the user’s environments ( Perry, 2008 ). In 2009 others AR applications, like AR Toolkit and SiteLens have been developed in order to add virtual information to the physical user’s surroundings. In 2011, Total Immersion developed D’Fusion, and AR system for designing projects ( Maurugeon, 2011 ). Finally, in 2013 and 2015, Google developed Google Glass and Google HoloLens, and their usability have begun to test in several field of application.

Virtual Reality Technologies

Technologically, the devices used in the virtual environments play an important role in the creation of successful virtual experiences. According to the literature, can be distinguished input and output devices ( Burdea et al., 1996 ; Burdea and Coiffet, 2003 ). Input devices are the ones that allow the user to communicate with the virtual environment, which can range from a simple joystick or keyboard to a glove allowing capturing finger movements or a tracker able to capture postures. More in detail, keyboard, mouse, trackball, and joystick represent the desktop input devices easy to use, which allow the user to launch continuous and discrete commands or movements to the environment. Other input devices can be represented by tracking devices as bend-sensing gloves that capture hand movements, postures and gestures, or pinch gloves that detect the fingers movements, and trackers able to follow the user’s movements in the physical world and translate them in the virtual environment.

On the contrary, the output devices allow the user to see, hear, smell, or touch everything that happens in the virtual environment. As mentioned above, among the visual devices can be found a wide range of possibilities, from the simplest or least immersive (monitor of a computer) to the most immersive one such as VR glasses or helmets or HMD or CAVE systems.

Furthermore, auditory, speakers, as well as haptic output devices are able to stimulate body senses providing a more real virtual experience. For example, haptic devices can stimulate the touch feeling and force models in the user.

Virtual Reality Applications

Since its appearance, VR has been used in different fields, as for gaming ( Zyda, 2005 ; Meldrum et al., 2012 ), military training ( Alexander et al., 2017 ), architectural design ( Song et al., 2017 ), education ( Englund et al., 2017 ), learning and social skills training ( Schmidt et al., 2017 ), simulations of surgical procedures ( Gallagher et al., 2005 ), assistance to the elderly or psychological treatments are other fields in which VR is bursting strongly ( Freeman et al., 2017 ; Neri et al., 2017 ). A recent and extensive review of Slater and Sanchez-Vives (2016) reported the main VR application evidences, including weakness and advantages, in several research areas, such as science, education, training, physical training, as well as social phenomena, moral behaviors, and could be used in other fields, like travel, meetings, collaboration, industry, news, and entertainment. Furthermore, another review published this year by Freeman et al. (2017) focused on VR in mental health, showing the efficacy of VR in assessing and treating different psychological disorders as anxiety, schizophrenia, depression, and eating disorders.

There are many possibilities that allow the use of VR as a stimulus, replacing real stimuli, recreating experiences, which in the real world would be impossible, with a high realism. This is why VR is widely used in research on new ways of applying psychological treatment or training, for example, to problems arising from phobias (agoraphobia, phobia to fly, etc.) ( Botella et al., 2017 ). Or, simply, it is used like improvement of the traditional systems of motor rehabilitation ( Llorens et al., 2014 ; Borrego et al., 2016 ), developing games that ameliorate the tasks. More in detail, in psychological treatment, Virtual Reality Exposure Therapy (VRET) has showed its efficacy, allowing to patients to gradually face fear stimuli or stressed situations in a safe environment where the psychological and physiological reactions can be controlled by the therapist ( Botella et al., 2017 ).

Augmented Reality Concept

Milgram and Kishino (1994) , conceptualized the Virtual-Reality Continuum that takes into consideration four systems: real environment, augmented reality (AR), augmented virtuality, and virtual environment. AR can be defined a newer technological system in which virtual objects are added to the real world in real-time during the user’s experience. Per Azuma et al. (2001) an AR system should: (1) combine real and virtual objects in a real environment; (2) run interactively and in real-time; (3) register real and virtual objects with each other. Furthermore, even if the AR experiences could seem different from VRs, the quality of AR experience could be considered similarly. Indeed, like in VR, feeling of presence, level of realism, and the degree of reality represent the main features that can be considered the indicators of the quality of AR experiences. Higher the experience is perceived as realistic, and there is congruence between the user’s expectation and the interaction inside the AR environments, higher would be the perception of “being there” physically, and at cognitive and emotional level. The feeling of presence, both in AR and VR environments, is important in acting behaviors like the real ones ( Botella et al., 2005 ; Juan et al., 2005 ; Bretón-López et al., 2010 ; Wrzesien et al., 2013 ).

Augmented Reality Technologies

Technologically, the AR systems, however various, present three common components, such as a geospatial datum for the virtual object, like a visual marker, a surface to project virtual elements to the user, and an adequate processing power for graphics, animation, and merging of images, like a pc and a monitor ( Carmigniani et al., 2011 ). To run, an AR system must also include a camera able to track the user movement for merging the virtual objects, and a visual display, like glasses through that the user can see the virtual objects overlaying to the physical world. To date, two-display systems exist, a video see-through (VST) and an optical see-though (OST) AR systems ( Botella et al., 2005 ; Juan et al., 2005 , 2007 ). The first one, disclosures virtual objects to the user by capturing the real objects/scenes with a camera and overlaying virtual objects, projecting them on a video or a monitor, while the second one, merges the virtual object on a transparent surface, like glasses, through the user see the added elements. The main difference between the two systems is the latency: an OST system could require more time to display the virtual objects than a VST system, generating a time lag between user’s action and performance and the detection of them by the system.

Augmented Reality Applications

Although AR is a more recent technology than VR, it has been investigated and used in several research areas such as architecture ( Lin and Hsu, 2017 ), maintenance ( Schwald and De Laval, 2003 ), entertainment ( Ozbek et al., 2004 ), education ( Nincarean et al., 2013 ; Bacca et al., 2014 ; Akçayır and Akçayır, 2017 ), medicine ( De Buck et al., 2005 ), and psychological treatments ( Juan et al., 2005 ; Botella et al., 2005 , 2010 ; Bretón-López et al., 2010 ; Wrzesien et al., 2011a , b , 2013 ; see the review Chicchi Giglioli et al., 2015 ). More in detail, in education several AR applications have been developed in the last few years showing the positive effects of this technology in supporting learning, such as an increased-on content understanding and memory preservation, as well as on learning motivation ( Radu, 2012 , 2014 ). For example, Ibáñez et al. (2014) developed a AR application on electromagnetism concepts’ learning, in which students could use AR batteries, magnets, cables on real superficies, and the system gave a real-time feedback to students about the correctness of the performance, improving in this way the academic success and motivation ( Di Serio et al., 2013 ). Deeply, AR system allows the possibility to learn visualizing and acting on composite phenomena that traditionally students study theoretically, without the possibility to see and test in real world ( Chien et al., 2010 ; Chen et al., 2011 ).

As well in psychological health, the number of research about AR is increasing, showing its efficacy above all in the treatment of psychological disorder (see the reviews Baus and Bouchard, 2014 ; Chicchi Giglioli et al., 2015 ). For example, in the treatment of anxiety disorders, like phobias, AR exposure therapy (ARET) showed its efficacy in one-session treatment, maintaining the positive impact in a follow-up at 1 or 3 month after. As VRET, ARET provides a safety and an ecological environment where any kind of stimulus is possible, allowing to keep control over the situation experienced by the patients, gradually generating situations of fear or stress. Indeed, in situations of fear, like the phobias for small animals, AR applications allow, in accordance with the patient’s anxiety, to gradually expose patient to fear animals, adding new animals during the session or enlarging their or increasing the speed. The various studies showed that AR is able, at the beginning of the session, to activate patient’s anxiety, for reducing after 1 h of exposition. After the session, patients even more than to better manage animal’s fear and anxiety, ware able to approach, interact, and kill real feared animals.

Materials and Methods

Data collection.

The input data for the analyses were retrieved from the scientific database Web of Science Core Collection ( Falagas et al., 2008 ) and the search terms used were “Virtual Reality” and “Augmented Reality” regarding papers published during the whole timespan covered.

Web of science core collection is composed of: Citation Indexes, Science Citation Index Expanded (SCI-EXPANDED) –1970-present, Social Sciences Citation Index (SSCI) –1970-present, Arts and Humanities Citation Index (A&HCI) –1975-present, Conference Proceedings Citation Index- Science (CPCI-S) –1990-present, Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) –1990-present, Book Citation Index– Science (BKCI-S) –2009-present, Book Citation Index– Social Sciences & Humanities (BKCI-SSH) –2009-present, Emerging Sources Citation Index (ESCI) –2015-present, Chemical Indexes, Current Chemical Reactions (CCR-EXPANDED) –2009-present (Includes Institut National de la Propriete Industrielle structure data back to 1840), Index Chemicus (IC) –2009-present.

The resultant dataset contained a total of 21,667 records for VR and 9,944 records for AR. The bibliographic record contained various fields, such as author, title, abstract, and all of the references (needed for the citation analysis). The research tool to visualize the networks was Cite space v.4.0.R5 SE (32 bit) ( Chen, 2006 ) under Java Runtime v.8 update 91 (build 1.8.0_91-b15). Statistical analyses were conducted using Stata MP-Parallel Edition, Release 14.0, StataCorp LP. Additional information can be found in Supplementary Data Sheet 1 .

The betweenness centrality of a node in a network measures the extent to which the node is part of paths that connect an arbitrary pair of nodes in the network ( Freeman, 1977 ; Brandes, 2001 ; Chen, 2006 ).

Structural metrics include betweenness centrality, modularity, and silhouette. Temporal and hybrid metrics include citation burstness and novelty. All the algorithms are detailed ( Chen et al., 2010 ).

The analysis of the literature on VR shows a complex panorama. At first sight, according to the document-type statistics from the Web of Science (WoS), proceedings papers were used extensively as outcomes of research, comprising almost 48% of the total (10,392 proceedings), with a similar number of articles on the subject amounting to about 47% of the total of 10, 199 articles. However, if we consider only the last 5 years (7,755 articles representing about 36% of the total), the situation changes with about 57% for articles (4,445) and about 33% for proceedings (2,578). Thus, it is clear that VR field has changed in areas other than at the technological level.

About the subject category, nodes and edges are computed as co-occurring subject categories from the Web of Science “Category” field in all the articles.

According to the subject category statistics from the WoS, computer science is the leading category, followed by engineering, and, together, they account for 15,341 articles, which make up about 71% of the total production. However, if we consider just the last 5 years, these categories reach only about 55%, with a total of 4,284 articles (Table ​ (Table1 1 and Figure ​ Figure1 1 ).

Category statistics from the WoS for the entire period and the last 5 years.

Category from the WoS: network for the last 5 years.

The evidence is very interesting since it highlights that VR is doing very well as new technology with huge interest in hardware and software components. However, with respect to the past, we are witnessing increasing numbers of applications, especially in the medical area. In particular, note its inclusion in the top 10 list of rehabilitation and clinical neurology categories (about 10% of the total production in the last 5 years). It also is interesting that neuroscience and neurology, considered together, have shown an increase from about 12% to about 18.6% over the last 5 years. However, historic areas, such as automation and control systems, imaging science and photographic technology, and robotics, which had accounted for about 14.5% of the total articles ever produced were not even in the top 10 for the last 5 years, with each one accounting for less than 4%.

About the countries, nodes and edges are computed as networks of co-authors countries. Multiple occurrency of a country in the same paper are counted once.

The countries that were very involved in VR research have published for about 47% of the total (10,200 articles altogether). Of the 10,200 articles, the United States, China, England, and Germany published 4921, 2384, 1497, and 1398, respectively. The situation remains the same if we look at the articles published over the last 5 years. However, VR contributions also came from all over the globe, with Japan, Canada, Italy, France, Spain, South Korea, and Netherlands taking positions of prominence, as shown in Figure ​ Figure2 2 .

Country network (node dimension represents centrality).

Network analysis was conducted to calculate and to represent the centrality index ( Freeman, 1977 ; Brandes, 2001 ), i.e., the dimension of the node in Figure ​ Figure2. 2 . The top-ranked country, with a centrality index of 0.26, was the United States (2011), and England was second, with a centrality index of 0.25. The third, fourth, and fifth countries were Germany, Italy, and Australia, with centrality indices of 0.15, 0.15, and 0.14, respectively.

About the Institutions, nodes and edges are computed as networks of co-authors Institutions (Figure ​ (Figure3 3 ).

Network of institutions: the dimensions of the nodes represent centrality.

The top-level institutions in VR were in the United States, where three universities were ranked as the top three in the world for published articles; these universities were the University of Illinois (159), the University of South California (147), and the University of Washington (146). The United States also had the eighth-ranked university, which was Iowa State University (116). The second country in the ranking was Canada, with the University of Toronto, which was ranked fifth with 125 articles and McGill University, ranked 10 th with 103 articles.

Other countries in the top-ten list were Netherlands, with the Delft University of Technology ranked fourth with 129 articles; Italy, with IRCCS Istituto Auxologico Italiano, ranked sixth (with the same number of publication of the institution ranked fifth) with 125 published articles; England, which was ranked seventh with 125 articles from the University of London’s Imperial College of Science, Technology, and Medicine; and China with 104 publications, with the Chinese Academy of Science, ranked ninth. Italy’s Istituto Auxologico Italiano, which was ranked fifth, was the only non-university institution ranked in the top-10 list for VR research (Figure ​ (Figure3 3 ).

About the Journals, nodes, and edges are computed as journal co-citation networks among each journals in the corresponding field.

The top-ranked Journals for citations in VR are Presence: Teleoperators & Virtual Environments with 2689 citations and CyberPsychology & Behavior (Cyberpsychol BEHAV) with 1884 citations; however, looking at the last 5 years, the former had increased the citations, but the latter had a far more significant increase, from about 70% to about 90%, i.e., an increase from 1029 to 1147.

Following the top two journals, IEEE Computer Graphics and Applications ( IEEE Comput Graph) and Advanced Health Telematics and Telemedicine ( St HEAL T) were both left out of the top-10 list based on the last 5 years. The data for the last 5 years also resulted in the inclusion of Experimental Brain Research ( Exp BRAIN RES) (625 citations), Archives of Physical Medicine and Rehabilitation ( Arch PHYS MED REHAB) (622 citations), and Plos ONE (619 citations) in the top-10 list of three journals, which highlighted the categories of rehabilitation and clinical neurology and neuroscience and neurology. Journal co-citation analysis is reported in Figure ​ Figure4, 4 , which clearly shows four distinct clusters.

Co-citation network of journals: the dimensions of the nodes represent centrality. Full list of official abbreviations of WoS journals can be found here: https://images.webofknowledge.com/images/help/WOS/A_abrvjt.html .

Network analysis was conducted to calculate and to represent the centrality index, i.e., the dimensions of the nodes in Figure ​ Figure4. 4 . The top-ranked item by centrality was Cyberpsychol BEHAV, with a centrality index of 0.29. The second-ranked item was Arch PHYS MED REHAB, with a centrality index of 0.23. The third was Behaviour Research and Therapy (Behav RES THER), with a centrality index of 0.15. The fourth was BRAIN, with a centrality index of 0.14. The fifth was Exp BRAIN RES, with a centrality index of 0.11.

Who’s Who in VR Research

Authors are the heart and brain of research, and their roles in a field are to define the past, present, and future of disciplines and to make significant breakthroughs to make new ideas arise (Figure ​ (Figure5 5 ).

Network of authors’ numbers of publications: the dimensions of the nodes represent the centrality index, and the dimensions of the characters represent the author’s rank.

Virtual reality research is very young and changing with time, but the top-10 authors in this field have made fundamentally significant contributions as pioneers in VR and taking it beyond a mere technological development. The purpose of the following highlights is not to rank researchers; rather, the purpose is to identify the most active researchers in order to understand where the field is going and how they plan for it to get there.

The top-ranked author is Riva G, with 180 publications. The second-ranked author is Rizzo A, with 101 publications. The third is Darzi A, with 97 publications. The forth is Aggarwal R, with 94 publications. The six authors following these three are Slater M, Alcaniz M, Botella C, Wiederhold BK, Kim SI, and Gutierrez-Maldonado J with 90, 90, 85, 75, 59, and 54 publications, respectively (Figure ​ (Figure6 6 ).

Authors’ co-citation network: the dimensions of the nodes represent centrality index, and the dimensions of the characters represent the author’s rank. The 10 authors that appear on the top-10 list are considered to be the pioneers of VR research.

Considering the last 5 years, the situation remains similar, with three new entries in the top-10 list, i.e., Muhlberger A, Cipresso P, and Ahmed K ranked 7th, 8th, and 10th, respectively.

The authors’ publications number network shows the most active authors in VR research. Another relevant analysis for our focus on VR research is to identify the most cited authors in the field.

For this purpose, the authors’ co-citation analysis highlights the authors in term of their impact on the literature considering the entire time span of the field ( White and Griffith, 1981 ; González-Teruel et al., 2015 ; Bu et al., 2016 ). The idea is to focus on the dynamic nature of the community of authors who contribute to the research.

Normally, authors with higher numbers of citations tend to be the scholars who drive the fundamental research and who make the most meaningful impacts on the evolution and development of the field. In the following, we identified the most-cited pioneers in the field of VR Research.

The top-ranked author by citation count is Gallagher (2001), with 694 citations. Second is Seymour (2004), with 668 citations. Third is Slater (1999), with 649 citations. Fourth is Grantcharov (2003), with 563 citations. Fifth is Riva (1999), with 546 citations. Sixth is Aggarwal (2006), with 505 citations. Seventh is Satava (1994), with 477 citations. Eighth is Witmer (2002), with 454 citations. Ninth is Rothbaum (1996), with 448 citations. Tenth is Cruz-neira (1995), with 416 citations.

Citation Network and Cluster Analyses for VR

Another analysis that can be used is the analysis of document co-citation, which allows us to focus on the highly-cited documents that generally are also the most influential in the domain ( Small, 1973 ; González-Teruel et al., 2015 ; Orosz et al., 2016 ).

The top-ranked article by citation counts is Seymour (2002) in Cluster #0, with 317 citations. The second article is Grantcharov (2004) in Cluster #0, with 286 citations. The third is Holden (2005) in Cluster #2, with 179 citations. The 4th is Gallagher et al. (2005) in Cluster #0, with 171 citations. The 5th is Ahlberg (2007) in Cluster #0, with 142 citations. The 6th is Parsons (2008) in Cluster #4, with 136 citations. The 7th is Powers (2008) in Cluster #4, with 134 citations. The 8th is Aggarwal (2007) in Cluster #0, with 121 citations. The 9th is Reznick (2006) in Cluster #0, with 121 citations. The 10th is Munz (2004) in Cluster #0, with 117 citations.

The network of document co-citations is visually complex (Figure ​ (Figure7) 7 ) because it includes 1000s of articles and the links among them. However, this analysis is very important because can be used to identify the possible conglomerate of knowledge in the area, and this is essential for a deep understanding of the area. Thus, for this purpose, a cluster analysis was conducted ( Chen et al., 2010 ; González-Teruel et al., 2015 ; Klavans and Boyack, 2015 ). Figure ​ Figure8 8 shows the clusters, which are identified with the two algorithms in Table ​ Table2 2 .

Network of document co-citations: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank, and the numbers represent the strengths of the links. It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past VR research to the current research.

Document co-citation network by cluster: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing reports the name of the cluster with a short description that was produced with the mutual information algorithm; the clusters are identified with colored polygons.

Cluster ID and silhouettes as identified with two algorithms ( Chen et al., 2010 ).

The identified clusters highlight clear parts of the literature of VR research, making clear and visible the interdisciplinary nature of this field. However, the dynamics to identify the past, present, and future of VR research cannot be clear yet. We analysed the relationships between these clusters and the temporal dimensions of each article. The results are synthesized in Figure ​ Figure9. 9 . It is clear that cluster #0 (laparoscopic skill), cluster #2 (gaming and rehabilitation), cluster #4 (therapy), and cluster #14 (surgery) are the most popular areas of VR research. (See Figure ​ Figure9 9 and Table ​ Table2 2 to identify the clusters.) From Figure ​ Figure9, 9 , it also is possible to identify the first phase of laparoscopic skill (cluster #6) and therapy (cluster #7). More generally, it is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past VR research to the current research.

Network of document co-citation: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing on the right hand side reports the number of the cluster, such as in Table ​ Table2, 2 , with a short description that was extracted accordingly.

We were able to identify the top 486 references that had the most citations by using burst citations algorithm. Citation burst is an indicator of a most active area of research. Citation burst is a detection of a burst event, which can last for multiple years as well as a single year. A citation burst provides evidence that a particular publication is associated with a surge of citations. The burst detection was based on Kleinberg’s algorithm ( Kleinberg, 2002 , 2003 ). The top-ranked document by bursts is Seymour (2002) in Cluster #0, with bursts of 88.93. The second is Grantcharov (2004) in Cluster #0, with bursts of 51.40. The third is Saposnik (2010) in Cluster #2, with bursts of 40.84. The fourth is Rothbaum (1995) in Cluster #7, with bursts of 38.94. The fifth is Holden (2005) in Cluster #2, with bursts of 37.52. The sixth is Scott (2000) in Cluster #0, with bursts of 33.39. The seventh is Saposnik (2011) in Cluster #2, with bursts of 33.33. The eighth is Burdea et al. (1996) in Cluster #3, with bursts of 32.42. The ninth is Burdea and Coiffet (2003) in Cluster #22, with bursts of 31.30. The 10th is Taffinder (1998) in Cluster #6, with bursts of 30.96 (Table ​ (Table3 3 ).

Cluster ID and references of burst article.

Citation Network and Cluster Analyses for AR

Looking at Augmented Reality scenario, the top ranked item by citation counts is Azuma (1997) in Cluster #0, with citation counts of 231. The second one is Azuma et al. (2001) in Cluster #0, with citation counts of 220. The third is Van Krevelen (2010) in Cluster #5, with citation counts of 207. The 4th is Lowe (2004) in Cluster #1, with citation counts of 157. The 5th is Wu (2013) in Cluster #4, with citation counts of 144. The 6th is Dunleavy (2009) in Cluster #4, with citation counts of 122. The 7th is Zhou (2008) in Cluster #5, with citation counts of 118. The 8th is Bay (2008) in Cluster #1, with citation counts of 117. The 9th is Newcombe (2011) in Cluster #1, with citation counts of 109. The 10th is Carmigniani et al. (2011) in Cluster #5, with citation counts of 104.

The network of document co-citations is visually complex (Figure ​ (Figure10) 10 ) because it includes 1000s of articles and the links among them. However, this analysis is very important because can be used to identify the possible conglomerate of knowledge in the area, and this is essential for a deep understanding of the area. Thus, for this purpose, a cluster analysis was conducted ( Chen et al., 2010 ; González-Teruel et al., 2015 ; Klavans and Boyack, 2015 ). Figure ​ Figure11 11 shows the clusters, which are identified with the two algorithms in Table ​ Table3 3 .

Network of document co-citations: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank, and the numbers represent the strengths of the links. It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past AR research to the current research.

The identified clusters highlight clear parts of the literature of AR research, making clear and visible the interdisciplinary nature of this field. However, the dynamics to identify the past, present, and future of AR research cannot be clear yet. We analysed the relationships between these clusters and the temporal dimensions of each article. The results are synthesized in Figure ​ Figure12. 12 . It is clear that cluster #1 (tracking), cluster #4 (education), and cluster #5 (virtual city environment) are the current areas of AR research. (See Figure ​ Figure12 12 and Table ​ Table3 3 to identify the clusters.) It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past AR research to the current research.

We were able to identify the top 394 references that had the most citations by using burst citations algorithm. Citation burst is an indicator of a most active area of research. Citation burst is a detection of a burst event, which can last for multiple years as well as a single year. A citation burst provides evidence that a particular publication is associated with a surge of citations. The burst detection was based on Kleinberg’s algorithm ( Kleinberg, 2002 , 2003 ). The top ranked document by bursts is Azuma (1997) in Cluster #0, with bursts of 101.64. The second one is Azuma et al. (2001) in Cluster #0, with bursts of 84.23. The third is Lowe (2004) in Cluster #1, with bursts of 64.07. The 4th is Van Krevelen (2010) in Cluster #5, with bursts of 50.99. The 5th is Wu (2013) in Cluster #4, with bursts of 47.23. The 6th is Hartley (2000) in Cluster #0, with bursts of 37.71. The 7th is Dunleavy (2009) in Cluster #4, with bursts of 33.22. The 8th is Kato (1999) in Cluster #0, with bursts of 32.16. The 9th is Newcombe (2011) in Cluster #1, with bursts of 29.72. The 10th is Feiner (1993) in Cluster #8, with bursts of 29.46 (Table ​ (Table4 4 ).

Our findings have profound implications for two reasons. At first the present work highlighted the evolution and development of VR and AR research and provided a clear perspective based on solid data and computational analyses. Secondly our findings on VR made it profoundly clear that the clinical dimension is one of the most investigated ever and seems to increase in quantitative and qualitative aspects, but also include technological development and article in computer science, engineer, and allied sciences.

Figure ​ Figure9 9 clarifies the past, present, and future of VR research. The outset of VR research brought a clearly-identifiable development in interfaces for children and medicine, routine use and behavioral-assessment, special effects, systems perspectives, and tutorials. This pioneering era evolved in the period that we can identify as the development era, because it was the period in which VR was used in experiments associated with new technological impulses. Not surprisingly, this was exactly concomitant with the new economy era in which significant investments were made in information technology, and it also was the era of the so-called ‘dot-com bubble’ in the late 1990s. The confluence of pioneering techniques into ergonomic studies within this development era was used to develop the first effective clinical systems for surgery, telemedicine, human spatial navigation, and the first phase of the development of therapy and laparoscopic skills. With the new millennium, VR research switched strongly toward what we can call the clinical-VR era, with its strong emphasis on rehabilitation, neurosurgery, and a new phase of therapy and laparoscopic skills. The number of applications and articles that have been published in the last 5 years are in line with the new technological development that we are experiencing at the hardware level, for example, with so many new, HMDs, and at the software level with an increasing number of independent programmers and VR communities.

Finally, Figure ​ Figure12 12 identifies clusters of the literature of AR research, making clear and visible the interdisciplinary nature of this field. The dynamics to identify the past, present, and future of AR research cannot be clear yet, but analyzing the relationships between these clusters and the temporal dimensions of each article tracking, education, and virtual city environment are the current areas of AR research. AR is a new technology that is showing its efficacy in different research fields, and providing a novel way to gather behavioral data and support learning, training, and clinical treatments.

Looking at scientific literature conducted in the last few years, it might appear that most developments in VR and AR studies have focused on clinical aspects. However, the reality is more complex; thus, this perception should be clarified. Although researchers publish studies on the use of VR in clinical settings, each study depends on the technologies available. Industrial development in VR and AR changed a lot in the last 10 years. In the past, the development involved mainly hardware solutions while nowadays, the main efforts pertain to the software when developing virtual solutions. Hardware became a commodity that is often available at low cost. On the other hand, software needs to be customized each time, per each experiment, and this requires huge efforts in term of development. Researchers in AR and VR today need to be able to adapt software in their labs.

Virtual reality and AR developments in this new clinical era rely on computer science and vice versa. The future of VR and AR is becoming more technological than before, and each day, new solutions and products are coming to the market. Both from software and hardware perspectives, the future of AR and VR depends on huge innovations in all fields. The gap between the past and the future of AR and VR research is about the “realism” that was the key aspect in the past versus the “interaction” that is the key aspect now. First 30 years of VR and AR consisted of a continuous research on better resolution and improved perception. Now, researchers already achieved a great resolution and need to focus on making the VR as realistic as possible, which is not simple. In fact, a real experience implies a realistic interaction and not just great resolution. Interactions can be improved in infinite ways through new developments at hardware and software levels.

Interaction in AR and VR is going to be “embodied,” with implication for neuroscientists that are thinking about new solutions to be implemented into the current systems ( Blanke et al., 2015 ; Riva, 2018 ; Riva et al., 2018 ). For example, the use of hands with contactless device (i.e., without gloves) makes the interaction in virtual environments more natural. The Leap Motion device 1 allows one to use of hands in VR without the use of gloves or markers. This simple and low-cost device allows the VR users to interact with virtual objects and related environments in a naturalistic way. When technology is able to be transparent, users can experience increased sense of being in the virtual environments (the so-called sense of presence).

Other forms of interactions are possible and have been developing continuously. For example, tactile and haptic device able to provide a continuous feedback to the users, intensifying their experience also by adding components, such as the feeling of touch and the physical weight of virtual objects, by using force feedback. Another technology available at low cost that facilitates interaction is the motion tracking system, such as Microsoft Kinect, for example. Such technology allows one to track the users’ bodies, allowing them to interact with the virtual environments using body movements, gestures, and interactions. Most HMDs use an embedded system to track HMD position and rotation as well as controllers that are generally placed into the user’s hands. This tracking allows a great degree of interaction and improves the overall virtual experience.

A final emerging approach is the use of digital technologies to simulate not only the external world but also the internal bodily signals ( Azevedo et al., 2017 ; Riva et al., 2017 ): interoception, proprioception and vestibular input. For example, Riva et al. (2017) recently introduced the concept of “sonoception” ( www.sonoception.com ), a novel non-invasive technological paradigm based on wearable acoustic and vibrotactile transducers able to alter internal bodily signals. This approach allowed the development of an interoceptive stimulator that is both able to assess interoceptive time perception in clinical patients ( Di Lernia et al., 2018b ) and to enhance heart rate variability (the short-term vagally mediated component—rMSSD) through the modulation of the subjects’ parasympathetic system ( Di Lernia et al., 2018a ).

In this scenario, it is clear that the future of VR and AR research is not just in clinical applications, although the implications for the patients are huge. The continuous development of VR and AR technologies is the result of research in computer science, engineering, and allied sciences. The reasons for which from our analyses emerged a “clinical era” are threefold. First, all clinical research on VR and AR includes also technological developments, and new technological discoveries are being published in clinical or technological journals but with clinical samples as main subject. As noted in our research, main journals that publish numerous articles on technological developments tested with both healthy and patients include Presence: Teleoperators & Virtual Environments, Cyberpsychology & Behavior (Cyberpsychol BEHAV), and IEEE Computer Graphics and Applications (IEEE Comput Graph). It is clear that researchers in psychology, neuroscience, medicine, and behavioral sciences in general have been investigating whether the technological developments of VR and AR are effective for users, indicating that clinical behavioral research has been incorporating large parts of computer science and engineering. A second aspect to consider is the industrial development. In fact, once a new technology is envisioned and created it goes for a patent application. Once the patent is sent for registration the new technology may be made available for the market, and eventually for journal submission and publication. Moreover, most VR and AR research that that proposes the development of a technology moves directly from the presenting prototype to receiving the patent and introducing it to the market without publishing the findings in scientific paper. Hence, it is clear that if a new technology has been developed for industrial market or consumer, but not for clinical purpose, the research conducted to develop such technology may never be published in a scientific paper. Although our manuscript considered published researches, we have to acknowledge the existence of several researches that have not been published at all. The third reason for which our analyses highlighted a “clinical era” is that several articles on VR and AR have been considered within the Web of Knowledge database, that is our source of references. In this article, we referred to “research” as the one in the database considered. Of course, this is a limitation of our study, since there are several other databases that are of big value in the scientific community, such as IEEE Xplore Digital Library, ACM Digital Library, and many others. Generally, the most important articles in journals published in these databases are also included in the Web of Knowledge database; hence, we are convinced that our study considered the top-level publications in computer science or engineering. Accordingly, we believe that this limitation can be overcome by considering the large number of articles referenced in our research.

Considering all these aspects, it is clear that clinical applications, behavioral aspects, and technological developments in VR and AR research are parts of a more complex situation compared to the old platforms used before the huge diffusion of HMD and solutions. We think that this work might provide a clearer vision for stakeholders, providing evidence of the current research frontiers and the challenges that are expected in the future, highlighting all the connections and implications of the research in several fields, such as clinical, behavioral, industrial, entertainment, educational, and many others.

Author Contributions

PC and GR conceived the idea. PC made data extraction and the computational analyses and wrote the first draft of the article. IG revised the introduction adding important information for the article. PC, IG, MR, and GR revised the article and approved the last version of the article after important input to the article rationale.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer GC declared a shared affiliation, with no collaboration, with the authors PC and GR to the handling Editor at the time of the review.

1 https://www.leapmotion.com/

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2018.02086/full#supplementary-material

• Akçayır M., Akçayır G. (2017). Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ. Res. Rev. 20 1–11. 10.1016/j.edurev.2016.11.002 [ CrossRef ] [ Google Scholar ]
• Alexander T., Westhoven M., Conradi J. (2017). “Virtual environments for competency-oriented education and training,” in Advances in Human Factors, Business Management, Training and Education , (Berlin: Springer International Publishing; ), 23–29. 10.1007/978-3-319-42070-7_3 [ CrossRef ] [ Google Scholar ]
• Andersen S. M., Thorpe J. S. (2009). An if–thEN theory of personality: significant others and the relational self. J. Res. Pers. 43 163–170. 10.1016/j.jrp.2008.12.040 [ CrossRef ] [ Google Scholar ]
• Azevedo R. T., Bennett N., Bilicki A., Hooper J., Markopoulou F., Tsakiris M. (2017). The calming effect of a new wearable device during the anticipation of public speech. Sci. Rep. 7 : 2285 . 10.1038/s41598-017-02274-2 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Azuma R., Baillot Y., Behringer R., Feiner S., Julier S., MacIntyre B. (2001). Recent advances in augmented reality. IEEE Comp. Graph. Appl. 21 34–47. 10.1109/38.963459 [ CrossRef ] [ Google Scholar ]
• Bacca J., Baldiris S., Fabregat R., Graf S. (2014). Augmented reality trends in education: a systematic review of research and applications. J. Educ. Technol. Soc. 17 133 . [ Google Scholar ]
• Bailenson J. N., Yee N., Merget D., Schroeder R. (2006). The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction. Presence 15 359–372. 10.1162/pres.15.4.359 [ CrossRef ] [ Google Scholar ]
• Baños R. M., Botella C., Garcia-Palacios A., Villa H., Perpiñá C., Alcaniz M. (2000). Presence and reality judgment in virtual environments: a unitary construct? Cyberpsychol. Behav. 3 327–335. 10.1089/10949310050078760 [ CrossRef ] [ Google Scholar ]
• Baños R., Botella C., García-Palacios A., Villa H., Perpiñá C., Gallardo M. (2009). Psychological variables and reality judgment in virtual environments: the roles of absorption and dissociation. Cyberpsychol. Behav. 2 143–148. 10.1089/cpb.1999.2.143 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Baus O., Bouchard S. (2014). Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: a review. Front. Hum. Neurosci. 8 : 112 . 10.3389/fnhum.2014.00112 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Biocca F. (1997). The cyborg’s dilemma: progressive embodiment in virtual environments. J. Comput. Mediat. Commun. 3 10.1111/j.1083-6101.1997 [ CrossRef ] [ Google Scholar ]
• Biocca F., Harms C., Gregg J. (2001). “The networked minds measure of social presence: pilot test of the factor structure and concurrent validity,” in 4th Annual International Workshop on Presence , Philadelphia, PA, 1–9. [ Google Scholar ]
• Blanke O., Slater M., Serino A. (2015). Behavioral, neural, and computational principles of bodily self-consciousness. Neuron 88 145–166. 10.1016/j.neuron.2015.09.029 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bohil C. J., Alicea B., Biocca F. A. (2011). Virtual reality in neuroscience research and therapy. Nat. Rev. Neurosci. 12 : 752 . 10.1038/nrn3122 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Borrego A., Latorre J., Llorens R., Alcañiz M., Noé E. (2016). Feasibility of a walking virtual reality system for rehabilitation: objective and subjective parameters. J. Neuroeng. Rehabil. 13 : 68 . 10.1186/s12984-016-0174-171 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Botella C., Bretón-López J., Quero S., Baños R. M., García-Palacios A. (2010). Treating cockroach phobia with augmented reality. Behav. Ther. 41 401–413. 10.1016/j.beth.2009.07.002 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Botella C., Fernández-Álvarez J., Guillén V., García-Palacios A., Baños R. (2017). Recent progress in virtual reality exposure therapy for phobias: a systematic review. Curr. Psychiatry Rep. 19 : 42 . 10.1007/s11920-017-0788-4 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Botella C. M., Juan M. C., Baños R. M., Alcañiz M., Guillén V., Rey B. (2005). Mixing realities? An application of augmented reality for the treatment of cockroach phobia. Cyberpsychol. Behav. 8 162–171. 10.1089/cpb.2005.8.162 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Brandes U. (2001). A faster algorithm for betweenness centrality. J. Math. Sociol. 25 163–177. 10.1080/0022250X.2001.9990249 [ CrossRef ] [ Google Scholar ]
• Bretón-López J., Quero S., Botella C., García-Palacios A., Baños R. M., Alcañiz M. (2010). An augmented reality system validation for the treatment of cockroach phobia. Cyberpsychol. Behav. Soc. Netw. 13 705–710. 10.1089/cyber.2009.0170 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Brown A., Green T. (2016). Virtual reality: low-cost tools and resources for the classroom. TechTrends 60 517–519. 10.1007/s11528-016-0102-z [ CrossRef ] [ Google Scholar ]
• Bu Y., Liu T. Y., Huang W. B. (2016). MACA: a modified author co-citation analysis method combined with general descriptive metadata of citations. Scientometrics 108 143–166. 10.1007/s11192-016-1959-5 [ CrossRef ] [ Google Scholar ]
• Burdea G., Richard P., Coiffet P. (1996). Multimodal virtual reality: input-output devices, system integration, and human factors. Int. J. Hum. Compu. Interact. 8 5–24. 10.1080/10447319609526138 [ CrossRef ] [ Google Scholar ]
• Burdea G. C., Coiffet P. (2003). Virtual Reality Technology Vol. 1 Hoboken, NJ: John Wiley & Sons. [ Google Scholar ]
• Carmigniani J., Furht B., Anisetti M., Ceravolo P., Damiani E., Ivkovic M. (2011). Augmented reality technologies, systems and applications. Multimed. Tools Appl. 51 341–377. 10.1007/s11042-010-0660-6 [ CrossRef ] [ Google Scholar ]
• Castelvecchi D. (2016). Low-cost headsets boost virtual reality’s lab appeal. Nature 533 153–154. 10.1038/533153a [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cathy (2011). The History of Augmented Reality. The Optical Vision Site. Available at: http://www.theopticalvisionsite.com/history-of-eyewear/the-history-of-augmented-reality/#.UelAUmeAOyA [ Google Scholar ]
• Chen C. (2006). CiteSpace II: detecting and visualizing emerging trends and transient patterns in scientific literature. J. Assoc. Inform. Sci. Technol. 57 359–377. 10.1002/asi.20317 [ CrossRef ] [ Google Scholar ]
• Chen C., Ibekwe-SanJuan F., Hou J. (2010). The structure and dynamics of cocitation clusters: a multipleperspective cocitation analysis. J. Assoc. Inform. Sci. Technol. 61 1386–1409. 10.1002/jez.b.22741 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chen Y. C., Chi H. L., Hung W. H., Kang S. C. (2011). Use of tangible and augmented reality models in engineering graphics courses. J. Prof. Issues Eng. Educ. Pract. 137 267–276. 10.1061/(ASCE)EI.1943-5541.0000078 [ CrossRef ] [ Google Scholar ]
• Chicchi Giglioli I. A., Pallavicini F., Pedroli E., Serino S., Riva G. (2015). Augmented reality: a brand new challenge for the assessment and treatment of psychological disorders. Comput. Math. Methods Med. 2015 : 862942 . 10.1155/2015/862942 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chien C. H., Chen C. H., Jeng T. S. (2010). “An interactive augmented reality system for learning anatomy structure,” in Proceedings of the International Multiconference of Engineers and Computer Scientists , Vol. 1 (Hong Kong: International Association of Engineers; ), 17–19. [ Google Scholar ]
• Choi S., Jung K., Noh S. D. (2015). Virtual reality applications in manufacturing industries: past research, present findings, and future directions. Concurr. Eng. 23 40–63. 10.1177/1063293X14568814 [ CrossRef ] [ Google Scholar ]
• Cipresso P. (2015). Modeling behavior dynamics using computational psychometrics within virtual worlds. Front. Psychol. 6 : 1725 . 10.3389/fpsyg.2015.01725 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cipresso P., Serino S. (2014). Virtual Reality: Technologies, Medical Applications and Challenges. Hauppauge, NY: Nova Science Publishers, Inc. [ Google Scholar ]
• Cipresso P., Serino S., Riva G. (2016). Psychometric assessment and behavioral experiments using a free virtual reality platform and computational science. BMC Med. Inform. Decis. Mak. 16 : 37 . 10.1186/s12911-016-0276-5 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cruz-Neira C. (1993). “Virtual reality overview,” in SIGGRAPH 93 Course Notes 21st International Conference on Computer Graphics and Interactive Techniques, Orange County Convention Center , Orlando, FL. [ Google Scholar ]
• De Buck S., Maes F., Ector J., Bogaert J., Dymarkowski S., Heidbuchel H., et al. (2005). An augmented reality system for patient-specific guidance of cardiac catheter ablation procedures. IEEE Trans. Med. Imaging 24 1512–1524. 10.1109/TMI.2005.857661 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Di Lernia D., Cipresso P., Pedroli E., Riva G. (2018a). Toward an embodied medicine: a portable device with programmable interoceptive stimulation for heart rate variability enhancement. Sensors (Basel) 18 : 2469 . 10.3390/s18082469 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Di Lernia D., Serino S., Pezzulo G., Pedroli E., Cipresso P., Riva G. (2018b). Feel the time. Time perception as a function of interoceptive processing. Front. Hum. Neurosci. 12 : 74 . 10.3389/fnhum.2018.00074 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Di Serio Á., Ibáñez M. B., Kloos C. D. (2013). Impact of an augmented reality system on students’ motivation for a visual art course. Comput. Educ. 68 586–596. 10.1016/j.compedu.2012.03.002 [ CrossRef ] [ Google Scholar ]
• Ebert C. (2015). Looking into the future. IEEE Softw. 32 92–97. 10.1109/MS.2015.142 [ CrossRef ] [ Google Scholar ]
• Englund C., Olofsson A. D., Price L. (2017). Teaching with technology in higher education: understanding conceptual change and development in practice. High. Educ. Res. Dev. 36 73–87. 10.1080/07294360.2016.1171300 [ CrossRef ] [ Google Scholar ]
• Falagas M. E., Pitsouni E. I., Malietzis G. A., Pappas G. (2008). Comparison of pubmed, scopus, web of science, and Google scholar: strengths and weaknesses. FASEB J. 22 338–342. 10.1096/fj.07-9492LSF [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Feiner S., MacIntyre B., Hollerer T., Webster A. (1997). “A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment,” in Digest of Papers. First International Symposium on Wearable Computers , (Cambridge, MA: IEEE; ), 74–81. 10.1109/ISWC.1997.629922 [ CrossRef ] [ Google Scholar ]
• Freeman D., Reeve S., Robinson A., Ehlers A., Clark D., Spanlang B., et al. (2017). Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol. Med. 47 2393–2400. 10.1017/S003329171700040X [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Freeman L. C. (1977). A set of measures of centrality based on betweenness. Sociometry 40 35–41. 10.2307/3033543 [ CrossRef ] [ Google Scholar ]
• Fuchs H., Bishop G. (1992). Research Directions in Virtual Environments. Chapel Hill, NC: University of North Carolina at Chapel Hill. [ Google Scholar ]
• Gallagher A. G., Ritter E. M., Champion H., Higgins G., Fried M. P., Moses G., et al. (2005). Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann. Surg. 241 : 364 . 10.1097/01.sla.0000151982.85062.80 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gigante M. A. (1993). Virtual reality: definitions, history and applications. Virtual Real. Syst. 3–14. 10.1016/B978-0-12-227748-1.50009-3 [ CrossRef ] [ Google Scholar ]
• González-Teruel A., González-Alcaide G., Barrios M., Abad-García M. F. (2015). Mapping recent information behavior research: an analysis of co-authorship and co-citation networks. Scientometrics 103 687–705. 10.1007/s11192-015-1548-z [ CrossRef ] [ Google Scholar ]
• Heeter C. (1992). Being there: the subjective experience of presence. Presence 1 262–271. 10.1162/pres.1992.1.2.262 [ CrossRef ] [ Google Scholar ]
• Heeter C. (2000). Interactivity in the context of designed experiences. J. Interact. Adv. 1 3–14. 10.1080/15252019.2000.10722040 [ CrossRef ] [ Google Scholar ]
• Heilig M. (1962). Sensorama simulator . U.S. Patent No - 3, 870. Virginia: United States Patent and Trade Office. [ Google Scholar ]
• Ibáñez M. B., Di Serio Á., Villarán D., Kloos C. D. (2014). Experimenting with electromagnetism using augmented reality: impact on flow student experience and educational effectiveness. Comput. Educ. 71 1–13. 10.1016/j.compedu.2013.09.004 [ CrossRef ] [ Google Scholar ]
• Juan M. C., Alcañiz M., Calatrava J., Zaragozá I., Baños R., Botella C. (2007). “An optical see-through augmented reality system for the treatment of phobia to small animals,” in Virtual Reality, HCII 2007 Lecture Notes in Computer Science Vol. 4563 ed. Schumaker R. (Berlin: Springer; ), 651–659. [ Google Scholar ]
• Juan M. C., Alcaniz M., Monserrat C., Botella C., Baños R. M., Guerrero B. (2005). Using augmented reality to treat phobias. IEEE Comput. Graph. Appl. 25 31–37. 10.1109/MCG.2005.143 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kim G. J. (2005). A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence 14 119–146. 10.1162/1054746053967094 [ CrossRef ] [ Google Scholar ]
• Klavans R., Boyack K. W. (2015). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? J. Assoc. Inform. Sci. Technol. 68 984–998. 10.1002/asi.23734 [ CrossRef ] [ Google Scholar ]
• Kleinberg J. (2002). “Bursty and hierarchical structure in streams,” in Paper Presented at the Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2002; Edmonton , Alberta, NT: 10.1145/775047.775061 [ CrossRef ] [ Google Scholar ]
• Kleinberg J. (2003). Bursty and hierarchical structure in streams. Data Min. Knowl. Discov. 7 373–397. 10.1023/A:1024940629314 [ CrossRef ] [ Google Scholar ]
• Korolov M. (2014). The real risks of virtual reality. Risk Manag. 61 20–24. [ Google Scholar ]
• Krueger M. W., Gionfriddo T., Hinrichsen K. (1985). “Videoplace—an artificial reality,” in Proceedings of the ACM SIGCHI Bulletin Vol. 16 New York, NY: ACM, 35–40. 10.1145/317456.317463 [ CrossRef ] [ Google Scholar ]
• Lin C. H., Hsu P. H. (2017). “Integrating procedural modelling process and immersive VR environment for architectural design education,” in MATEC Web of Conferences Vol. 104 Les Ulis: EDP Sciences; 10.1051/matecconf/201710403007 [ CrossRef ] [ Google Scholar ]
• Llorens R., Noé E., Ferri J., Alcañiz M. (2014). Virtual reality-based telerehabilitation program for balance recovery. A pilot study in hemiparetic individuals with acquired brain injury. Brain Inj. 28 : 169 . [ Google Scholar ]
• Lombard M., Ditton T. (1997). At the heart of it all: the concept of presence. J. Comput. Mediat. Commun. 3 10.1111/j.1083-6101.1997.tb00072.x [ CrossRef ] [ Google Scholar ]
• Loomis J. M., Blascovich J. J., Beall A. C. (1999). Immersive virtual environment technology as a basic research tool in psychology. Behav. Res. Methods Instr. Comput. 31 557–564. 10.3758/BF03200735 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Loomis J. M., Golledge R. G., Klatzky R. L. (1998). Navigation system for the blind: auditory display modes and guidance. Presence 7 193–203. 10.1162/105474698565677 [ CrossRef ] [ Google Scholar ]
• Luckerson V. (2014). Facebook Buying Oculus Virtual-Reality Company for $2 Billion. Available at: http://time.com/37842/facebook-oculus-rift [ Google Scholar ]
• Maurugeon G. (2011). New D’Fusion Supports iPhone4S and 3DSMax 2012. Available at: http://www.t-immersion.com/blog/2011-12-07/augmented-reality-dfusion-iphone-3dsmax [ Google Scholar ]
• Mazuryk T., Gervautz M. (1996). Virtual Reality-History, Applications, Technology and Future. Vienna: Institute of Computer Graphics Vienna University of Technology. [ Google Scholar ]
• Meldrum D., Glennon A., Herdman S., Murray D., McConn-Walsh R. (2012). Virtual reality rehabilitation of balance: assessment of the usability of the nintendo Wii ® fit plus. Disabil. Rehabil. 7 205–210. 10.3109/17483107.2011.616922 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Milgram P., Kishino F. (1994). A taxonomy of mixed reality visual displays. IEICE Trans. Inform. Syst. 77 1321–1329. [ Google Scholar ]
• Minderer M., Harvey C. D., Donato F., Moser E. I. (2016). Neuroscience: virtual reality explored. Nature 533 324–325. 10.1038/nature17899 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Neri S. G., Cardoso J. R., Cruz L., Lima R. M., de Oliveira R. J., Iversen M. D., et al. (2017). Do virtual reality games improve mobility skills and balance measurements in community-dwelling older adults? Systematic review and meta-analysis. Clin. Rehabil. 31 1292–1304. 10.1177/0269215517694677 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Nincarean D., Alia M. B., Halim N. D. A., Rahman M. H. A. (2013). Mobile augmented reality: the potential for education. Procedia Soc. Behav. Sci. 103 657–664. 10.1016/j.sbspro.2013.10.385 [ CrossRef ] [ Google Scholar ]
• Orosz K., Farkas I. J., Pollner P. (2016). Quantifying the changing role of past publications. Scientometrics 108 829–853. 10.1007/s11192-016-1971-9 [ CrossRef ] [ Google Scholar ]
• Ozbek C. S., Giesler B., Dillmann R. (2004). “Jedi training: playful evaluation of head-mounted augmented reality display systems,” in Proceedings of SPIE. The International Society for Optical Engineering Vol. 5291 eds Norwood R. A., Eich M., Kuzyk M. G. (Denver, CO: ), 454–463. [ Google Scholar ]
• Perry S. (2008). Wikitude: Android App with Augmented Reality: Mind Blow-Ing. Digital Lifestyles. [ Google Scholar ]
• Radu I. (2012). “Why should my students use AR? A comparative review of the educational impacts of augmented-reality,” in Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on , (IEEE) , 313–314. 10.1109/ISMAR.2012.6402590 [ CrossRef ] [ Google Scholar ]
• Radu I. (2014). Augmented reality in education: a meta-review and cross-media analysis. Pers. Ubiquitous Comput. 18 1533–1543. 10.1007/s00779-013-0747-y [ CrossRef ] [ Google Scholar ]
• Riva G. (2018). The neuroscience of body memory: From the self through the space to the others. Cortex 104 241–260. 10.1016/j.cortex.2017.07.013 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Riva G., Gaggioli A., Grassi A., Raspelli S., Cipresso P., Pallavicini F., et al. (2011). NeuroVR 2-A free virtual reality platform for the assessment and treatment in behavioral health care. Stud. Health Technol. Inform. 163 493–495. [ PubMed ] [ Google Scholar ]
• Riva G., Serino S., Di Lernia D., Pavone E. F., Dakanalis A. (2017). Embodied medicine: mens sana in corpore virtuale sano. Front. Hum. Neurosci. 11 : 120 . 10.3389/fnhum.2017.00120 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Riva G., Wiederhold B. K., Mantovani F. (2018). Neuroscience of virtual reality: from virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. 10.1089/cyber.2017.29099.gri [Epub ahead of print]. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rosenberg L. (1993). “The use of virtual fixtures to enhance telemanipulation with time delay,” in Proceedings of the ASME Winter Anual Meeting on Advances in Robotics, Mechatronics, and Haptic Interfaces Vol. 49 (New Orleans, LA: ), 29–36. [ Google Scholar ]
• Schmidt M., Beck D., Glaser N., Schmidt C. (2017). “A prototype immersive, multi-user 3D virtual learning environment for individuals with autism to learn social and life skills: a virtuoso DBR update,” in International Conference on Immersive Learning , Cham: Springer, 185–188. 10.1007/978-3-319-60633-0_15 [ CrossRef ] [ Google Scholar ]
• Schwald B., De Laval B. (2003). An augmented reality system for training and assistance to maintenance in the industrial context. J. WSCG 11 . [ Google Scholar ]
• Serino S., Cipresso P., Morganti F., Riva G. (2014). The role of egocentric and allocentric abilities in Alzheimer’s disease: a systematic review. Ageing Res. Rev. 16 32–44. 10.1016/j.arr.2014.04.004 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Skalski P., Tamborini R. (2007). The role of social presence in interactive agent-based persuasion. Media Psychol. 10 385–413. 10.1080/15213260701533102 [ CrossRef ] [ Google Scholar ]
• Slater M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364 3549–3557. 10.1098/rstb.2009.0138 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Slater M., Sanchez-Vives M. V. (2016). Enhancing our lives with immersive virtual reality. Front. Robot. AI 3 : 74 10.3389/frobt.2016.00074 [ CrossRef ] [ Google Scholar ]
• Small H. (1973). Co-citation in the scientific literature: a new measure of the relationship between two documents. J. Assoc. Inform. Sci. Technol. 24 265–269. 10.1002/asi.4630240406 [ CrossRef ] [ Google Scholar ]
• Song H., Chen F., Peng Q., Zhang J., Gu P. (2017). Improvement of user experience using virtual reality in open-architecture product design. Proc. Inst. Mech. Eng. B J. Eng. Manufact. 232 . [ Google Scholar ]
• Sundar S. S., Xu Q., Bellur S. (2010). “Designing interactivity in media interfaces: a communications perspective,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , (Boston, MA: ACM; ), 2247–2256. 10.1145/1753326.1753666 [ CrossRef ] [ Google Scholar ]
• Sutherland I. E. (1965). The Ultimate Display. Multimedia: From Wagner to Virtual Reality. New York, NY: Norton. [ Google Scholar ]
• Sutherland I. E. (1968). “A head-mounted three dimensional display,” in Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I , (ACM) , 757–764. 10.1145/1476589.1476686 [ CrossRef ] [ Google Scholar ]
• Thomas B., Close B., Donoghue J., Squires J., De Bondi P., Morris M., et al. (2000). “ARQuake: an outdoor/indoor augmented reality first person application,” in Digest of Papers. Fourth International Symposium on Wearable Computers , (Atlanta, GA: IEEE; ), 139–146. 10.1109/ISWC.2000.888480 [ CrossRef ] [ Google Scholar ]
• Ware C., Arthur K., Booth K. S. (1993). “Fish tank virtual reality,” in Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems , (Amsterdam: ACM; ), 37–42. 10.1145/169059.169066 [ CrossRef ] [ Google Scholar ]
• Wexelblat A. (ed.) (2014). Virtual Reality: Applications and Explorations. Cambridge, MA: Academic Press. [ Google Scholar ]
• White H. D., Griffith B. C. (1981). Author cocitation: a literature measure of intellectual structure. J. Assoc. Inform. Sci. Technol. 32 163–171. 10.1002/asi.4630320302 [ CrossRef ] [ Google Scholar ]
• Wrzesien M., Alcañiz M., Botella C., Burkhardt J. M., Bretón-López J., Ortega M., et al. (2013). The therapeutic lamp: treating small-animal phobias. IEEE Comput. Graph. Appl. 33 80–86. 10.1109/MCG.2013.12 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Wrzesien M., Burkhardt J. M., Alcañiz M., Botella C. (2011a). How technology influences the therapeutic process: a comparative field evaluation of augmented reality and in vivo exposure therapy for phobia of small animals. Hum. Comput. Interact. 2011 523–540. [ Google Scholar ]
• Wrzesien M., Burkhardt J. M., Alcañiz Raya M., Botella C. (2011b). “Mixing psychology and HCI in evaluation of augmented reality mental health technology,” in CHI’11 Extended Abstracts on Human Factors in Computing Systems , (Vancouver, BC: ACM; ), 2119–2124. [ Google Scholar ]
• Zyda M. (2005). From visual simulation to virtual reality to games. Computer 38 25–32. 10.1109/MC.2005.297 [ CrossRef ] [ Google Scholar ]