Augmented Information System for Virtual Production Supervision using Smart Glasses

Research Team: Dr Kien Tran (PI), Prof Minh Nguyen, Dr Dafydd Sills-Jones, and Dr Catherine Shi

In 2024, AUT opened its Virtual Production Studio. This facility not only provides Hollywood-grade technologies for filmmaking and visual effects, but also serves a wide range of users, including engineers, scientists, designers, journalists, and artists.

In a Virtual Production (VP) environment, the VP Supervisor (VPS – see Fig. 1(1)) requires access to real-time shot information and continuous monitoring of the physical scene and crew. This information includes stage configurations, colour adjustments, camera positions, lighting setups, sound parameters, and in-camera VFX rendering details. The physical scene encompasses the status and content of the LED display (see Fig. 1(4)), actors (see Fig. 1(5)), physical cameras (see Fig. 1(6)), and interactions among on-site production crew members (see Fig. 1(7)).

Managing and supervising such a complex environment imposes a high cognitive load on the VPS, particularly due to the technical complexity and time-sensitive demands of commands. This research aims to design an immersive interface using Augmented Reality (AR – see Fig. 1(2 & 3)) to abstract and simplify this complexity, thereby supporting the VPS in performing their tasks more effectively and efficiently.

We selected Xreal Air 2 Ultra glasses as a compact platform for delivering the AR interface, leveraging their ability to overlay information while preserving natural vision of the real-world environment. This enables seamless studio supervision and in-person communication.

To support the VP Supervisor, we propose a proof-of-concept system, the Virtual Production Volume Augmented Information System (VPVAIS) (see Fig. 2). This system integrates the XREAL smart glasses platform with AUT’s Virtual Production Volume studio infrastructure to deliver real-time, critical information to the VPS for effective decision-making, collaboration, and oversight.

Therefore, the VPVAIS enhances the VPS’s ability to manage both virtual and physical aspects of production, enabling a more fluid, interactive, and collaborative workflow across the production pipeline.

A user study has been approved by the AUT Ethics Committee (Reference No. 25/324, Blueprint Your Scene: Directing with Virtual Compass) to evaluate the user experience of performing VPS filming tasks using the proposed system, with a focus on usability.

Acknowledgement: I sincerely thank the Faculty of Design and Creative Technologies (DCT), the Virtual Creative Research Centre (VDCRC), and Computer and Information Sciences Research Centre (CISRC) for supporting me and my team in various capacities.

^{Figure 1: An illustration of a VPS experience, where a user wearing AR smart glasses and holding a smartphone controller manages a real-world studio environment (LED display, actor, camera) while interacting with an AR-assisted interface.}

^{Figure 2: VPVAIS’s System Architecture Diagram.}